Frequent clangd test failures on Windows buildbot

[HighCommander4]

It appears there are frequent clangd test failures on the Windows buildbots.

A few samples:

https://lab.llvm.org/buildbot/#/builders/123/builds/20148/steps/4/logs/stdio:

Failed Tests (3):
  Clangd Unit Tests :: ./ClangdTests.exe/LSPTest/DiagnosticsHeaderSaved
  Clangd Unit Tests :: ./ClangdTests.exe/LSPTest/FeatureModulesThreadingTest
  Clangd Unit Tests :: ./ClangdTests.exe/LSPTest/ModulesTest

https://lab.llvm.org/buildbot/#/builders/123/builds/20145/steps/4/logs/stdio

Failed Tests (2):
  Clangd Unit Tests :: ./ClangdTests.exe/LSPTest/FeatureModulesThreadingTest
  Clangd Unit Tests :: ./ClangdTests.exe/LSPTest/ModulesTest

https://lab.llvm.org/buildbot/#/builders/123/builds/20142/steps/4/logs/stdio

Failed Tests (1):
  Clangd Unit Tests :: ./ClangdTests.exe/LSPTest/DiagnosticsHeaderSaved

https://lab.llvm.org/buildbot/#/builders/123/builds/20137/steps/4/logs/stdio

Failed Tests (8):
  Clangd Unit Tests :: ./ClangdTests.exe/12/38
  Clangd Unit Tests :: ./ClangdTests.exe/31/38
  Clangd Unit Tests :: ./ClangdTests.exe/7/38
  Clangd Unit Tests :: ./ClangdTests.exe/ClangdServerTest/ReparseOnHeaderChange
  Clangd Unit Tests :: ./ClangdTests.exe/ClangdServerTest/TestStackOverflow
  Clangd Unit Tests :: ./ClangdTests.exe/ClangdThreadingTest/NoConcurrentDiagnostics
  Clangd Unit Tests :: ./ClangdTests.exe/CompletionTest/DynamicIndexIncludeInsertion
  Clangd Unit Tests :: ./ClangdTests.exe/LSPTest/CDBConfigIntegration

https://lab.llvm.org/buildbot/#/builders/123/builds/20134/steps/4/logs/stdio

Failed Tests (3):
  Clangd Unit Tests :: ./ClangdTests.exe/28/38
  Clangd Unit Tests :: ./ClangdTests.exe/LSPTest/GoToDefinition
  Clangd Unit Tests :: ./ClangdTests.exe/LSPTest/RecordsLatencies

https://lab.llvm.org/buildbot/#/builders/123/builds/20131/steps/4/logs/stdio

Failed Tests (1):
  Clangd Unit Tests :: ./ClangdTests.exe/LSPTest/CDBConfigIntegration

https://lab.llvm.org/buildbot/#/builders/123/builds/20022/steps/4/logs/stdio

Failed Tests (2):
  Clangd Unit Tests :: ./ClangdTests.exe/28/38
  Clangd Unit Tests :: ./ClangdTests.exe/LSPTest/GoToDefinition

A common element seems to be the following message printed before the failures:

No result from call after 10 seconds!

cc @kadircet @hokein @sam-mccall

[AaronBallman]

I'm noticing a new message that might explain some of the instability.

https://lab.llvm.org/buildbot/#/builders/123/builds/20277

has a failure:

C:\b\slave\clang-x64-windows-msvc\llvm-project\llvm\include\llvm/Support/PointerLikeTypeTraits.h(58): fatal error C1088: Cannot flush compiler intermediate file: 'C:\Users\buildbot\AppData\Local\Temp\_CL_3eb71680ex': No space left on device

so I'm contacting the bot owner to see if they can address that; hopefully that will resolve the mysterious failures. However, if the only error you get is "no result from call after 10 seconds" but the root cause is disk space issues, perhaps the diagnostic can be improved in clangd?

[AaronBallman]

Any update on this?

Spot-checking https://lab.llvm.org/buildbot/#/builders/clang-x64-windows-msvc shows most of the failures are spurious ones from clangd tests. Given that this has been happening for a few weeks it's become kind of disruptive; should we be exploring disabling these tests?

[kadircet]

Considering that even the Clangd Unit Tests :: ./ClangdTests.exe/LSPTest/RecordsLatencies test is failing, for which I can't get the logs. I am inclined to believe that load on this build-bot is just too high and our 10 seconds default timeouts waiting for async work to finish is not enough (This test literally calls a non-existent function, and gets a reply from the ClangdLSPServer, without even reaching ClangdServer).

So sending out a patch to bump the deadlines. It'd be great to get rid of them completely, but they're the only thing preventing clangd from hanging the buildbots forever if things go wrong.

[kadircet]

patch for LSP tests in https://reviews.llvm.org/D158426, will take a look at ManyUpdates separately as it's a little bit more delicate.