Windows CI unstable with SourceKitLSPTests

[kavon]

Description

I've seen numerous CI jobs randomly failing with an issue in SourceKitLSPTests, which is unrelated to the changes being tested in the PR. Example logs:

21:58:22  Error: Error: swift exited with code 1.
21:58:22  Invocation:
21:58:22    swift test --scratch-path T:\x86_64-unknown-windows-msvc\SourceKitLSPTests --package-path C:\Users\swift-ci\jenkins\workspace\apple-llvm-project-pull-request-windows\sourcekit-lsp -c release -debug-info-format none --parallel -Xcc -IC:\Users\swift-ci\jenkins\workspace\apple-llvm-project-pull-request-windows\swift-corelibs-libdispatch -Xcc -IC:\Users\swift-ci\jenkins\workspace\apple-llvm-project-pull-request-windows\swift-corelibs-libdispatch\src\BlocksRuntime -Xswiftc -IT:\5\lib\swift\host -Xswiftc -LT:\5\lib\swift\host -Xswiftc -IC:\Users\swift-ci\jenkins\workspace\apple-llvm-project-pull-request-windows\cmark\src\include -Xswiftc -IC:\Users\swift-ci\jenkins\workspace\apple-llvm-project-pull-request-windows\cmark\extensions\include -Xlinker -IC:\Users\swift-ci\jenkins\workspace\apple-llvm-project-pull-request-windows\cmark\extensions\include -Xlinker T:\x86_64-unknown-windows-msvc\cmark-gfm-0.29.0.gfm.13\src\cmark-gfm.lib -Xlinker T:\x86_64-unknown-windows-msvc\cmark-gfm-0.29.0.gfm.13\extensions\cmark-gfm-extensions.lib -Xswiftc -IC:\Users\swift-ci\jenkins\workspace\apple-llvm-project-pull-request-windows\swift-system\Sources\CSystem\include -Xswiftc -IT:\x86_64-unknown-windows-msvc\System\swift -Xlinker -LT:\x86_64-unknown-windows-msvc\System\lib -Xswiftc -IT:\x86_64-unknown-windows-msvc\ToolsSupportCore\swift -Xlinker -LT:\x86_64-unknown-windows-msvc\ToolsSupportCore\lib -Xswiftc -IC:\Users\swift-ci\jenkins\workspace\apple-llvm-project-pull-request-windows\llbuild\products\libllbuild\include -Xswiftc -IT:\x86_64-unknown-windows-msvc\LLBuild\products\llbuildSwift -Xlinker -LT:\x86_64-unknown-windows-msvc\LLBuild\lib -Xswiftc -IT:\x86_64-unknown-windows-msvc\ArgumentParser\swift -Xlinker -LT:\x86_64-unknown-windows-msvc\ArgumentParser\lib -Xswiftc -IT:\x86_64-unknown-windows-msvc\Crypto\swift -Xlinker -LT:\x86_64-unknown-windows-msvc\Crypto\lib -Xlinker T:\x86_64-unknown-windows-msvc\Crypto\lib\libCCryptoBoringSSL.lib -Xswiftc -IT:\x86_64-unknown-windows-msvc\ASN1\swift -Xlinker -LT:\x86_64-unknown-windows-msvc\ASN1\lib -Xswiftc -IT:\x86_64-unknown-windows-msvc\PackageManager\swift -Xlinker -LT:\x86_64-unknown-windows-msvc\PackageManager\lib -Xswiftc -IC:\Users\swift-ci\jenkins\workspace\apple-llvm-project-pull-request-windows\swift-markdown\Sources\CAtomic\inclde -Xlinker T:\x86_64-unknown-windows-msvc\Markdown\lib\CAtomic.lib -Xswiftc -IT:\x86_64-unknown-windows-msvc\Markdown\swift -Xlinker -LT:\x86_64-unknown-windows-msvc\Markdown\lib -Xswiftc -IT:\x86_64-unknown-windows-msvc\Format\swift -Xlinker -LT:\x86_64-unknown-windows-msvc\Format\lib -Xswiftc -IT:\x86_64-unknown-windows-msvc\IndexStoreDB\swift -Xlinker -LT:\x86_64-unknown-windows-msvc\IndexStoreDB\Sources\IndexStoreDB -Xlinker T:\x86_64-unknown-windows-msvc\IndexStoreDB\Sources\IndexStoreDB_CIndexStoreDB\CIndexStoreDB.lib -Xlinker T:\x86_64-unknown-windows-msvc\IndexStoreDB\Sources\IndexStoreDB_Core\Core.lib -Xlinker T:\x86_64-unknown-windows-msvc\IndexStoreDB\Sources\IndexStoreDB_Database\Database.lib -Xlinker T:\x86_64-unknown-windows-msvc\IndexStoreDB\Sources\IndexStoreDB_Index\Index.lib -Xlinker T:\x86_64-unknown-windows-msvc\IndexStoreDB\Sources\IndexStoreDB_LLVMSupport\LLVMSupport.lib -Xlinker T:\x86_64-unknown-windows-msvc\IndexStoreDB\Sources\IndexStoreDB_Support\Support.lib -Xlinker T:\x86_64-unknown-windows-msvc\LMDB\lib\CLMDB.lib -Xswiftc -IC:\Users\swift-ci\jenkins\workspace\apple-llvm-project-pull-request-windows\sourcekit-lsp\Sources\CAtomics\include -Xswiftc -IC:\Users\swift-ci\jenkins\workspace\apple-llvm-project-pull-request-windows\sourcekit-lsp\Sources\CSourcekitd\include -Xlinker T:\x86_64-unknown-windows-msvc\SourceKitLSP\lib\CSourcekitd.lib -Xswiftc -IC:\Users\swift-ci\jenkins\workspace\apple-llvm-project-pull-request-windows\sourcekit-lsp\Sources\CCompletionScoring\include -Xswiftc -IT:\x86_64-unknown-windows-msvc\SourceKitLSP\swift -Xlinker -LT:\x86_64-unknown-windows-msvc\SourceKitLSP\lib
21:58:22  Call stack:
21:58:22    at Invoke-Program, C:\Users\swift-ci\jenkins\workspace\apple-llvm-project-pull-request-windows\swift\utils\build.ps1: line 1047
21:58:22    at <ScriptBlock>, C:\Users\swift-ci\jenkins\workspace\apple-llvm-project-pull-request-windows\swift\utils\build.ps1: line 2015
21:58:22    at Invoke-IsolatingEnvVars, C:\Users\swift-ci\jenkins\workspace\apple-llvm-project-pull-request-windows\swift\utils\build.ps1: line 1066
21:58:22    at Build-SPMProject, C:\Users\swift-ci\jenkins\workspace\apple-llvm-project-pull-request-windows\swift\utils\build.ps1: line 1980
21:58:22    at <ScriptBlock>, C:\Users\swift-ci\jenkins\workspace\apple-llvm-project-pull-request-windows\swift\utils\build.ps1: line 3900
21:58:22    at Invoke-IsolatingEnvVars, C:\Users\swift-ci\jenkins\workspace\apple-llvm-project-pull-request-windows\swift\utils\build.ps1: line 1066
21:58:22    at Test-SourceKitLSP, C:\Users\swift-ci\jenkins\workspace\apple-llvm-project-pull-request-windows\swift\utils\build.ps1: line 3884
21:58:22    at <ScriptBlock>, C:\Users\swift-ci\jenkins\workspace\apple-llvm-project-pull-request-windows\swift\utils\build.ps1: line 829
21:58:22    at Record-OperationTime, C:\Users\swift-ci\jenkins\workspace\apple-llvm-project-pull-request-windows\swift\utils\build.ps1: line 653
21:58:22    at Invoke-BuildStep, C:\Users\swift-ci\jenkins\workspace\apple-llvm-project-pull-request-windows\swift\utils\build.ps1: line 828
21:58:22    at <ScriptBlock>, C:\Users\swift-ci\jenkins\workspace\apple-llvm-project-pull-request-windows\swift\utils\build.ps1: line 4392
21:58:22  
21:58:22      at Invoke-Program, C:\Users\swift-ci\jenkins\workspace\apple-llvm-project-pull-request-windows\swift\utils\build.ps1: line 1051
21:58:22      at <ScriptBlock>, C:\Users\swift-ci\jenkins\workspace\apple-llvm-project-pull-request-windows\swift\utils\build.ps1: line 2015
21:58:22      at Invoke-IsolatingEnvVars, C:\Users\swift-ci\jenkins\workspace\apple-llvm-project-pull-request-windows\swift\utils\build.ps1: line 1066
21:58:22      at Build-SPMProject, C:\Users\swift-ci\jenkins\workspace\apple-llvm-project-pull-request-windows\swift\utils\build.ps1: line 1980
21:58:22      at <ScriptBlock>, C:\Users\swift-ci\jenkins\workspace\apple-llvm-project-pull-request-windows\swift\utils\build.ps1: line 3900
21:58:22      at Invoke-IsolatingEnvVars, C:\Users\swift-ci\jenkins\workspace\apple-llvm-project-pull-request-windows\swift\utils\build.ps1: line 1066
21:58:22      at Test-SourceKitLSP, C:\Users\swift-ci\jenkins\workspace\apple-llvm-project-pull-request-windows\swift\utils\build.ps1: line 3884
21:58:22      at <ScriptBlock>, C:\Users\swift-ci\jenkins\workspace\apple-llvm-project-pull-request-windows\swift\utils\build.ps1: line 829
21:58:22      at Record-OperationTime, C:\Users\swift-ci\jenkins\workspace\apple-llvm-project-pull-request-windows\swift\utils\build.ps1: line 653
21:58:22      at Invoke-BuildStep, C:\Users\swift-ci\jenkins\workspace\apple-llvm-project-pull-request-windows\swift\utils\build.ps1: line 828
21:58:22      at <ScriptBlock>, C:\Users\swift-ci\jenkins\workspace\apple-llvm-project-pull-request-windows\swift\utils\build.ps1: line 4392
21:58:22    From System.Management.Automation.RuntimeException: Error: swift exited with code 1.

The job specifically is: https://ci-external.swift.org/job/apple-llvm-project-pull-request-windows/

It's causing people to re-run the job multiple times.

Reproduction

Expected behavior

Not to spuriously fail.

Environment

Windows CI

Additional information

No response

[kavon]

cc @compnerd

[compnerd]

Can you provide any more context on the failure itself? Seems that the past few runs are green. I think that this might be the issue that @bnbarham and I were chasing down with a race condition somewhere in swift-driver/SPM.

[kavon]

Here's an example log and job: https://ci-external.swift.org/job/apple-llvm-project-pull-request-windows/3144/

consoleText-36.txt

[kavon]

@compnerd Can we disable the parallel testing of SourceKitLSP then until that's resolved?

[delcypher]

Here's another example https://ci-external.swift.org/job/apple-llvm-project-pull-request-windows/3137/console

[compnerd]

Yeah, that definitely looks like the same issue. I don't have a concern with this being switched to single threaded testing, but please check with @bnbarham and @shahmishal as this will impact CI utilization.

[bnbarham]

Seems that the past few runs are green. I think that this might be the issue that @bnbarham and I were chasing down with a race condition somewhere in swift-driver/SPM.

That's already fixed on main.

Can we disable the parallel testing of SourceKitLSP then until that's resolved?

I'd be surprised if this is to do with parallel tests. Really need to attach logs when this happens, we don't keep much CI history. The one attached log failed in CompilationDatabaseTests.testJSONCompilationDatabaseLookup, which definitely has nothing to do with the driver issue. I can look into that test, but it would be nice to know if there's a wider issue here.