Nightly Test: Test Explorer Suite - Debugs specified XCTest test failed on Test Insiders / Linux (6.0 - jammy)

[rbenegal]

Test failed with:

Could not find pty 11 on pty host: CodeExpectedError: Could not find pty 11 on pty host
    at M.W (file:///tmp/code/.vscode-test/vscode-linux-x64-insiders/resources/app/out/vs/platform/terminal/node/ptyHostMain.js:45:12233)
    at M.resize (file:///tmp/code/.vscode-test/vscode-linux-x64-insiders/resources/app/out/vs/platform/terminal/node/ptyHostMain.js:45:8773)
    at O.i.<computed> (file:///tmp/code/.vscode-test/vscode-linux-x64-insiders/resources/app/out/vs/platform/terminal/node/ptyHostMain.js:45:2962)
    at Object.call (file:///tmp/code/.vscode-test/vscode-linux-x64-insiders/resources/app/out/vs/platform/terminal/node/ptyHostMain.js:28:4203)
    at Gr.s (file:///tmp/code/.vscode-test/vscode-linux-x64-insiders/resources/app/out/vs/platform/terminal/node/ptyHostMain.js:26:80677)
    at Gr.q (file:///tmp/code/.vscode-test/vscode-linux-x64-insiders/resources/app/out/vs/platform/terminal/node/ptyHostMain.js:26:80200)
    at li.value (file:///tmp/code/.vscode-test/vscode-linux-x64-insiders/resources/app/out/vs/platform/terminal/node/ptyHostMain.js:26:79602)
    at C.B (file:///tmp/code/.vscode-test/vscode-linux-x64-insiders/resources/app/out/vs/platform/terminal/node/ptyHostMain.js:26:2373)
    at C (file:///tmp/code/.vscode-test/vscode-linux-x64-insiders/resources/app/out/vs/platform/terminal/node/ptyHostMain.js:26:2443)
    at C.fire (file:///tmp/code/.vscode-test/vscode-linux-x64-insiders/resources/app/out/vs/platform/terminal/node/ptyHostMain.js:26:2660)
    at MessagePortMain.b (file:///tmp/code/.vscode-test/vscode-linux-x64-insiders/resources/app/out/vs/platform/terminal/node/ptyHostMain.js:24:29136)

...

  1) Test Explorer Suite
       Debugging
         CodeLLDB
           Debugs specified XCTest test:
     Error: Timeout of 300000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/tmp/code/dist/test/integration-tests/testexplorer/TestExplorerIntegration.test.js)
      at listOnTimeout (node:internal/timers:581:17)
      at processTimers (node:internal/timers:519:7)

Link to failed run: https://github.com/swiftlang/vscode-swift/actions/runs/12778956212/job/35622742552

[rbenegal]

Another failure in: https://github.com/swiftlang/vscode-swift/actions/runs/12858610127/job/35847958476 (Linux nightly main jammy)

Test logs

  1) Test Explorer Suite
       Debugging
         CodeLLDB
           Debugs specified XCTest test:
     Error: Timeout of 300000ms exceeded. For async tests and hooks, ensure "done()" is called; if returning a Promise, ensure it resolves. (/tmp/code/dist/test/integration-tests/testexplorer/TestExplorerIntegration.test.js)
      at listOnTimeout (node:internal/timers:581:17)
      at processTimers (node:internal/timers:519:7)

  2) Test Explorer Suite
       Standard
         Run Tests (Parallel)
           XCTests (Run Tests (Parallel))
             Runs failing test:

      AssertionError [ERR_ASSERTION]: Expected values to be loosely deep-equal:

{
  errored: [],
  failed: [
    {
      issues: [],
      test: 'PackageTests.FailingXCTestSuite'
    }
  ],
  passed: [
    'PackageTests.FailingXCTestSuite/testFailing'
  ],
  skipped: [],
  unknown: 0
}

should loosely deep-equal

{
  errored: [],
  failed: [
    {
      issues: [
        'failed - oh no'
      ],
      test: 'PackageTests.FailingXCTestSuite/testFailing'
    },
    {
      issues: [],
      test: 'PackageTests.FailingXCTestSuite'
    }
  ],
  passed: [],
  skipped: [],
  unknown: 0
}
      + expected - actual

       {
         "errored": []
         "failed": [
           {
      +      "issues": [
      +        "failed - oh no"
      +      ]
      +      "test": "PackageTests.FailingXCTestSuite/testFailing"
      +    }
      +    {
             "issues": []
             "test": "PackageTests.FailingXCTestSuite"
           }
         ]
      -  "passed": [
      -    "PackageTests.FailingXCTestSuite/testFailing"
      -  ]
      +  "passed": []
         "skipped": []
         "unknown": 0
       }
      
      at assertTestResults (test/integration-tests/testexplorer/utilities.ts:137:12)
      at Context.<anonymous> (test/integration-tests/testexplorer/TestExplorerIntegration.test.ts:746:42)

[rbenegal]

Strangely, when I all the integration tests together, the test gets stuck after the swift build stage (when run in Dev Containers for 5.10, since many jobs are now failing consistently on 5.10 as well (for example: https://github.com/swiftlang/vscode-swift/actions/runs/12890321891/job/35942486703?pr=1285)).

  Test Explorer Suite
17:14:34: Activating Swift for Visual Studio Code...
17:14:34: SourceKit-LSP setup
17:14:34: focus: undefined
[7612:0123/171434.691999:ERROR:bus.cc(407)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")
17:14:34: test/defaultPackage: add: /workspaces/vscode-swift/assets/test/defaultPackage
17:14:34: WARN: Adding package folder file:///workspaces/vscode-swift/assets/test/defaultPackage twice
17:14:35: test/defaultPackage: Listing tests: starting ... 
[7612:0123/171435.194633:ERROR:bus.cc(407)] Failed to connect to the bus: Could not parse server address: Unknown address type (examples of valid types are "tcp" and on UNIX "unix")
17:14:35: test/defaultPackage: Listing tests: ... done.
    Debugging
      lldb-dap
        - Debugs specified XCTest test
        - Debugs specified swift-testing test
      CodeLLDB
17:14:35: Exec Task: (test/defaultPackage) swift build --build-tests -Xswiftc -diagnostic-style=llvm -Xswiftc -DTEST_ARGUMENT_SET_VIA_TEST_BUILD_ARGUMENTS_SETTING

However, when I run just that test, it seems to have no problem passing. It does seem like the CodeLLDB extension was update recently (vadimcn/codelldb@v1.11.2...master) so perhaps that could be the cause of flakiness since not much should have changed with Swift 5.10 recently.

Will try to post a PR to move this test to the nightly runs so we can keep an eye on it and not generate false positives for other submissions.

[rbenegal]

It seems like setting the lldb.library path (which happens in the test when the full suite runs), causes an issue with the CodeLLDB extension with Swift 5.10, similar to vadimcn/codelldb#1221, where it will fail with:

[ERROR codelldb] Unable to resolve liblldb symbols: "dlsym(0x67663cd0, _ZN4lldb19SBWatchpointOptions21SetWatchpointTypeReadEb): symbol not found"
Error: "\"dlsym(0x67663cd0, _ZN4lldb19SBWatchpointOptions21SetWatchpointTypeReadEb): symbol not found\""
Debug adapter exit code=1 (0x1), signal=null.

or on Linux may fail with something like:

[ERROR codelldb] Unable to resolve liblldb symbols: "/home/user/swift/swift-5.10.1-RELEASE-ubuntu24.04-aarch64/usr/lib/liblldb.so: undefined symbol: _ZN4lldb8SBTarget25WatchpointCreateByAddressEmmNS_19SBWatchpointOptionsERNS_7SBErrorE"
Error: "\"/home/user/swift/swift-5.10.1-RELEASE-ubuntu24.04-aarch64/usr/lib/liblldb.so: undefined symbol: _ZN4lldb8SBTarget25WatchpointCreateByAddressEmmNS_19SBWatchpointOptionsERNS_7SBErrorE\""

Until the extension and/or its dependencies are fixed, it looks like the following temporary workaround options are possible:

• Install a previous version of CodeLLDB (version 1.11.1 should work, can install by going to Extensions panel > Gear icon > Install Specific Version...)
• Clear out the lldb.library path in the VSCode settings. Features like getting variable info will not work correctly, and you may receive a prompt to update CodeLLDB settings at startup again the next time you open the project.
• Move to to a Swift 6.0.3 toolchain by updating Xcode to a newer version or getting a newer toolchain from https://www.swift.org/install/