Skip to content

Test execution timeout it not properly detected #1496

Open
@Marcono1234

Description

@Marcono1234

Describe the bug
When test execution times out (after 5 minutes by default), the extension does not seem to notice that properly:

Version
Extension:

CodeQL extension version: 1.6.12 
CodeQL CLI version: 2.10.4 
Platform: win32 x64

VS Code:

Version: 1.71.0 (user setup)
Commit: 784b0177c56c607789f9638da7b6bf3230d47a8c
Date: 2022-09-01T07:36:10.600Z
Electron: 19.0.12
Chromium: 102.0.5005.167
Node.js: 16.14.2
V8: 10.2.154.15-electron.0
OS: Windows_NT x64 10.0.19043
Sandboxed: No

To reproduce

  1. Open the extension settings
  2. Edit "Additional Test Arguments" and add "--timeout=1"
  3. Run a test
    ❌ The extension does not properly handle the test failure

(remember to revert your settings changes)

Expected behavior

  • The extension should properly detect that the test failed
  • The timeout error message should appear in the "CodeQL Tests" output and not in the .actual file

Additional context
The .actual file contains:

Timeout (5m0s) in api-visibility-issues#09cf2679::getATypeExposingElement#6#fffffff

(with api-visibility-issues being the query name)

Running the test with codeql test run --format=json yields the following response (might helpful to adjust the parsing logic in the extension?):

{
  "test" : "<test-dir>/api-visibility-issues.qlref",
  "pass" : false,
  "failureStage" : "EXECUTION",
  "messages" : [ ],
  "evaluationMs" : 305698,
  "expected" : "<test-dir>/api-visibility-issues.expected",
  "actual" : "<test-dir>/api-visibility-issues.actual",
  "diff" : [
    "--- expected",
    "+++ actual",
    "@@ -1,30 +1,1 @@",
    "-| ...",
    "+Timeout (5m0s) in api-visibility-issues#09cf2679::getATypeExposingElement#6#fffffff"
  ]
}

(slightly minified)

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions