Open
Description
Describe the bug
When test execution times out (after 5 minutes by default), the extension does not seem to notice that properly:
- The "Testing" tab still shows the test as running
- The timeout error appears (only) in the
.actual
file instead of the "CodeQL Tests" output (similar to Test compilation errors appear in output diff #819)
Version
Extension:
CodeQL extension version: 1.6.12
CodeQL CLI version: 2.10.4
Platform: win32 x64
VS Code:
Version: 1.71.0 (user setup)
Commit: 784b0177c56c607789f9638da7b6bf3230d47a8c
Date: 2022-09-01T07:36:10.600Z
Electron: 19.0.12
Chromium: 102.0.5005.167
Node.js: 16.14.2
V8: 10.2.154.15-electron.0
OS: Windows_NT x64 10.0.19043
Sandboxed: No
To reproduce
- Open the extension settings
- Edit "Additional Test Arguments" and add
"--timeout=1"
- Run a test
❌ The extension does not properly handle the test failure
(remember to revert your settings changes)
Expected behavior
- The extension should properly detect that the test failed
- The timeout error message should appear in the "CodeQL Tests" output and not in the
.actual
file
Additional context
The .actual
file contains:
Timeout (5m0s) in api-visibility-issues#09cf2679::getATypeExposingElement#6#fffffff
(with api-visibility-issues
being the query name)
Running the test with codeql test run --format=json
yields the following response (might helpful to adjust the parsing logic in the extension?):
{
"test" : "<test-dir>/api-visibility-issues.qlref",
"pass" : false,
"failureStage" : "EXECUTION",
"messages" : [ ],
"evaluationMs" : 305698,
"expected" : "<test-dir>/api-visibility-issues.expected",
"actual" : "<test-dir>/api-visibility-issues.actual",
"diff" : [
"--- expected",
"+++ actual",
"@@ -1,30 +1,1 @@",
"-| ...",
"+Timeout (5m0s) in api-visibility-issues#09cf2679::getATypeExposingElement#6#fffffff"
]
}
(slightly minified)