Skip to content

Commit

Permalink
feat(coverage): Basic test coverage support (#366)
Browse files Browse the repository at this point in the history
This commit lays the foundation for displaying coverage results from `bazel
coverage`.

Currently, the functionality is only exposed through the user-defined tasks in
the `tasks.json`. It is thereby a bit hard to discover. But this is fine for
the time being, because coverage still has a couple of rough edges anyway. As
soon as it is more stable, we should add builtin commands and expose coverage
runs also in the "Bazel Build Target" tree.

Changes in this commit:
* Bumps the VS Code version to 1.88, i.e. the first VS Code version which
  supports the test coverage API.
* Upgrades to ES2022.  I wanted to use `replaceAll` which was introduced in
  ES2021. VS Code 1.88 is based on Node 18 which in turn is based on V8 10.1.
  V8 10.18 supports ECMA-262 also known as ES2023.  However, ES2023 is not yet
  available a target language in the `tsconfig.json`.  Furthermore, Firefox
  does not fully support ES2023, yet. While web browsers are currently not
  relevant, they might become so in the future if we want to turn this into a
  browser-enabled VSCode extension. An upgrade to ES2021 would have been
  sufficient, but I went directly to ES2022 because it might have some of the
  new features might also turn out useful.
* Introduces a custom LCOV parser. I could not find any other high-quality
  open-source parser. E.g., most other parser don't properly parse function
  names with `:` and / or `,` in them.
* Introduces test cases for that custom LCOV parser.
* Add the test cases to GitHub Actions. I followed the
  instructions from https://code.visualstudio.com/api/working-with-extensions/continuous-integration.

Future work:
* Support for branch coverage
* Demangling of function names
* Builtin commands to trigger coverage runs & offer them in the "Bazel Build Tree"

Tested with: Java, C++, Go, Rust
Untested: Python, Swift, Kotlin, Scala and many more

This is the first step towards #362
  • Loading branch information
vogelsgesang committed May 2, 2024
1 parent 8972b0e commit 0d293b8
Show file tree
Hide file tree
Showing 19 changed files with 9,351 additions and 772 deletions.
3 changes: 3 additions & 0 deletions .github/workflows/build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,9 @@ jobs:
- name: Lint
run: npm run check-lint

- run: xvfb-run -a npm test
if: runner.os == 'Linux'

- name: Package VS Code extension
run: npm run package

Expand Down
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
out
node_modules
.vscode-test
*.vsix

# Ignore the generated .d.ts and .js files for protos that end up in the src/
Expand Down
6 changes: 6 additions & 0 deletions .vscode-test.js
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
const { defineConfig } = require("@vscode/test-cli");

module.exports = defineConfig({
files: "out/test/**/*.test.js",
mocha: { ui: "bdd" },
});
1 change: 1 addition & 0 deletions .vscodeignore
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@ out/test/**
scripts/**
src/**
test/**
.vscode-test/

**/*.map

Expand Down
46 changes: 39 additions & 7 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,8 @@ This extension provides support for Bazel in Visual Studio.
- **Buildifier** integration to lint and format your Bazel files (requires that
[Buildifier](https://github.com/bazelbuild/buildtools/releases) be installed)
- **Bazel Task** definitions for `tasks.json`
- **Coverage Support** showing coverage results from `bazel coverage` directly
in VS Code.
- Debug Starlark code in your `.bzl` files during a build (set breakpoints, step
through code, inspect variables, etc.)

Expand Down Expand Up @@ -77,7 +79,7 @@ We can't currently make any recommendation between these two. Both are under act

Bazel tasks can be configured from the `tasks.json` using the following structure:

```json
```jsonc
{
// See https://go.microsoft.com/fwlink/?LinkId=733558
// for the documentation about the tasks.json format
Expand All @@ -88,8 +90,8 @@ Bazel tasks can be configured from the `tasks.json` using the following structur
"type": "bazel",
"command": "test",
"targets": ["${input:pickFlakyTest}"],
"options": ["--runs_per_test=9"]
}
"options": ["--runs_per_test=9"],
},
],
"inputs": [
{
Expand All @@ -98,13 +100,43 @@ Bazel tasks can be configured from the `tasks.json` using the following structur
"command": "bazel.pickTarget",
"args": {
"query": "kind('.*_test', //...:*)",
"placeHolder": "Which test to check for flakyness?"
}
}
]
"placeHolder": "Which test to check for flakyness?",
},
},
],
}
```

## Coverage support (Experimental)

For all `coverage` tasks, the coverage results are automatically loaded into VS
Code upon completion of the task. E.g., you could define your own task to
display the coverage provided by your integration tests using the following task
definition:

```jsonc
{
"label": "Show test coverage from integration test",
"type": "bazel",
"command": "coverage",
"targets": ["//test/integration/...", "//cpp/test/integration/..."],
"options": ["--instrumentation_filter=.*"],
}
```

You might need additional Bazel `options` to get the intended coverage results.
In particular if are using remote builds, you might need to use the
`--experimental_split_coverage_postprocessing` and `--experimental_fetch_all_coverage_outputs`
options. See the documentation on [Code Coverage with Bazel](https://bazel.build/configure/coverage)
for more details.

Code coverage support in this extension is still rather fresh and might still
have rough edges. It was tested with the Java, C++, Go and Rust rules.
In case you are using the code coverage integration with any other language
(Python, Swift, Kotlin, Scala, ...), please let us know how things are going in
bazelbuild/vscode-bazel#367. Please share both positive and negative experiences
you might have.

## Contributing

If you would like to contribute to the Bazel Visual Studio extension, please
Expand Down
7 changes: 4 additions & 3 deletions eslint.config.js
Original file line number Diff line number Diff line change
Expand Up @@ -175,12 +175,13 @@ module.exports = tseslint.config(
},
},
{
files: ["eslint.config.js"],
// `@eslint/js` is currnetly missing type information.
// `@eslint/js` is currently missing type information.
// Re-enable the type checks as soon as we have type infos.
// For vscode-test.js, we also don't use TypeScript, yet.
files: ["eslint.config.js", ".vscode-test.js"],
extends: [tseslint.configs.disableTypeChecked],
rules: {
// Re-enable as soon as we are using ES modules for this config file.
// Re-enable as soon as we are using ES modules for the config files.
"@typescript-eslint/no-var-requires": "off",
},
},
Expand Down
Loading

0 comments on commit 0d293b8

Please sign in to comment.