Add e2e tests (wdio-vscode-service)

[mattseddon]

This PR implements end-to-end tests for the repository. It does not wire the new tests up with the CI (that will be the next PR).

I have used the wdio-vscode-service to do this and used example code from vscode-marquee to make the tests worthwhile.

yarn test:e2e:

1. Create the Python venv and install the required packages (runs yarn setup:venv)
2. Build the webviews, move them into the appropriate folder and compile the extension (using existing turbo repo commands)
3. Run the end-to-end tests.

The end-to-end test suite performs the following actions:

1. Open the view container and wait for it to load
2. Open the experiments webview
3. Check the webview contains a table
4. Check the experiments webviews table has rows
5. Run an experiment
6. Check the experiments webview has 15 more rows
7. Close the experiments webview
8. Open the plots webview
9. Check the webview has 6 vega plots
10. Check each plot has data

There are no mock/stubs in place so we are relying on the CLI to provide the correct data.

Demo (video broken into 2 parts because it was too big)

e2e.experiments.mov

e2e.plots.mov

[qlty-cloud-legacy]

Code Climate has analyzed commit 4d637c7 and detected 0 issues on this pull request.

The test coverage on the diff in this pull request is 100.0% (85% is the threshold).

This pull request will bring the total coverage in the repository to 96.6% (0.0% change).

View more on Code Climate.

[rogermparent]

Exciting stuff! Some comments/questions

• It seems this test cannot use the Python extension, so it can't test our integration with it nor can it run in environments that depend on using virtual environments through python (and maybe at all).
• I see this doesn't run on CI yet, does it work? I'd imagine so since the integration tests that spawn a VS Code instance work on CI, but anything could happen. just noticed this was mentioned in the OP

[rogermparent]

You mention test:e2e in the OP, but it's test-e2e here. Also, do we want to add an alias to this script in the workspace root?

[mattseddon]

I was really confused by this. The problem is that I was working on this late last night and didn't push my final commit up. What you asked for is here. Sorry.

[mattseddon]

• It seems this test cannot use the Python extension, so it can't test our integration with it nor can it run in environments that depend on using virtual environments through python (and maybe at all).

I'll start with dvc.pythonPath and then try to use the Python extension. I can't remember if it is possible to start the session with another extension installed. I'll try and work it out.

[qlty-cloud-legacy]

Code Climate has analyzed commit f98d17a and detected 0 issues on this pull request.

The test coverage on the diff in this pull request is 100.0% (85% is the threshold).

This pull request will bring the total coverage in the repository to 96.6% (0.0% change).

View more on Code Climate.

[mattseddon]

• It seems this test cannot use the Python extension, so it can't test our integration with it nor can it run in environments that depend on using virtual environments through python (and maybe at all).

I'll start with dvc.pythonPath and then try to use the Python extension. I can't remember if it is possible to start the session with another extension installed. I'll try and work it out.

@rogermparent I've looked into the service source code and it is not currently possible to start the service without passing --disable-extensions. Look here and work backwards if you're interested. I've sent the service maintainer/author a DM on the VS Code community slack to ask if he'll accept a PR to change this. Will go with setting the option explicitly for the time being.

[christian-bromann]

🎉