Add e2e tests into CI #2002
Add e2e tests into CI #2002
Conversation
There was a problem hiding this comment.
Ideally we would add this to the Continuous Integration CI, it's just a question of how liberal we want to be with our GitHub Actions minutes. We could add a few conditions to save minutes, maybe something like skipping the e2e run on draft PRs?
I'm also curious how much time we could save on e2e tests by stripping down the dvc project that drives it to something that only layers the bare minimum above dvc, removing even torch in favor of something like static fixture data; what we lose in authenticity would easily be made up for by how huge it would be to be able to run e2e tests as liberally as our other suites.
A hyper-minimal testbed project wouldn't necessarily have to replace our current demo, but there could also arguments that it should.
As far as cross-platform tests go, that's a whole other question. Considering windows and mac runners consume 2 and 10 times as much minutes as linux runners, I think our best bet is to save cross-platform e2e runs for situations when they're specifically requested through either workflow-dispatch or some more convenient form of trigger like a special label.
|
I am going to add to the Continuous Integration workflow and strip the For me, having a realistic project is important as it is an end to end test. It is also important to use something realistic when we are manually testing the project. |
| strategy: | ||
| matrix: | ||
| os: [macOS-latest, windows-latest] | ||
| os: [windows-latest] |
There was a problem hiding this comment.
[F] By removing the less valuable macOS-latest tests we make get back more minutes than we expend running the E2E tests in the main workflow.
| import { join } from 'path' | ||
| import { getProcessPlatform } from '../env' | ||
|
|
||
| export const getVenvBinPath = (cwd: string, envDir: string, name: string) => |
There was a problem hiding this comment.
[F] This needs to be separated so we don't have to require 'vscode'
| await workbench.executeCommand('DVC: Show Experiments') | ||
|
|
||
| await webview.open() | ||
| await webview.focus() |
There was a problem hiding this comment.
[F] These seem like better names to me because the methods don't actually open/close the webview
| expandRowButton: 'button[title="Expand Row"]', | ||
| row: '[role=row]', | ||
| table: '[role=table]' | ||
| table: '[role=tree]' |
There was a problem hiding this comment.
[F] This changed when we move up to v18 of React
| await browser.switchToFrame(null) | ||
| await browser.switchToFrame(null) | ||
| }, | ||
| afterTest: async (test, __, { passed }) => { |
There was a problem hiding this comment.
[F] This is close to what is provided in the example tests for the service. If I get time I will extend the way that we do timeouts to also take screenshots.
|
Code Climate has analyzed commit 01e59a9 and detected 0 issues on this pull request. The test coverage on the diff in this pull request is 100.0% (85% is the threshold). This pull request will bring the total coverage in the repository to 96.6% (0.0% change). View more on Code Climate. |
3 participants
1/2
main<- this <- #2013Closes #1471.
Please review the approach. Not much has changed from the last PR (#1993). We just have to
dvc pullthe required data and ensure we checkout the entire git history to get things working in the CI.Comments inline.
There is one remaining test that I would like to write. I'll produce a follow up PR.