feat: Initial E2E test framework for v2 #288

bryphe-coder · 2022-02-15T04:46:36Z

This brings an initial E2E test (really, an integration test - it's only running the server locally, as opposed to against a deployment - but it'd be easy to point playwright to a deployment).

Demo gif:

This test exercises a minimal flow for login:

Run the coderd binary to start a server on 3000
Create an initial user as part of setup
Go through the login flow and verify we land on the projects page

It will be useful to have to ensure that #360 doesn't introduce a regression in the login flow

Future E2E tests that would be useful:

Create a project & verify it shows in the UI
Create a workspace and verify it shows in the UI

Some remaining work left:

Add test to CI (make sure playwright dependencies work correctly)
Call coderd directly instead of develop.sh
Use port 3000 instead of 8080, to exercise the non-next-dev server flow, and create an initial user.
Consolidate the test_results vs test-results paths
Hook up junit reporter output so the results can be sent to datadog
Add tag for datadog
Grab a video / gif of the test

codecov · 2022-02-15T04:51:08Z

Codecov Report

Merging #288 (787213c) into main (2b0f471) will decrease coverage by 0.18%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##             main     #288      +/-   ##
==========================================
- Coverage   67.94%   67.75%   -0.19%     
==========================================
  Files         150      150              
  Lines        8482     8482              
  Branches       72       72              
==========================================
- Hits         5763     5747      -16     
- Misses       2143     2156      +13     
- Partials      576      579       +3

Flag	Coverage Δ
unittest-go-macos-latest	`66.12% <ø> (-0.25%)`	⬇️
unittest-go-ubuntu-latest	`67.44% <ø> (+0.13%)`	⬆️
unittest-go-windows-2022	`65.43% <ø> (+0.02%)`	⬆️
unittest-js	`66.10% <ø> (ø)`

Impacted Files	Coverage Δ
peer/conn.go	`78.00% <0.00%> (-2.82%)`	⬇️
peerbroker/proxy.go	`55.97% <0.00%> (-1.26%)`	⬇️
provisionerd/provisionerd.go	`68.56% <0.00%> (-0.56%)`	⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2b0f471...787213c. Read the comment docs.

greyscaled

This looks nice

bryphe-coder · 2022-02-15T16:29:14Z

Thanks @vapurrmaid ! It's basically what we have in v1 (following the same POM structure you outlined). I'm just started with the integration-style tests first since we don't have an environment

I like the idea of having at least login verified build-over-build as we're standing stuff up

bryphe-coder · 2022-03-01T00:14:54Z

.github/workflows/coder.yaml

+        if: (success() || failure()) && github.actor != 'dependabot[bot]' && runner.os == 'Linux'
+        env:
+          DATADOG_API_KEY: ${{ secrets.DATADOG_API_KEY }}
+          DD_CATEGORY: e2e


@kylecarbs - I added a new DataDog trait for test runs ("category") - intending to be one of unit | integration | e2e

This is awesome!

kylecarbs

Awesome change!

kylecarbs · 2022-03-01T00:26:53Z

.github/workflows/coder.yaml

+        os:
+          - ubuntu-latest
+          - macos-latest
+          - windows-2022


This is so cool! That means we can even test things like cross-os keybinds in the UI 😍

For sure! Ideally - we'd have a matrix of windows | macos | ubuntu x relevant browsers (edge | chrome | firefox | safari). Getting macos + safari would be a huge value add, since we saw some bugs there in v1 (ie https://github.com/coder/m/pull/11093 & https://github.com/coder/m/pull/9508)

Unfortunately, I had to pull out Windows for now because the build is failing - some of our build scripts aren't cross-plat ready - ie, the yarn_install.sh script which fails with:

./scripts/yarn_install.sh Windows Subsystem for Linux has no installed distributions. Distributions can be installed by visiting the Microsoft Store:

https://github.com/coder/coder/runs/5368058980?check_suite_focus=true#step:8:6

Opened a couple issues to break down next steps:

Windows build + run E2E: Build coderd + run E2E tests on Windows #384

Run w/ other browsers: E2E | Run E2E tests against other browsers #385

This reverts commit 15a9a11.

greyscaled

Looks great!!!

.github/workflows/coder.yaml

.gitignore

site/.eslintignore

site/e2e/globalSetup.ts

site/e2e/tests/login.spec.ts

presleyp · 2022-03-01T15:08:10Z

site/e2e/tests/login.spec.ts

+import { SignInPage } from "../pom"
+
+test("Login takes user to /projects", async ({ page, baseURL }) => {
+  await page.goto(baseURL + "/", { waitUntil: "networkidle" })


Interesting that unpacking the baseURL works here since it didn't in v1 - good sign! I believe you can avoid doing the string concatenation here though if you define url in the POM.

Interesting that unpacking the baseURL works here since it didn't in v1 - good sign!

Yep! The v1 E2E tests use a special (external) address to point to the server under test (picked up from RUNTIME_CONFIG: https://github.com/coder/m/blob/833f23892c4802bfcdf9b39c86f415a1dc70a3b2/product/coder/e2e/configuration/runtime.ts#L18)

With these tests - using the webServer automatically populates the baseURL which is convenient.

I believe you can avoid doing the string concatenation here though if you define url in the POM.

Good point - we have some more utilities in the POM layer in v1, I brought them over (ie, BasePOM) here in this change: 7c568de

It's a little bit awkward - since I'm passing both the baseURL and page into the POM's - let me know if you were thinking of a different approach!

Implemented @vapurrmaid 's suggestions which help improve this in: fae6b3f - I'll merge what we have so the test is in place for the NextJS -> Webpack changes, but happy to iterate on it more

@bryphe-coder I'm definitely in favor of this merging sooner than later for iteration, it's in a really good starting spot!

Co-authored-by: G r e y <vapurrmaid@pm.me>

…r into bryphe/feat/initial-e2e-tests

greyscaled · 2022-03-02T00:22:54Z

Thanks for this one @bryphe-coder 🎉 !

bryphe-coder · 2022-03-02T04:38:07Z

@vapurrmaid - I like the proposal, IMO it's simpler than the current implementation I have. Thanks for the suggestion!

EDIT: Incorporated it in fae6b3f

greyscaled · 2022-03-02T06:19:34Z

site/e2e/constants.ts

+export const username = "admin"
+export const password = "password"
+export const organization = "acme-crop"
+export const email = "admin@coder.com"


@bryphe-coder thanks for implementing this! 🎉

I def didn't mean for it to block the work, but it's a nice touch 🎊

Thanks for the great feedback & review @vapurrmaid !

bryphe-coder added 3 commits February 15, 2022 03:39

Add playwright dependency

470d06d

Add initial playwright command to run against development server

ab81ef9

feat: Initial E2E framework

7c970b5

bryphe-coder self-assigned this Feb 15, 2022

greyscaled reviewed Feb 15, 2022

View reviewed changes

bryphe-coder added 17 commits February 28, 2022 19:19

Merge main

853a2b9

Add test/e2e job

2b0f0cb

Add test/e2e job

78ceb9c

Merge branch 'main' into bryphe/feat/initial-e2e-tests

5243dea

Use coderd directly, create initial user in setup script

5eb9314

Ignore e2e tests from jest

8fc553a

Fix typo

1b0e0d3

Consolidate test_results/test-results

f39f59a

Add junit reporter for DataDog

0428a62

Fix junit path

94c7684

Upload E2E test results to DataDog

dfe0c7c

Formatting

8af7d02

Fix lint issues

18f21da

Fix unit test DataDog uploads

bcef4ac

Clean up globalSetup

91bda34

Fix upload of E2E datadog reports

08dc699

Add 'category' for classifying tests in datadog (unit, integration, e2e)

b8cc80f

bryphe-coder marked this pull request as ready for review March 1, 2022 00:12

bryphe-coder requested review from greyscaled, kylecarbs and presleyp March 1, 2022 00:12

Remove coverage collection from E2E folder

a88a722

bryphe-coder commented Mar 1, 2022

View reviewed changes

Run e2e on all platforms

4bad40e

kylecarbs approved these changes Mar 1, 2022

View reviewed changes

bryphe-coder added 3 commits March 1, 2022 00:30

Try running on mac/ubuntu for now

3b0b4cd

Experiment: Unblock windows e2e tests

15a9a11

Revert "Experiment: Unblock windows e2e tests"

081c9cf

This reverts commit 15a9a11.

bryphe-coder mentioned this pull request Mar 1, 2022

Build coderd + run E2E tests on Windows #384

Closed

bryphe-coder added 2 commits March 1, 2022 01:49

Revert change to test/js

439c13b

Add link back to Windows issue

d7f9257

greyscaled approved these changes Mar 1, 2022

View reviewed changes

.github/workflows/coder.yaml Show resolved Hide resolved

.gitignore Show resolved Hide resolved

site/.eslintignore Outdated Show resolved Hide resolved

site/e2e/globalSetup.ts Outdated Show resolved Hide resolved

site/e2e/tests/login.spec.ts Outdated Show resolved Hide resolved

presleyp reviewed Mar 1, 2022

View reviewed changes

presleyp approved these changes Mar 1, 2022

View reviewed changes

bryphe-coder and others added 6 commits March 1, 2022 22:53

Merge branch 'main' into bryphe/feat/initial-e2e-tests

5eedfbe

Consolidate test-results/test_results in .gitignore

7381ca2

Update site/.eslintignore

d3c8e50

Co-authored-by: G r e y <vapurrmaid@pm.me>

Switch to using const vs function

e66c685

Merge branch 'bryphe/feat/initial-e2e-tests' of github.com:coder/code…

bee5e25

…r into bryphe/feat/initial-e2e-tests

Refactor use email, password, username to constants

66b7d60

bryphe-coder and others added 3 commits March 2, 2022 00:24

Expand POM model

7c568de

Fix broken import

41a4516

Merge branch 'main' into bryphe/feat/initial-e2e-tests

7997a16

Push functionality up to BasePOM API

fae6b3f

greyscaled reviewed Mar 2, 2022

View reviewed changes

Merge branch 'main' into bryphe/feat/initial-e2e-tests

787213c

bryphe-coder merged commit 86994da into main Mar 2, 2022

bryphe-coder deleted the bryphe/feat/initial-e2e-tests branch March 2, 2022 17:26

ketang unassigned bryphe-coder Mar 22, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Initial E2E test framework for v2 #288

feat: Initial E2E test framework for v2 #288

bryphe-coder commented Feb 15, 2022 •

edited

codecov bot commented Feb 15, 2022 •

edited

greyscaled left a comment

bryphe-coder commented Feb 15, 2022

bryphe-coder Mar 1, 2022

kylecarbs Mar 1, 2022

kylecarbs left a comment

kylecarbs Mar 1, 2022

bryphe-coder Mar 1, 2022

bryphe-coder Mar 1, 2022

bryphe-coder Mar 1, 2022 •

edited

greyscaled left a comment

presleyp Mar 1, 2022

bryphe-coder Mar 2, 2022

bryphe-coder Mar 2, 2022

bryphe-coder Mar 2, 2022

greyscaled Mar 2, 2022

greyscaled commented Mar 2, 2022 •

edited

bryphe-coder commented Mar 2, 2022 •

edited

greyscaled Mar 2, 2022

bryphe-coder Mar 2, 2022

feat: Initial E2E test framework for v2 #288

feat: Initial E2E test framework for v2 #288

Conversation

bryphe-coder commented Feb 15, 2022 • edited

codecov bot commented Feb 15, 2022 • edited

Codecov Report

greyscaled left a comment

Choose a reason for hiding this comment

bryphe-coder commented Feb 15, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kylecarbs left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bryphe-coder Mar 1, 2022 • edited

Choose a reason for hiding this comment

greyscaled left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

greyscaled commented Mar 2, 2022 • edited

bryphe-coder commented Mar 2, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

bryphe-coder commented Feb 15, 2022 •

edited

codecov bot commented Feb 15, 2022 •

edited

bryphe-coder Mar 1, 2022 •

edited

greyscaled commented Mar 2, 2022 •

edited

bryphe-coder commented Mar 2, 2022 •

edited