[Test]: Set up E2E testing

[brandonroberts]

Setup E2E suites for

• Cypress
• Playwright

[LayZeeDK]

Do you mean when consumers use the create-analog package to generate an Analog workspace or end-to-end tests internal to this workspace?

[brandonroberts]

Internal for this project to validate against those test runners, but I'd like to eventually generate one with a new project also.

I don't personally have a strong preference over what the default would be but I do know that Vite integration opens up the possibility of better integration with Playwright

[LayZeeDK]

Okay, so this issue is about end-to-end test suites for the analog-app project 👍

[LayZeeDK]

I've been setting up Cypress end-to-end tests and here is what I found.

Targets in project.json

Ideal setup

Ideally, we would configure our analog-app-e2e:e2e target like so:

{
  "targets": {
    "e2e": {
      "executor": "@nrwl/cypress:cypress",
      "options": {
        "cypressConfig": "apps/analog-app-e2e/cypress.json",
        "devServerTarget": "analog-app:serve"
      }
    }
  }
}

where the analog-app:serve target is:

{
  "targets": {
    "serve": {
      "executor": "nx:run-commands",
      "options": {
        "cwd": "apps/analog-app",
        "command": "vite --strict-port"
      }
    }
  }
}

However, Nx and/or @nrwl/cypress:cypress doesn't seem to be able to detect:

• When the Vite development server is running
• The port the development server is exposed on

Both of these are detected when using @angular-devkit/build-angular:dev-server with @nrwl/cypress:cypress in regular Angular workspaces.

start-server-and-test

The start-server-and-test package is created exactly for this scenario.

We can configure a analog-app-e2e:cypress target that uses the @nrwl/cypress:cypress executor but expects the development server to already be running. We then use start-server-and-test in the analog-app-e2e:e2e target to start the analog-app:serve target and wait for it to respond on port 3000 before starting the analog-app-e2e:cypress target.

{
  "targets": {
    "e2e": {
      "executor": "nx:run-commands",
      "options": {
        "cwd": "",
        "command": "start-server-and-test 'nx serve analog-app' 3000 'nx cypress analog-app-e2e'"
      }
    },
    "cypress": {
      "executor": "@nrwl/cypress:cypress",
      "options": {
        "cypressConfig": "apps/analog-app-e2e/cypress.json",
        "baseUrl": "http://localhost:3000"
      }
    }
  }
}

where the analog-app:serve target is:

{
  "targets": {
    "serve": {
      "executor": "nx:run-commands",
      "options": {
        "cwd": "apps/analog-app",
        "command": "vite --strict-port"
      }
    }
  }
}

start-server-and-test kills both processes both on success and failure.

Using a separate serve target

Without the start-server-and-test package, we can specify the baseUrl option for the @nrwl/cypress:cypress executor and use a separate analog-app:serve-e2e target instead of analog-app:serve:

{
  "targets": {
    "e2e": {
      "executor": "@nrwl/cypress:cypress",
      "options": {
        "cypressConfig": "apps/analog-app-e2e/cypress.json",
        "devServerTarget": "analog-app:serve-e2e",
        "baseUrl": "http://localhost:3000"
      }
    }
  }
}

where the analog-app:serve-e2e target is:

{
  "targets": {
    "serve-e2e": {
      "executor": "nx:run-commands",
      "options": {
        "color": true,
        "commands": ["vite --strict-port"],
        "cwd": "apps/analog-app",
        "parallel": true,
        "readyWhen": "ready in"
      }
    }
  }
}

The tests run but the development server isn't killed after success or failure.

[brandonroberts]

As we don't have a custom executor/builder for Vite yet, which I'm intentionally holding off on, my preference for the given options would be the second one using start-server-and-test

[LayZeeDK]

As we don't have a custom executor/builder for Vite yet, which I'm intentionally holding off on

Why is that? 🙂

Is it not something that a feature request could be opened for that could be tagged with help wanted?

[brandonroberts]

As we don't have a custom executor/builder for Vite yet, which I'm intentionally holding off on

Why is that? 🙂

Is it not something that a feature request could be opened for that could be tagged with help wanted?

I put some thoughts up here https://twitter.com/brandontroberts/status/1555598666278359042

I'm not opposed to have an executor/builder for Vite/Vitest but it would be very thin as I don't want to cover up the Vite config behind a wall of JSON config

[LayZeeDK]

Seems like Nxext has Vite and Vitest executors 👀

[LayZeeDK]

@brandonroberts
Reopen to include Playwright.

[timdeschryver]

I can pick up the Playwright tests if you want.
I just don't know how much value these will have because these will test the same thing as the Cypress tests? Which is that the app is running correctly.

[LayZeeDK]

@timdeschryver

Vite integration opens up the possibility of better integration with Playwright

According to @brandonroberts

[timdeschryver]

Yea, that's correct. Playwright can be used with any test runner, also Vitest.
The downside is that you have to do the setup to launch/close the browser manually, this is done for you with @playwright/test. The latter is used in #51 , but we can swap it out if we want to add these tests.

[LayZeeDK]

The downside is that you have to do the setup to launch/close the browser manually, this is done for you with @playwright/test.

I used start-server-and-test for the Cypress end-to-end test suites.

[LayZeeDK]

@timdeschryver
Which test runner is currently used in #51?

I think it would be a noteworthy novelty to showcase a full Vitest + Playwright + Angular setup.

[brandonroberts]

I agree, a full Vitest + Playwright example would be a good start. I did some testing with the experimental component tests using Playwright and the Vite plugin also and it works as expected with standalone components.

[LayZeeDK]

Cypress Component Tests which support Angular as of Cypress version 10.5 also use Vite as test runner/development server as far as I'm aware.

What are the experimental component tests you are referring to, @brandonroberts?

[brandonroberts]

Here's the link to the docs

https://playwright.dev/docs/test-components

[timdeschryver]

@LayZeeDK first it used the @playwright/test runner/assertions.
This does most of the heavy lifting for you, and can be configured to your needs.
E.g. which browsers to run the test in, retries, recording, servers to run, ...

Now, it uses vitest to run the tests, and the assertions by vitest.
This seems faster, but we have to create the browsers manually in the before/after hooks.