E2E Tests

E2E Tests

The Oeffis project uses E2E tests to ensure that the application works as expected. These are based on the following technologies:

• Cypress
• Gherkin
• Cypress-Cucumber-Preprocessor
• Mocks-Server
• Testing Library

This document describes how to run the E2E tests and how to write new ones.

Running the tests

The easiest way to run the E2E tests is by pushing them onto a branch within the GitHub repository. This will trigger a GitHub Action that will run the tests and report the results back to GitHub. The results can be found in the "Actions" tab of the repository, which also include a cypress test report that contains images and videos of the test run.

However, as this is a rather inconvenient way to develop new tests, there is also a VSCode Task to run the tests locally. First, start the application by running the "Full Stack" launch configuration as usual. Then press "Ctrl + Shift + P" and select "Run Task", followed by "Open Cypress". Alternatively you can enter npx cypress open in a terminal within the e2e directory. A Cypress window will open and request you to select, which kind of tests should be run. Choose "E2E Testing". The next window will ask you to select the browser in which the tests should be run. Choose any one you like (I recommend Electron) and click "Start testing E2E Testing in {chosen browser}". The last window will ask you, which test suite (or in our case - which feature file) to run. Pick the one you are working on and the scenarios of this feature will start to run. You will be able to see the steps on the left and the application on the right hand side.

Developing new tests

Test Structure

The End-To-End tests are described in .feature-files written in the Gherkin language. Theses files contain descriptions of features in multiple scenarios, which is made of individual steps. The steps are written plain english. Each file is accompanied by a TypeScript file that contains definitions on what an individual step does. Steps can be parameterized and reused steps can be defined in typescript files further up the hierarchy. All of these files are located in the e2e/features folder. They are preprocessed by the Cypress-Cucumber-Preprocessor to make them usable for Cypress The following example shows a feature file and its corresponding step definitions:

# e2e/features/routePlanner/routePlanner.feature
Feature: RoutePlanner
  Background:
    Given the start page is open

  Scenario: a start location can be picked
    When the start input is clicked
    And the mock server is prepared to return a canned response for "Gelsenkirchen Hbf"
    And "Gelsenkirchen Hbf" is entered into the search field
    Then one of the results is "Gelsenkirchen, Hbf"
    And one of the results is "Gelsenkirchen, Hauptbahnhof Parkhaus"
    When the result "Gelsenkirchen, Hbf" is clicked
    Then the start location should be "Gelsenkirchen, Hbf"

// e2e/features/routePlanner/routePlanner.ts
import { Given, Then, When } from "@badeball/cypress-cucumber-preprocessor";

Given("the start page is open", () => {
  cy.visit("/");
});

When("the start input is clicked", () => {
  cy.findByTestId("origin-input-clickable").click();
});

When("the mock server is prepared to return a canned response for {string}", (query: string) => {
  cy.mocksSetCollection(`Query for '${query}'`);
});

When("{string} is entered into the search field", (query: string) => {
  cy.findByTestId("location-search-input").type(query);
});

Then("one of the results is {string}", (result: string) => {
  cy.findAllByTestId("locationName").contains(result).should("exist");
});

When("the result {string} is clicked", (result: string) => {
  cy.findAllByTestId("locationName").contains(result).click();
});

Then("the start location should be {string}", (result: string) => {
  cy.findByTestId("origin-input-clickable").should("have.value", result);
});

Matching UI elements

The E2E tests use the data-testid attribute to match UI elements. This is done to ensure that the tests are not dependent on the actual implementation of the UI. A UI element can be found using it's test-id by using either the cy.findByTestId(id: string) or the cy.findAllByTestId(id: string) commands from the Testing Library, the later being used to find multiple items. The following example shows how the data-testid attribute is used to identify a search input field:

<input
  data-testid="location-search-input"
  type="text"
  placeholder="Start"
  value=""
  spellcheck="false"
/>

Mocking Web Service Requests

The E2E tests use a mock server to requests to the VRR Service. This is done to ensure that the tests are not dependent on the availability of the actual service and to have better control over the responses. The mock server is based on Mocks-Server and is configured in the e2e/mocks folder.

The mock server configuration consists of routes and collections. Each route defines in it's own TypeScript file how the mock server responds to requests. As different responses may be required for different tests, each route can configure multiple variants, each representing one response configuration. A variety of options can be configured, including canned or computed responses, failure status codes, pass-through to the actual service (please don't actually use this) and others. This configuration can also be done for a group of routes that share the same path prefix. The following returns a canned response, which is stored in a separate typescript file for better readability:

// e2e/mocks/stopFinder/stopFinder.ts
export default [
    {
        id: "stop_finder",
        url: "/static03/XML_STOPFINDER_REQUEST",
        method: "GET",
        variants: [
            {
                id: "result for 'Gelsenkirchen Hbf'",
                type: "json",
                options: {
                    status: 200,
                    body: StopFinderResponseGelsenkirchenHbf
                }
            }
        ]
    }
];

A collection groups multiple routes together and is defined in the collections.json file. To make the mock server use a specific route variant configuration, a collection containing must be selected using the cy.mocksSetSelection(collection: string) command. The following file shows an example collections file.

/* e2e/mocks/collections.json */
[
  {
    "id": "base",
    "routes": []
  },
  {
    "id": "Query for 'Gelsenkirchen Hbf'",
    "from": "base",
    "routes": [
      "stop_finder:result for 'Gelsenkirchen Hbf'"
    ]
  }
]

Debugging

Debugging the tests itself is fairly limited, as Cypress is using a queue system to avoid asynchronousness in the tests. The code under test however can be debugged. The frontend by opening the inspection tools within the Test Run Window and the Backend by setting breakpoints in VSCode. Note that setting breakpoints and debugging will likely cause the tests to fail, as the assertions will timeout.