dolthub-cypress

A suite of Cypress.io tests written in Typescript to UI test DoltHub and DoltLab.

Installation

$ yarn && yarn compile

Note: If you're running the Cypress tests on the Apple M1 ARM Architecture, you may need to install Rosetta before running the tests:

$ softwareupdate --install-rosetta --agree-to-license

Running the tests

You can either run our Cypress suite against our deployed production (dolthub.com) or against the local webserver (localhost:3000).

To run the tests against production, you can simply run these commands:

# runs tests using the full UI in Chrome against prod (recommended)
$ yarn cy-open

# runs tests against prod (default browser is Electron)
$ yarn cy-run

# runs tests headless against prod (using Chrome)
$ yarn cy-chrome

To run the tests against the local webserver, make sure you have the server running. (Please note: this option is only currently available for our DoltHub devs. If you want to add a test to our suite, please file an issue or pull request so we can add the appropriate data-cy tag.)

Then, to run the Cypress tests against the local server:

# runs tests in Chrome against local server
$ yarn cy-open-local-dolthub

# runs tests against local server
$ yarn cy-run-local-dolthub

# runs specific tests against local server
$ yarn cy-run-local-dolthub --spec 'cypress/e2e/dolthub/publicPaths/render/database/*'

Running tests against our local webserver gets slightly more complicated when testing our blog, which is a separate application (learn more about our front-end stack and architecture here). Cypress can only run against a single host, so running our blog tests against our local DoltHub server won't work (localhost:3000/blog does not exist, but dolthub.com/blog does). You can test our blog against their local webservers by running these commands:

# For the blog
$ yarn cy-open-local-blog
$ yarn cy-run-local-blog

All the dolthub tests are located in cypress/e2e/dolthub. All the doltlab tests are located in cypress/e2e/doltlab.

Private paths

To run the tests in the privatePaths folder you need to put the test username and password in a Cypress env file. Only DoltHub devs have access to this information and can run these tests locally. This file should not be checked in. The file should look like this:

// cypress.env.json
{
  "TEST_USERNAME": "xxx",
  "TEST_PASSWORD": "xxx"
}

Viewing console outputs

You can view console logs when using cypress run commands by setting the env variable ELECTRON_ENABLE_LOGGING=1. This will not work for Chrome.

Writing tests

To write tests, first, ensure that the element you want to test has a data-cy attribute on it. This attribute is the main way we select elements within the cypress tests, using the cy.get() method.

For example, if you were going to test a Like button

  <button onClick="countLike();"> Like </button>

Add the data-cy attribute like this:

  <button data-cy="like-button" onClick="countLike();"> Like </button>

Then, to select the element in a cypress test you would do something like: cy.get("[data-cy=like-button]").click();

We use this data-cy attribute on elements so that changes to a component or element do not break our tests' selectors, which happens frequently if selecting an element based on it's class or text. See the Cypress documentation for more information about best practices.

Cypress utility functions and types

To help in test writing, included are some helper functions (which can be found in cypress/e2e/utils) designed to abstract away some of the details of cypress test writing and allow for a collection of tests to be written once, then tested across a variety of device sizes.

Most type definitions within cypress/e2e/utils/types.ts have a corresponding new[typeName] function in cypress/e2e/utils/helpers.ts. This helps with writing tests without worrying about the type requirements.

We'll go through the concepts the types were derived from:

Expectation

An Expectation consists of a test description, an element to select, and some assertions to make about that element. The current helper function that creates Expectations is newExpectation.

This is the type definition for an Expectation:

type Expectation = {
  description: string;
  selector: Selector;
  shouldArgs: ShouldArgs;
  clickFlow?: ClickFlow;
  scrollTo?: ScrollTo;
  skip?: boolean;
};

Let's create a sample Expectation for our previously described Like button element. For clarity, each parameter to the newExpectation helper function will be defined as it's own variable.

const testDescription = "should render a Like button";
const selectorString = "[data-cy=like-button]";
const shouldArgs = newShouldArgs("be.visible");
const skip = false;

const likeButtonRendersExp = newExpectation(
  testDescription,
  selectorString,
  shouldArgs,
  skip, // optional and defaults to false
);

testDescription explains what the test tests for, and, since Cypress uses describe and it testing blocks, test descriptions are usually written as if they are the description of the it testing block, reading all together: "it should render a Like button".

Test writers will also be supplying a description to the the outer describe block that houses inner describe and it blocks, so for our example you can imagine a test looking something like:

const pageName = "Page";

describe(`${pageName} should render a Like button`, () => {
  it("should render a Like button", () => {
    // make assertions
  });
});

Getting back to our variablized parameters above, selector is the selector intended to be used with the cy.get() method. This method is optimized to traverse the DOM and find the element(s) containing the specified data-cy attributes.

ShouldArgs

shouldArgs is another object intended to be used with Cypress's assertion method .should().

type ShouldArgs = { chainer: string; value?: any };

chainer refers to the assertion string "be.visible". If you're not familiar with assertions, you can learn more here. While "be.visible" does not require any values, some chainers like "have.length" require a value of a number. Here are some examples:

const beVisible = newShouldArgs("be.visible");
const haveLength = newShouldArgs("have.length", 10);
// note that if you need to provide more than one value you can do so in an array
const contain = newShouldArgs("contain", ["title1", "title2", "title3"]);

Device

A Device contains either a predefined Cypress supported device specified by a certain string that maps to the device's name IRL, ie "macbook-13" or "iphone-6", or it's the custom height and width of a device's viewport (our utility functions currently only support the predefined viewport presets). The value is passed into cy.viewport(). Each Device also comes with a description and a list of tests to run against the provided viewport.

type Device = {
  device: Cypress.ViewportPreset;
  description: string;
  loggedIn: boolean;
  tests: Tests;
};

Continuing with our example above, lets do two things to define the device we want to run likeButtonRendersExp on. First, we will make an array of all our Expectations. These are our tests. Second, we will use newDevice to define an iphone6 to test on (we have some pre-baked Device helper functions in cypress/e2e/utils/device.ts).

const deviceDescription = "iphone6 renders a Like button";
const deviceScreen = "iphone-6";
const loggedIn = false;

const tests = [likeButtonRendersExp];

const iphone6 = newDevice(deviceScreen, deviceDescription, loggedIn, tests);

deviceDescription above describes the test(s) we will be running on this device. Recall that our testDescription ran after the it for an it test block, reading "it should render a Like button". deviceDescription runs after describe in a nested describe block, yielding a testing structure like this:

const pageName = "Page";

describe(`${pageName} should render a Like button on all devices`, () => {
  describe(`${pageName} should render a Like button on iphone-6`, () => {
    it("should render a Like button", () => {
      // make assertions
    });
  });
});

deviceScreen is the predefined viewport size defined by Cypress and represented by the string "iphone-6".

loggedIn is a boolean representing whether this page requires authentication. Our tests currently only run against our logged out pages. We're working on support for running tests against our private pages.

Finally, tests are the collection of Expectations defined that will run against this Device. The next and final step required to run our tests is the helper function: runTestsForDevices.

runTestsForDevices

To use this function, it must be called inside a describe block, so let's put all our previously defined variables together to run a test on an "iphone-6", that checks if the Like button is rendered.

const pageName = "Page";
const currentPage = "/some-page";

describe(`${pageName} should render a Like button on all devices`, () => {
  const loggedIn = false;
  const testDescription = "should render a Like button";
  const selectorString = "[data-cy=like-button]";
  const assertionArgs = newShouldArgs("be.visible");

  const likeButtonRendersExp = newExpectation(
    testDescription,
    selectorString,
    assertionArgs,
  );

  const deviceScreen = "iphone-6";
  const deviceDescription = `should render a Like button on ${deviceScreen}`;

  const tests = [likeButtonRendersExp];

  const iphone6 = newDevice(deviceDescription, deviceScreen, loggedIn, tests);

  const devices = [iphone6];
  runTestsForDevices({ currentPage, devices });
});

Notice in the above test, just inside the describe block, we've defined the currentPage we want to test and we don't need authentication for this page, so loggedIn is false. We then define our Expectation likeButtonRendersExp, and our Device "iphone-6", and create an array of Devices to pass to runTestsForDevices.

First test on first device finished.

ClickFlow

Now when we need to take some actions on a page, click some stuff, then assert some changes, etc. There's an additional helper methods to assist with this.

Let's imagine our Like button element changes some state and element when clicked:

// some state manager somewhere
const [likes, setLikes] = useState(0)

// our Like button with a magic onClick method that updates state
<button data-cy="like-button" onClick={() => setLikes(likes + 1) }> Like </button>
<div data-cy="like-count">{likes}</div>

We can setup our test in a similar way to how we did before, only this time, instead of creating an Expectation, we want to create an Expectation with ClickFlows.

A ClickFlow is conceptually like a story, in that it has a beginning a middle and an end. More specifically, it is a series of optional click actions, followed by a series of Expectations, followed by another series of optional click actions. To simplify, a ClickFlow just wants to know what you want to be clicked first, then, what you want tested, then what you want to click on last.

type ClickFlow = {
  toClickBefore?: Selector; // can be a string or array of strings
  expectations: Expectation[];
  toClickAfter?: Selector;
};

So for our example above, we can think about the ClickFlow we want to define by thinking about how we might test this functionality if we were interacting with the UI directly. First we would assert the Like count to be 0, it's initial value. Then we would want to click the Like button. Finally, we would want to assert that the Like count equaled 1.

Here's how that ClickFlow might be defined using our helper functions newClickFlow and newExpectationWithClickFlow:

const likeButton = "[data-cy=like-button]";
const likeCount = "[data-cy=like-count]";

const containZero = newShouldArgs("to.contain", 0);
const containOne = newShouldArgs("to.contain", 1);

const singleLikeCountExp = newExpectation("", likeCount, containOne);

const testsBetweenClicks = [singleLikeCountExp];

const likeCountClickFlow = newClickFlow(
  // first click
  likeButton,

  // tests to run
  testsBetweenClicks,

  // last click, if any
);

const testDescription =
  "should increase Like count when Like button is clicked";

const likeCountIncreasesExp = newExpectationWithClickFlow(
  testDescription,
  likeCount,
  containZero,
  likeCountClickFlow,
);

Again, we've variablized everything above in order to improve the readability a bit. Lets walk through it. likeButton and likeCount are the selectors we want to work with. They will be passed by way of the helper functions to cy.get().

containZero and containOne are ShouldArgs that will be passed to Cypress's .should() method, and that method will then assert for an element contains either a 0 or a 1 respectively.

Next we make a simple Expectation that we want the test runner to run after we click the Like button and provide an empty string, as the description (it's not needed this deep).

Remember we want to make an assertion before we click the Like button, and an assertion after we click the Like button. We want the Expectation singleLikeCountExp to run after we click the Like button. All it tells the test runner to do is grab the Like count element, and make sure it contains 1.

We then wrap that Expectation in an array, and give it the name testsBetweenClicks. It happens to only contain one test, but can contain more.

Now we are at our ClickFlow definition likeCountClickFlow which is the story we've created to make sure our Like button works correctly. The first argument we pass to newClickFlow is the string (or array of strings) we want Cypress to click first. And these strings are simply our selector strings, so we pass in the likeButton selector. This tells Cypress, click these first, evaluating in Cypress talk to cy.get(selectorString).click().

When Cypress is finished clicking our initial selectors, our test runner will run the Expectations we've passed to to newClickFlow as the second argument. Above, this argument is testsBetweenClicks. And as the variable name suggestions, after the initial clicks run, our runner will run all tests in testsBetweenClicks.

Finally, we can also define a selector(s) to be clicked after testsBetweenClicks finishes, but in our case, this isn't necessary, so we omit this argument.

That is a ClickFlow friends!

We write a simple description, testDescription, for our highest layer of tests and we add our ClickFlows to an array clickFlow.

Now we use our other helper function newExpectationWithClickFlow that accepts all the same arguments newExpectation takes with an additional argument, an array of ClickFlows. These ClickFlows will then run after the Expectation they are coupled to. To clarify, our Expectation with ClickFlows above, likeCountIncreasesExp, will run the same way a simple Expectation will run. likeCount will be selected and expected to contain 0, as the containZero argument specifies. After that, all attached ClickFlows will run, meaning the likeButton will be clicked, and then the likeCount will be selected and expected to contain 1.

Putting it all together

That's it! All that remains is to wrap this in a testing describe block, and we now have a test that checks for state changes!

const pageName = "Page";
const currentPage = "/somePage";

describe(`${pageName} should render a Like button on all devices`, () => {
  const loggedIn = false;

  const likeButton = "[data-cy=like-button]";
  const likeCount = "[data-cy=like-count]";

  const containZero = newShouldArgs("to.contain", 0);
  const containOne = newShouldArgs("to.contain", 1);

  const singleLikeCountExp = newExpectation("", likeCount, containOne);

  const likeCountClickFlow = newClickFlow(
    // first click
    likeButton,
    // tests to run
    singleLikeCountExp,
  );

  const testDescription =
    "should increase Like count when Like button is clicked";

  const likeCountIncreasesExp = newExpectationWithClickFlow(
    testDescription,
    likeCount,
    containZero,
    likeCountClickFlow,
  );

  const deviceDescription = `${pageName} should render a Like button on all devices`;
  const deviceScreen = "iphone-6";

  const tests = [likeCountIncreasesExp];

  const iphone6 = newDevice(deviceDescription, deviceScreen, loggedIn, tests);

  const devices = [iphone6];
  runTestsForDevices({ currentPage, devices });
});

Relevant Blogs

• https://www.dolthub.com/blog/2020-08-10-testing-dolthub-cypress/
• https://www.dolthub.com/blog/2020-08-17-cypress-open-source/
• https://www.dolthub.com/blog/2020-10-23-cypress-login-tests/