How to automatically test storybook components using Backstop?

[Denis-Evseev]

Does anybody know what is the best way to test Storybook components with BackstopJS?
How to use Storybook metadata to generate a config which will test all components in a storybook?

Found an interesting solution that might help:
https://stackoverflow.com/questions/50408579/backstopjs-set-common-selector-for-all-scenarios/51223140#51223140

Here a guy managed to properly isolate components but still doesn't give an answer of how to create a config with all available components in a storybook.

[garris]

Question here to clarify the problem since I am not a storybook user. Let's say you have a storybook with a custom menu dropdown component.

In storybook -- how do you test states currently? Are there already something like integration tests for testing click & hover states etc?

[Denis-Evseev]

We do it in jest unit tests, testing that components have the correct behaviour on click and hover.
Also, storybook allows creating different stories of one component, where each story (with different parameters) can be accessed via a special iframe url.

[Denis-Evseev]

However, Storybook is not used for testing - it is more like visual and simple storage of components.
Useful to present new functionality to business without any server-side implementation.

There are no "proper" integration tests, the only thing you can do there is to create another story of the same component and apply visual testing on it: https://storybook.js.org/docs/react/workflows/testing-with-storybook

[garris]

How are the jest tests integrated into storybook? In ember, when you generate a component, a test partial is also created along with the component scaffolding. When the test suite is run, all partials are imported and aggregated so they run together. Is this similar in storybook? And does this kind of integration make sense in a storybook world?

[Denis-Evseev]

The only thing you can do with Jest & Storybook is to use snapshot testing:
https://storybook.js.org/docs/react/workflows/snapshot-testing

The integration tests (with clicks and hovers) have to be done separately using only jest, where you access components directly (no Storybook is used). Storybook serves as visual storage of components and only works well with visual (snapshot) testing

[garris]

ok. I see. then it sounds like the optimal solution would allow you to use backstop in a similar way to jest? would that be the goal that would best solve this problem?

[Denis-Evseev]

That would be fantastic if you could implement something similar to:
https://storybook.js.org/docs/react/workflows/snapshot-testing

Pretty sure it will boost the popularity of BackstopJS

[garris]

@Denis-Evseev Yes, something like this seems very possible. This can potentially be modeled after the storyshots-puppeteer approach here.

I quickly looked through the code -- it looks like this could be rewritten to wrap backstop instead of puppeteer. If that were to be done -- a backstop helper would wake up during a test run and test each story as a scenario. After all story tests are run a backstop report would be available like normal. Backstop would be configured via a modified backstop config file.

Backstop has a dynamic test creation mode (where there is a config file with no set scenarios -- scenarios are instead generated on the fly (by passing in URLs, scenarioLabels and testIds) to support other test runners. see here some example code.

It looks like the storyshots-puppeteer flow would contain the required metadata to enable this mode. This is similar to how the https://github.com/garris/ember-backstop solution works.

I won't have time to implement this myself -- but If you want to take a look at this I can certainly support your investigation.

[Denis-Evseev]

Thank you @garris for the suggestion.

Managed to get it to work in my project by following steps:

1. Extract all stories using npx sb extract command
2. Used this module.export javascript to access stories

const baseUrl = "http://localhost:6006/"; 
const projectId = "Visual testing"; 

var storiesContainer = require('../storybook-static/stories.json');
var storybookIds = Object.keys(storiesContainer.stories);

console.log(storybookIds);

const relativeUrls = storybookIds.map(storyKey => "iframe.html?id=" + storiesContainer.stories[storyKey].id + "&viewMode=story");

// viewports are: phone (320px X 480px), tablet (1024px X 768px), and desktop (1280px X 1024px).
const viewports = [
  "desktop"
];

module.exports = {
  baseUrl,
  projectId,
  relativeUrls,
  viewports,
};

and the actual config file:

const storybookConfig = require("./storybook-scenarios");
const THREE_SECONDS_IN_MS = 1000;
const scenarios = [];
const viewports = [];

storybookConfig.relativeUrls.map(relativeUrl => {
  scenarios.push({
    label: relativeUrl,
    url: `${storybookConfig.baseUrl}${relativeUrl}`,
    delay: THREE_SECONDS_IN_MS,
    requireSameDimensions: false,
  });
});

storybookConfig.viewports.map(viewport => {
  if (viewport === "phone") {
    pushViewport(viewport, 320, 480);
  }
  if (viewport === "tablet") {
    pushViewport(viewport, 1024, 768);
  }
  if (viewport === "desktop") {
    pushViewport(viewport, 1280, 1024);
  }
});

function pushViewport(viewport, width, height) {
  viewports.push({
    name: viewport,
    width,
    height,
  });
}

module.exports = {
  id: storybookConfig.projectId,
  viewports,
  scenarios,
  onReadyScript: "onReadyScript.js",
  
  paths: {
    bitmaps_reference: "visual-test/backstop_data/bitmaps_reference",
    bitmaps_test: "visual-test/backstop_data/bitmaps_test",
    html_report: "visual-test/backstop_data/html_report"
  },
  report: ["browser"],
  engine: "puppeteer",
  engineOptions: {
    args: ["--no-sandbox"]
  },
  asyncCaptureLimit: 10,
  asyncCompareLimit: 100,
};

[garris]

Looks like a great solution!

What context do you run npx sb extract from? I assume this creates artifacts? I am curious about the workflow. Do you bundle this flow in some node project -- or the command line? How would this look in a CI environment like GH actions?

[garris]

@Denis-Evseev 👆

[Denis-Evseev]

I am running it in React + Storybook project using the command line.
The npx sb extract depends on strorybook-static built folder but only generates one stories.json file.
You can even use command: npx sb extract my-built-storybook-directory my-other-directory. More details here:
https://storybook.js.org/docs/react/workflows/storybook-composition

This is how storybook-static looks like:

Haven't started working with CI - can't answer it yet but should not be a problem, in my opinion

[stoyko-stanchev-pfpt]

If you are using an older version of storybook that doesn't generate a json, or just want an alternative approach:

Open storybook. Iterate through the menu items. "Click" on each item to expand it. That loads all submenu into the dom. Query those subitems. Iterate over and generate backstop scenarios.

[agent-simon-s]

@Denis-Evseev I'm trying to do something along these lines: We're building web components, not react, using storybook book for documentation, and using backstop for regression.

I'm using the iframe url for my scenarios, which isolates the component, but i can't seem select the components for hover/click did you run into this?

[stoyko-stanchev-pfpt]

@agent-simon-s you are asking Denis but figured I might suggest a few things ( aside from perhaps worth a different ticket ).

1. Are you able to select the components from the browser's console manually, using the same selectors you use in backstop? E.g. wondering if you are using shadow dom in closed mode, and sanity checking the iframe url is correct
2. Are you sure the elements are rendered at the time when the selectors kick in? Perhaps try adding some timeout and see if that helps
3. Perhaps try breakpoint debugging - put a breakpoint inside backstop's code right before the selection, see what's happening for yourself
4. Maybe add 1-2 non web-components and try selecting them, to sanity check everything else is working

[agent-simon-s]

Thanks Stoyko Stanchev
nod I didn't want to open a ticket, because i wasn't sure there's something broken. I'm not getting the result I expect, but that could be a number of things.
I'd already played with most of the variables you called out, but through debugging BS i identified i want including puppet script correctly.

carry on ;>

[sergeylukin]

Thanks @Denis-Evseev ! Worked like a charm

[Poimen]

Just FYI to anyone who needs it...

We have a number of stories that use the args settings to pass in variants for components. Writing stories for each of those clutters the StoryBook UI (for instance, button enabled/disabled x4 variants you end up with 8 stories where a single story will suffice for developers).

In this case, we found using the ignore story useful (https://storybook.js.org/docs/react/api/csf#non-story-exports) to export stories that only appear in the extract but not in the UI. So we can write VRT stories but not expose them to the UI.