[Experiment] Use puppeteer-testing-library in e2e tests #28380
Conversation
kevin940726
added
the
[Type] Automated Testing
|
Size Change: +579 B (0%) Total Size: 1.47 MB
ℹ️ View Unchanged
|
gziolo
added
[Type] Technical Prototype
There was a problem hiding this comment.
Very nice!
I like that it's easier to read as a code reviewer, that it tests the editor as it "appears" to a screen reader instead of relying on CSS selectors that might change, and that the tests might be a little less prone to intermittent failure because find implicitly uses waitForSelector.
I'm keen to try it out in widgets. If it works well for us, then we can use it in other parts of Gutenberg. If it doesn't, we can remove it.
Does it affect how long tests take to run? By how much?
|
The original The bottleneck seems to be the |
|
It’s great that you explore all those improvements, I appreciate all the efforts 👍🏻 I see that Noting, that it's discouraged from usage in Gutenberg with an ESLint rule: Lines 116 to 120 in c888d27 Not sure if that is an issue, but it's hidden in this PR so I thought I will mention it. As for the API proposed: const button = await find({
role: 'button',
name: 'My button',
});I agree that it looks clean and simple on the surface. The remaining question is whether, for folks writing tests, it'd be more preferable to use over web APIs they already might know |
I think that that usage of |
I think there's no harm in trying out this new approach in a small area of the codebase (widgets). If it helps a lot with test stability (it might not!) then we can think about how to roll it out more broadly, whether that means re-writing tests (probably not for the reasons you mention) or just recommending this for new tests. |
I didn’t know about that PR! It looks like
I totally agree that convincing others to adopt this new API might be the most important part. I can try to write up some more docs or perhaps some tutorials to help developers know the benefits and the trade-offs. I'll also reach out to the javascript testing community outside Gutenberg and gain some feedbacks from them. |
Yes, it sounds like a reasonable plan 👍 |
kevin940726
force-pushed
the
experiment/puppeteer-testing-library
branch
from
c4301c9
to
16a029f
Compare
kevin940726
force-pushed
the
experiment/puppeteer-testing-library
branch
from
16a029f
to
df28ee8
Compare
kevin940726
force-pushed
the
experiment/puppeteer-testing-library
branch
2 times, most recently
from
f95e303
to
3ae06e4
Compare
There was a problem hiding this comment.
I'd be happy to try this if we can convert it from a draft and get the tests passing.
I do wonder if there's a way we can indicate to other developers that this is being trialled as an experiment ... something like a linting rule for import { something } from 'puppeteer-testing-library'; that has to be specifically allowed in the widgets test file?
@gziolo Are you ok with the changes to the config now?
kevin940726
force-pushed
the
experiment/puppeteer-testing-library
branch
from
3ae06e4
to
f5a1cba
Compare
Sure, it's fine. |
|
Howdy! 👋 I've noticed this experiment while revisiting the idea of migration to Playwright. Since the last time it was considered/tried, Playwright got quite a few updates that IMO have moved it to the front of the competition, making it a solid candidate for Gutenberg. I'm writing here because it seems to me like adopting this library would make the potential migration much more difficult due to API differences (unless that lib itself moves to Playwright). I think that moving to Playwright would actually lower the dev entry threshold and increase the overall stability of the tests (e.g. thanks to explicit waits with auto-wait). I'm happy to make a case for Playwright using some examples/comparisons, but right now I just wanted to raise the concern, and find out the current status and plans for this experiment. For the most prominent features that IMO make Playwright the goto framework, check out the auto-waiting mechanism, a wide range of supported selectors, debugging tools, and browsers support. |
|
@WunderBart Hi, thanks for the ping! This library is designed to work with playwright, as long as we keep using chromium to run our tests. We might need a few tweak here and there, but it shouldn't be a blocker. I'm happy to help when the migration begin. I still believe that the queries and helpful matchers this library brings have benefits, even with all the more advanced selectors playwright supports. |
Which means this will prevent any attempts at running tests on other browsers like Firefox :/ |
Yep, as for the current implementation, it will only work in chromium-based browsers. |
Description
An alternative approach to #27260.
This PR integrates with
puppeteer-testing-library. A queries/matchers library I created to help us write accessible and consistent e2e tests with puppeteer and jest.You can find the full documentation of the library in its README. Or, here's a simplified version of it:
Basically there are only two queries available:
findandfindAll.find(query), well, finds a single element which matches the query, or throw an error if there're more than one matching elements or there're no matching elements. It will by default wait for up to 3 seconds until it finds an element, or throw an error if timeout. The query will return anElementHandleif found.The query is just a collection of accessible properties of the DOM node, like
role,name,value, etc. The most commonly used fields are, by the order of their priories,role,name,text,selector.findAllfinds all matching elements, simply a multiple version offind. It returns an array ofElementHandles.There're also some helpful matchers:
toMatchQuery,toHaveFocus,toBeVisible, to name a few.This PR refactors widgets test (
adding-widgets.test.js) to usepuppeteer-testing-libraryto demonstrate its usage and the before&after.How has this been tested?
Types of changes
New feature
Checklist: