Add e2e tests for inline package #22646

bvaughn · 2021-10-28T15:38:19Z

DevTools has good unit test coverage but no e2e test coverage. This is because the support for testing extensions is pretty limited in e2e testing libraries. That being said, we could probably get 80% of the value of e2e testing leveraging our react-devtools-inline package.

We should create a few example tests (e.g. load hook names, select a component and edit props/state, etc) to see how viable this is.

For now, these tests should not block PRs from landing.

The text was updated successfully, but these errors were encountered:

akgupta0777 · 2021-10-31T09:35:06Z

Speaking of libraries which one we should use, Cypress or jest-puppeteer for E2E testing ? I think Cypress is a great choice.

Biki-das · 2021-11-01T09:17:31Z

Would like to work on this, Puppeter would be a good choice for this

akgupta0777 · 2021-11-01T10:00:19Z

If we use puppeteer we can only test extension on chrome.
But with cypress E2E tests can run on firefox,chrome and edge too.

Biki-das · 2021-11-01T10:33:04Z

If we use puppeteer we can only test extension on chrome.
But with cypress E2E tests can run on firefox,chrome and edge too.

Yeah and the docs of cypress is super beginner friendly how about we both work together and make pull request in patches?

akgupta0777 · 2021-11-01T10:36:54Z

If we use puppeteer we can only test extension on chrome.
But with cypress E2E tests can run on firefox,chrome and edge too.

Yeah and the docs of cypress is super beginner friendly how about we both work together and make pull request in patches?

Yeah I have no issues in working together.

But I want opinion on this issue by @bvaughn .
Also want some info for a Kickstart.

Biki-das · 2021-11-01T10:43:58Z

If we use puppeteer we can only test extension on chrome.
But with cypress E2E tests can run on firefox,chrome and edge too.

Yeah and the docs of cypress is super beginner friendly how about we both work together and make pull request in patches?

Yeah I have no issues in working together.

But I want opinion on this issue by @bvaughn .
Also want some info for a Kickstart.

Well he gave us some, firstly we would have to individually write test for. Hooks, props etc too see if it's viable or not

akgupta0777 · 2021-11-01T10:59:00Z

I mean the Library isn't decided yet.

Biki-das · 2021-11-01T11:03:33Z

I mean the Library isn't decided yet.

Yeah let bvaughn decide!

bvaughn · 2021-11-01T12:50:20Z

To be honest, I don't know 😄

Let's see what others say:
https://twitter.com/brian_d_vaughn/status/1455155155830194184

Biki-das · 2021-11-01T13:02:20Z

Already voted let see! 😀😀

bvaughn · 2021-11-01T13:11:26Z

So far it seems like a lot of votes for Cypress with some write-ins for Playwright

akgupta0777 · 2021-11-01T14:18:54Z

I want to do some research on playwright as it might be better option than cypress.

bvaughn · 2021-11-01T14:25:33Z

Seems pretty promising, yes. I'm interested in Playwright now as well.

Biki-das · 2021-11-01T14:32:44Z

ok i would also try to do some work with playwright

bvaughn · 2021-11-01T16:54:56Z

At this point, my vote is for Playwright. If either of you would like to put together a demo/PR that tests the react-devtools-inline package– that would be a great starting point.

Biki-das · 2021-11-01T17:04:50Z

At this point, my vote is for Playwright. If either of you would like to put together a demo/PR that tests the react-devtools-inline package– that would be a great starting point.

sure will start working on it

akgupta0777 · 2021-11-02T07:34:17Z

@bvaughn
Here is the demo with Cypress.
In this demo I am running two tests
First one checks there are 3 todo items on the list by default
Second one adds a todo item Hello world on the list by typing it on the input.

As it is a demo I haven't emulated the devtools but we can surely do it with cypress.

2021-11-02.12-56-34.mp4

bvaughn · 2021-11-02T13:43:29Z

@akgupta0777 Thanks for sharing the demo.

As it is a demo I haven't emulated the devtools but we can surely do it with cypress.

Just in case there's any confusion, this is the part I'm specifically interested in 😄

akgupta0777 · 2021-11-02T14:12:46Z

Sure will ask from you.

akgupta0777 · 2021-11-03T06:40:33Z

@bvaughn Are these ids attached to inputs and other components dynamic or static ? I rebuild the react-devtools-shell many times but IDs are not changing so can I assume these are static ?

bvaughn · 2021-11-03T13:22:24Z

@akgupta0777 Should probably attach test ids to things we want to have automated tests for.

akgupta0777 · 2021-11-04T08:52:59Z

@bvaughn what assertion should I make to test props on ListItems. See attached image below for reference

I am thinking of making an assertion on props exist or not if exist test will PASS and if not it will FAIL.
Do you want to suggest any other assertion ?

bvaughn · 2021-11-04T14:00:49Z

Let's just start with a proof of concept test. The specifics of what it tests isn't super important.

akgupta0777 · 2021-11-04T14:06:23Z

Uhmm... Proof of concept ? Sorry I don't know what doest that mean. Can you explain a little bit please 😅😅

bvaughn · 2021-11-04T14:07:11Z

Just an example test.

bvaughn · 2021-11-04T14:07:53Z

To clarify, I don't have much bandwidth right now to describe/design specific test scenarios. I just filed this task for my team to consider for later.

akgupta0777 · 2021-11-04T14:08:04Z

Ok I will try my best and report back to you.

akgupta0777 · 2021-11-13T17:39:42Z

I tried a lot of things, different approaches ,strategies with cypress but it is really very hard to write tests with iframes.
I switched to playwright and want to try it and really playwright is a outstanding choice for apps with iframes. It's really a joy plus its very productive.
I wrote tests with playwright in a very short time than cypress.

Anyways, I am dropping the Proof of concept. #22754

bvaughn · 2021-11-15T17:34:17Z

I love the direction we're going with Playwright.

One additional thing we could consider in the future is to use the (still experimental) test selector API for e2e testing purposes. (See #22760.) This would not be as complete a solution as Playwright, but would have the added benefit of dog fooding our own APIs.

akgupta0777 · 2021-11-17T15:48:26Z

I am keeping an eye on test selector API too. Will try to get involved in development of API
.

bvaughn · 2021-11-17T15:54:49Z

You can now test it out in the react-dom@experimental build by aliasing react-dom to react-dom/testing if you'd like.

Here's a readme:
https://gist.github.com/bvaughn/d3c8b8842faf2ac2439bb11773a19cec

akgupta0777 · 2021-11-17T16:05:12Z

Here's a readme: https://gist.github.com/bvaughn/d3c8b8842faf2ac2439bb11773a19cec
Thanks I am about to ask you about this 😄.

bvaughn · 2021-11-17T18:28:15Z

@akgupta0777 I'm still not able to run #22754 locally. Would appreciate your help if you have any insight.

akgupta0777 · 2021-11-18T02:33:21Z

I am happy to help and I am on it to find the root cause.
Works on my machine though

akgupta0777 · 2021-11-18T06:51:00Z

running yarn test and yarn test --prod also gives the same error. Tests are failing.

bvaughn · 2021-11-18T15:51:29Z

@akgupta0777 So e2e tests pass for you, but (regular) unit tests fail with the react-is syntax error?

akgupta0777 · 2021-11-18T15:56:02Z

No just ignore that it was a mistake I did when trying to fix this.

Actually the problem is test runners like playwright only uses commonjs and react-is is using export.
Babel is not transpiling properly I think.

bvaughn · 2021-12-09T19:10:02Z

Hey @akgupta0777!

Thanks to the fix from @eps1lon, I can run these locally now 🥳

But the third test added in #22754 consistently fails for me:

akgupta0777 · 2021-12-10T05:20:06Z

@bvaughn I know this problem and I identified it and already fix this in the #22868 . The problem is why it fails is it waits for selector with exact span.Value__tNzum the span.Value part is correct but the next part __tNzum is the hash which is generally different on different OS. But I changed it to span[class^=Value] A regex which matches the span with className starts with Value.

akgupta0777 · 2021-12-10T05:28:45Z

Let me know if you need some help.

Builds on top of the existing Playwright tests to plug in the test selector API: https://gist.github.com/bvaughn/d3c8b8842faf2ac2439bb11773a19cec My goals in doing this are to... 1. Experiment with the new API to see what works and what doesn't. 2. Add some test selector attributes (and remove DOM-structure based selectors). 3. Focus the tests on DevTools itself (rather than the test app). I also took this opportunity to add a few new test cases– like named hooks, editable props, component search, and profiling- just to play around more with the Playwright API. Relates to issue #22646

bvaughn · 2021-12-22T21:19:22Z

Once #23019 lands, we can close this issue out. (It will run the new e2e tests we have in Circle CI automatically with each new PR.)

Remyelder99 · 2021-12-23T01:27:22Z

Yes

Builds on top of the existing Playwright tests to plug in the test selector API: https://gist.github.com/bvaughn/d3c8b8842faf2ac2439bb11773a19cec My goals in doing this are to... 1. Experiment with the new API to see what works and what doesn't. 2. Add some test selector attributes (and remove DOM-structure based selectors). 3. Focus the tests on DevTools itself (rather than the test app). I also took this opportunity to add a few new test cases– like named hooks, editable props, component search, and profiling- just to play around more with the Playwright API. Relates to issue facebook#22646

bvaughn added Component: Build Infrastructure Component: Developer Tools labels Oct 28, 2021

akgupta0777 mentioned this issue Nov 13, 2021

Proof of Concept for E2E tests using playwright #22754

Merged

akgupta0777 mentioned this issue Dec 7, 2021

Added E2E test for ParseHookNames #22868

Closed

bvaughn mentioned this issue Dec 18, 2021

Use ReactDOM Test Selector API in DevTools e2e tests #22978

Merged

bvaughn mentioned this issue Dec 22, 2021

Circle CI: Run DevTools Playwright e2e tests #23019

Merged

bvaughn self-assigned this Dec 22, 2021

bvaughn closed this as completed in #23019 Jan 4, 2022

Add e2e tests for inline package #22646

Add e2e tests for inline package #22646

Comments

bvaughn commented Oct 28, 2021

akgupta0777 commented Oct 31, 2021

Biki-das commented Nov 1, 2021

akgupta0777 commented Nov 1, 2021

Biki-das commented Nov 1, 2021

akgupta0777 commented Nov 1, 2021 • edited Loading

Biki-das commented Nov 1, 2021 • edited Loading

akgupta0777 commented Nov 1, 2021

Biki-das commented Nov 1, 2021

bvaughn commented Nov 1, 2021

Biki-das commented Nov 1, 2021

bvaughn commented Nov 1, 2021

akgupta0777 commented Nov 1, 2021

bvaughn commented Nov 1, 2021

Biki-das commented Nov 1, 2021

bvaughn commented Nov 1, 2021

Biki-das commented Nov 1, 2021

akgupta0777 commented Nov 2, 2021

bvaughn commented Nov 2, 2021

akgupta0777 commented Nov 2, 2021

akgupta0777 commented Nov 3, 2021

bvaughn commented Nov 3, 2021

akgupta0777 commented Nov 4, 2021

bvaughn commented Nov 4, 2021

akgupta0777 commented Nov 4, 2021

bvaughn commented Nov 4, 2021

bvaughn commented Nov 4, 2021

akgupta0777 commented Nov 4, 2021

akgupta0777 commented Nov 13, 2021

bvaughn commented Nov 15, 2021 • edited Loading

akgupta0777 commented Nov 17, 2021 • edited Loading

bvaughn commented Nov 17, 2021 • edited Loading

akgupta0777 commented Nov 17, 2021

bvaughn commented Nov 17, 2021

akgupta0777 commented Nov 18, 2021

akgupta0777 commented Nov 18, 2021

bvaughn commented Nov 18, 2021

akgupta0777 commented Nov 18, 2021

bvaughn commented Dec 9, 2021

akgupta0777 commented Dec 10, 2021 • edited Loading

akgupta0777 commented Dec 10, 2021

bvaughn commented Dec 22, 2021

Remyelder99 commented Dec 23, 2021

akgupta0777 commented Nov 1, 2021 •

edited

Loading

Biki-das commented Nov 1, 2021 •

edited

Loading

bvaughn commented Nov 15, 2021 •

edited

Loading

akgupta0777 commented Nov 17, 2021 •

edited

Loading

bvaughn commented Nov 17, 2021 •

edited

Loading

akgupta0777 commented Dec 10, 2021 •

edited

Loading