Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix intermittent transform e2e test failures #18662

Merged
merged 1 commit into from Nov 26, 2019
Merged

Conversation

@ellatrix
Copy link
Member

ellatrix commented Nov 21, 2019

Description

Creating a new post for every this seems to resolve this. All other tests start from a new post as well. I restarted the build a few times and the transform e2e tests didn't fail. There's some other intermittent failures that should be resolved separately.

How has this been tested?

Screenshots

Types of changes

Checklist:

  • My code is tested.
  • My code follows the WordPress code style.
  • My code follows the accessibility standards.
  • My code has proper inline documentation.
  • I've included developer documentation if appropriate.
  • I've updated all React Native files affected by any refactorings/renamings in this PR. .
beforeEach( async () => {
await setPostContent( '' );
await page.click( '.editor-post-title .editor-post-title__block' );
await createNewPost();

This comment has been minimized.

Copy link
@gziolo

gziolo Nov 21, 2019

Member

It might slow down the execution of those tests. It needs to be confirmed whether the total execution time on Travis doesn't increase.

This comment has been minimized.

Copy link
@ellatrix

ellatrix Nov 21, 2019

Author Member

It will be slower for sure. But slower is better than failing all the time?

This comment has been minimized.

Copy link
@ellatrix

ellatrix Nov 21, 2019

Author Member

Most other tests start from a new post as well, that's nothing new.

This comment has been minimized.

Copy link
@aduth

aduth Nov 21, 2019

Member

I agree, it may be slower than it was previously, but this is a pattern we adopt in many other tests.

I would actually very much like to see an improvement to the implementation of createNewPost where we try to artificially reset the state of the editor to an initial state, rather than perform a full reload. But that can be considered a separate task.

This comment has been minimized.

Copy link
@gziolo

gziolo Nov 21, 2019

Member

This one is quite unique, this test suite alone runs for 5 minutes on Travis.

This comment has been minimized.

Copy link
@jorgefilipecosta

jorgefilipecosta Nov 25, 2019

Member

I think these tests, proved useful and caught some bugs with transforms in the past. But now transforms are mostly stable and rarely change and given the complications, they are giving us I guess we can remove them and then include some specific tests for some transforms.

This comment has been minimized.

Copy link
@jorgefilipecosta

jorgefilipecosta Nov 25, 2019

Member

@ellatrix, would you prefer to update this PR to remove the tests instead, or I can submit a different PR that removes them?

This comment has been minimized.

Copy link
@ellatrix

ellatrix Nov 25, 2019

Author Member

Ok, let's remove them then.

This comment has been minimized.

Copy link
@ellatrix

ellatrix Nov 25, 2019

Author Member

@jorgefilipecosta I removed them in this PR.

This comment has been minimized.

Copy link
@gziolo

gziolo Nov 26, 2019

Member

But now transforms are mostly stable and rarely change and given the complications, they are giving us I guess we can remove them and then include some specific tests for some transforms.

Yes, we definitely want to keep them but for a limited set of blocks 👍 Let's make sure we add a test suite in a follow-up PR. We should have at least text blocks (List, Quote, Paragraph), media blocks (Image, Gallery, Cover) and the Group block covered.

@jorgefilipecosta thinking about it a bit more, would it be possible to recreate this test suite by creating 3 sections (text, media, group) and listing fixtures for the source of blocks and run checks on the expected snapshots?

@ellatrix ellatrix changed the title [Testing on Travis] Fix intermittent transform e2e test failures Nov 21, 2019
const insertButton = ( await page.$x(
`//button//span[contains(text(), '${ name }')]`
) )[ 0 ];
await insertButton.evaluate( ( element ) => element.scrollIntoView() );

This comment has been minimized.

Copy link
@aduth

aduth Nov 25, 2019

Member

It's a bit surprising to me that it's necessary, given that the default implementation of click should be doing the scrolling already on our behalf:

https://github.com/puppeteer/puppeteer/blob/d17708ba1fc8411941bfd07f633aaf2fb5cb671e/lib/JSHandle.js#L278-L285

I wonder if it has to do with the fact that the element is in it's own scrollable region, not the page's default scroll area. If that's the case, seems like an area of improvement for Puppeteer to consider.

Edit: Related:

This comment has been minimized.

Copy link
@ellatrix

ellatrix Nov 25, 2019

Author Member

Yeah, I stumbled upon the same code and the same issues. It's not exactly the same as element.scrollIntoView() will force the element to the top even if it's already in view.

@ellatrix ellatrix force-pushed the try/stable-transform-e2e branch from 8fb489a to 5f5b691 Nov 25, 2019
@ellatrix ellatrix force-pushed the try/stable-transform-e2e branch from 5f5b691 to 3b721bc Nov 25, 2019
@ellatrix ellatrix force-pushed the try/stable-transform-e2e branch from 3b721bc to 7d3bd44 Nov 25, 2019
Copy link
Member

jorgefilipecosta left a comment

LGTM 👍 I will try to follow up with some specific tests in the next few days.

@ellatrix ellatrix merged commit 1b15469 into master Nov 26, 2019
2 checks passed
2 checks passed
pull-request-automation
Details
Travis CI - Pull Request Build Passed
Details
@ellatrix ellatrix deleted the try/stable-transform-e2e branch Nov 26, 2019
@aduth

This comment has been minimized.

Copy link
Member

aduth commented Nov 26, 2019

Noting that I'm still observing an intermittent failure similar to what was intended to be addressed here.

FAIL packages/e2e-tests/specs/editor/blocks/quote.test.js (90.679s)
  ● Quote › can be created by converting a paragraph
    TimeoutError: waiting for selector ".block-editor-block-list__block[aria-label="Block: Quote"]" failed: timeout 30000ms exceeded
      at new WaitTask (../../node_modules/puppeteer/lib/DOMWorld.js:549:28)
      at DOMWorld._waitForSelectorOrXPath (../../node_modules/puppeteer/lib/DOMWorld.js:478:22)
      at DOMWorld.waitForSelector (../../node_modules/puppeteer/lib/DOMWorld.js:432:17)
      at Frame.waitForSelector (../../node_modules/puppeteer/lib/FrameManager.js:627:47)
      at Frame.<anonymous> (../../node_modules/puppeteer/lib/helper.js:112:23)
      at Page.waitForSelector (../../node_modules/puppeteer/lib/Page.js:1125:29)
      at page (../e2e-test-utils/build/@wordpress/e2e-test-utils/src/transform-block-to.js:18:8)
      at tryCatch (../../node_modules/regenerator-runtime/runtime.js:45:40)
      at Generator.invoke [as _invoke] (../../node_modules/regenerator-runtime/runtime.js:271:22)
      at Generator.prototype.<computed> [as next] (../../node_modules/regenerator-runtime/runtime.js:97:21)
        -- ASYNC --
      at Frame.<anonymous> (../../node_modules/puppeteer/lib/helper.js:111:15)
      at Page.waitForSelector (../../node_modules/puppeteer/lib/Page.js:1125:29)
      at page (../e2e-test-utils/build/@wordpress/e2e-test-utils/src/transform-block-to.js:18:8)
      at tryCatch (../../node_modules/regenerator-runtime/runtime.js:45:40)
      at Generator.invoke [as _invoke] (../../node_modules/regenerator-runtime/runtime.js:271:22)
      at Generator.prototype.<computed> [as next] (../../node_modules/regenerator-runtime/runtime.js:97:21)
      at asyncGeneratorStep (../../node_modules/@babel/runtime/helpers/asyncToGenerator.js:3:24)
      at _next (../../node_modules/@babel/runtime/helpers/asyncToGenerator.js:25:9)
          at runMicrotasks (<anonymous>)

At #18753, after a rebase to include revisions from this pull request.

@ellatrix

This comment has been minimized.

Copy link
Member Author

ellatrix commented Nov 26, 2019

@aduth Yeah that's odd. Not sure what is causing it. I'm also not immediately sure how to debug.

@youknowriad youknowriad added this to the Gutenberg 7.1 milestone Dec 9, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.