Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make navigation editor end-to-end tests more robust #28936

Merged
merged 2 commits into from Feb 11, 2021

Conversation

talldan
Copy link
Contributor

@talldan talldan commented Feb 11, 2021

Description

Fixes #28903

Both navigation editor tests have been failing intermittently.

I haven't completely worked out why! The most common failure was on this line:

await page.waitForXPath( '//h2[contains(., "No menus available")]' );

Changing this to seems to fix things

await page.waitForXPath( '//h2[contains(., "No menus available")]', {
	visible: true,
} );

I went from some failures to none locally. I had a quick look at puppeteer's code and it seems that using the visible or hidden options switches the wait from using a MutationObserver to requestAnimationFrame polling. It might be that there's an underlying issue in puppeteer when waiting for innerText to change using a MutationObserver, but I couldn't see what.

I added in a few other waitForSelector calls to fix a second issue, where the snapshot is sometimes tested before before any blocks have been created.

How has this been tested?

Ran the tests a bunch of times locally

Types of changes

Bug fix (non-breaking change which fixes an issue)

@talldan talldan added [Type] Automated Testing Testing infrastructure changes impacting the execution of end-to-end (E2E) and/or unit tests. [Feature] Navigation Screen labels Feb 11, 2021
@talldan talldan self-assigned this Feb 11, 2021
@github-actions
Copy link

Size Change: 0 B

Total Size: 1.37 MB

ℹ️ View Unchanged
Filename Size Change
build/a11y/index.js 1.14 kB 0 B
build/annotations/index.js 3.77 kB 0 B
build/api-fetch/index.js 3.4 kB 0 B
build/autop/index.js 2.84 kB 0 B
build/blob/index.js 665 B 0 B
build/block-directory/index.js 9.1 kB 0 B
build/block-directory/style-rtl.css 1.01 kB 0 B
build/block-directory/style.css 1.01 kB 0 B
build/block-editor/index.js 123 kB 0 B
build/block-editor/style-rtl.css 12.1 kB 0 B
build/block-editor/style.css 12.1 kB 0 B
build/block-library/blocks/archives/editor-rtl.css 61 B 0 B
build/block-library/blocks/archives/editor.css 60 B 0 B
build/block-library/blocks/audio/editor-rtl.css 58 B 0 B
build/block-library/blocks/audio/editor.css 58 B 0 B
build/block-library/blocks/audio/style-rtl.css 103 B 0 B
build/block-library/blocks/audio/style.css 103 B 0 B
build/block-library/blocks/block/editor-rtl.css 161 B 0 B
build/block-library/blocks/block/editor.css 161 B 0 B
build/block-library/blocks/button/editor-rtl.css 475 B 0 B
build/block-library/blocks/button/editor.css 474 B 0 B
build/block-library/blocks/button/style-rtl.css 465 B 0 B
build/block-library/blocks/button/style.css 464 B 0 B
build/block-library/blocks/buttons/editor-rtl.css 233 B 0 B
build/block-library/blocks/buttons/editor.css 233 B 0 B
build/block-library/blocks/buttons/style-rtl.css 303 B 0 B
build/block-library/blocks/buttons/style.css 303 B 0 B
build/block-library/blocks/calendar/style-rtl.css 208 B 0 B
build/block-library/blocks/calendar/style.css 208 B 0 B
build/block-library/blocks/categories/editor-rtl.css 84 B 0 B
build/block-library/blocks/categories/editor.css 83 B 0 B
build/block-library/blocks/categories/style-rtl.css 79 B 0 B
build/block-library/blocks/categories/style.css 79 B 0 B
build/block-library/blocks/code/style-rtl.css 90 B 0 B
build/block-library/blocks/code/style.css 90 B 0 B
build/block-library/blocks/columns/editor-rtl.css 190 B 0 B
build/block-library/blocks/columns/editor.css 190 B 0 B
build/block-library/blocks/columns/style-rtl.css 421 B 0 B
build/block-library/blocks/columns/style.css 421 B 0 B
build/block-library/blocks/cover/editor-rtl.css 390 B 0 B
build/block-library/blocks/cover/editor.css 389 B 0 B
build/block-library/blocks/cover/style-rtl.css 1.25 kB 0 B
build/block-library/blocks/cover/style.css 1.25 kB 0 B
build/block-library/blocks/embed/editor-rtl.css 486 B 0 B
build/block-library/blocks/embed/editor.css 486 B 0 B
build/block-library/blocks/embed/style-rtl.css 396 B 0 B
build/block-library/blocks/embed/style.css 395 B 0 B
build/block-library/blocks/file/editor-rtl.css 199 B 0 B
build/block-library/blocks/file/editor.css 198 B 0 B
build/block-library/blocks/file/style-rtl.css 248 B 0 B
build/block-library/blocks/file/style.css 248 B 0 B
build/block-library/blocks/freeform/editor-rtl.css 2.45 kB 0 B
build/block-library/blocks/freeform/editor.css 2.45 kB 0 B
build/block-library/blocks/gallery/editor-rtl.css 679 B 0 B
build/block-library/blocks/gallery/editor.css 679 B 0 B
build/block-library/blocks/gallery/style-rtl.css 1.07 kB 0 B
build/block-library/blocks/gallery/style.css 1.06 kB 0 B
build/block-library/blocks/group/editor-rtl.css 318 B 0 B
build/block-library/blocks/group/editor.css 317 B 0 B
build/block-library/blocks/group/style-rtl.css 57 B 0 B
build/block-library/blocks/group/style.css 57 B 0 B
build/block-library/blocks/heading/editor-rtl.css 129 B 0 B
build/block-library/blocks/heading/editor.css 129 B 0 B
build/block-library/blocks/heading/style-rtl.css 76 B 0 B
build/block-library/blocks/heading/style.css 76 B 0 B
build/block-library/blocks/html/editor-rtl.css 281 B 0 B
build/block-library/blocks/html/editor.css 281 B 0 B
build/block-library/blocks/image/editor-rtl.css 717 B 0 B
build/block-library/blocks/image/editor.css 716 B 0 B
build/block-library/blocks/image/style-rtl.css 477 B 0 B
build/block-library/blocks/image/style.css 478 B 0 B
build/block-library/blocks/latest-comments/editor-rtl.css 159 B 0 B
build/block-library/blocks/latest-comments/editor.css 158 B 0 B
build/block-library/blocks/latest-comments/style-rtl.css 269 B 0 B
build/block-library/blocks/latest-comments/style.css 269 B 0 B
build/block-library/blocks/latest-posts/editor-rtl.css 137 B 0 B
build/block-library/blocks/latest-posts/editor.css 137 B 0 B
build/block-library/blocks/latest-posts/style-rtl.css 523 B 0 B
build/block-library/blocks/latest-posts/style.css 522 B 0 B
build/block-library/blocks/list/editor-rtl.css 65 B 0 B
build/block-library/blocks/list/editor.css 65 B 0 B
build/block-library/blocks/list/style-rtl.css 63 B 0 B
build/block-library/blocks/list/style.css 63 B 0 B
build/block-library/blocks/media-text/editor-rtl.css 191 B 0 B
build/block-library/blocks/media-text/editor.css 191 B 0 B
build/block-library/blocks/media-text/style-rtl.css 535 B 0 B
build/block-library/blocks/media-text/style.css 532 B 0 B
build/block-library/blocks/more/editor-rtl.css 434 B 0 B
build/block-library/blocks/more/editor.css 434 B 0 B
build/block-library/blocks/navigation-link/editor-rtl.css 395 B 0 B
build/block-library/blocks/navigation-link/editor.css 397 B 0 B
build/block-library/blocks/navigation-link/style-rtl.css 704 B 0 B
build/block-library/blocks/navigation-link/style.css 702 B 0 B
build/block-library/blocks/navigation/editor-rtl.css 1.34 kB 0 B
build/block-library/blocks/navigation/editor.css 1.34 kB 0 B
build/block-library/blocks/navigation/style-rtl.css 195 B 0 B
build/block-library/blocks/navigation/style.css 195 B 0 B
build/block-library/blocks/nextpage/editor-rtl.css 395 B 0 B
build/block-library/blocks/nextpage/editor.css 395 B 0 B
build/block-library/blocks/page-list/editor-rtl.css 214 B 0 B
build/block-library/blocks/page-list/editor.css 214 B 0 B
build/block-library/blocks/page-list/style-rtl.css 527 B 0 B
build/block-library/blocks/page-list/style.css 526 B 0 B
build/block-library/blocks/paragraph/editor-rtl.css 109 B 0 B
build/block-library/blocks/paragraph/editor.css 109 B 0 B
build/block-library/blocks/paragraph/style-rtl.css 273 B 0 B
build/block-library/blocks/paragraph/style.css 273 B 0 B
build/block-library/blocks/post-author/editor-rtl.css 209 B 0 B
build/block-library/blocks/post-author/editor.css 209 B 0 B
build/block-library/blocks/post-author/style-rtl.css 183 B 0 B
build/block-library/blocks/post-author/style.css 184 B 0 B
build/block-library/blocks/post-comments-form/style-rtl.css 250 B 0 B
build/block-library/blocks/post-comments-form/style.css 250 B 0 B
build/block-library/blocks/post-content/editor-rtl.css 139 B 0 B
build/block-library/blocks/post-content/editor.css 139 B 0 B
build/block-library/blocks/post-excerpt/editor-rtl.css 73 B 0 B
build/block-library/blocks/post-excerpt/editor.css 73 B 0 B
build/block-library/blocks/post-featured-image/editor-rtl.css 338 B 0 B
build/block-library/blocks/post-featured-image/editor.css 338 B 0 B
build/block-library/blocks/post-featured-image/style-rtl.css 100 B 0 B
build/block-library/blocks/post-featured-image/style.css 100 B 0 B
build/block-library/blocks/preformatted/style-rtl.css 63 B 0 B
build/block-library/blocks/preformatted/style.css 63 B 0 B
build/block-library/blocks/pullquote/editor-rtl.css 183 B 0 B
build/block-library/blocks/pullquote/editor.css 183 B 0 B
build/block-library/blocks/pullquote/style-rtl.css 316 B 0 B
build/block-library/blocks/pullquote/style.css 316 B 0 B
build/block-library/blocks/query-loop/editor-rtl.css 90 B 0 B
build/block-library/blocks/query-loop/editor.css 89 B 0 B
build/block-library/blocks/query-loop/style-rtl.css 315 B 0 B
build/block-library/blocks/query-loop/style.css 317 B 0 B
build/block-library/blocks/query-pagination-numbers/editor-rtl.css 122 B 0 B
build/block-library/blocks/query-pagination-numbers/editor.css 121 B 0 B
build/block-library/blocks/query-pagination/editor-rtl.css 270 B 0 B
build/block-library/blocks/query-pagination/editor.css 262 B 0 B
build/block-library/blocks/query-pagination/style-rtl.css 168 B 0 B
build/block-library/blocks/query-pagination/style.css 168 B 0 B
build/block-library/blocks/query/editor-rtl.css 159 B 0 B
build/block-library/blocks/query/editor.css 160 B 0 B
build/block-library/blocks/quote/editor-rtl.css 61 B 0 B
build/block-library/blocks/quote/editor.css 61 B 0 B
build/block-library/blocks/quote/style-rtl.css 169 B 0 B
build/block-library/blocks/quote/style.css 169 B 0 B
build/block-library/blocks/rss/editor-rtl.css 201 B 0 B
build/block-library/blocks/rss/editor.css 202 B 0 B
build/block-library/blocks/rss/style-rtl.css 290 B 0 B
build/block-library/blocks/rss/style.css 290 B 0 B
build/block-library/blocks/search/editor-rtl.css 165 B 0 B
build/block-library/blocks/search/editor.css 165 B 0 B
build/block-library/blocks/search/style-rtl.css 342 B 0 B
build/block-library/blocks/search/style.css 344 B 0 B
build/block-library/blocks/separator/editor-rtl.css 99 B 0 B
build/block-library/blocks/separator/editor.css 99 B 0 B
build/block-library/blocks/separator/style-rtl.css 236 B 0 B
build/block-library/blocks/separator/style.css 236 B 0 B
build/block-library/blocks/shortcode/editor-rtl.css 504 B 0 B
build/block-library/blocks/shortcode/editor.css 504 B 0 B
build/block-library/blocks/site-logo/editor-rtl.css 201 B 0 B
build/block-library/blocks/site-logo/editor.css 201 B 0 B
build/block-library/blocks/site-logo/style-rtl.css 117 B 0 B
build/block-library/blocks/site-logo/style.css 117 B 0 B
build/block-library/blocks/social-link/editor-rtl.css 164 B 0 B
build/block-library/blocks/social-link/editor.css 165 B 0 B
build/block-library/blocks/social-links/editor-rtl.css 696 B 0 B
build/block-library/blocks/social-links/editor.css 696 B 0 B
build/block-library/blocks/social-links/style-rtl.css 1.37 kB 0 B
build/block-library/blocks/social-links/style.css 1.37 kB 0 B
build/block-library/blocks/spacer/editor-rtl.css 302 B 0 B
build/block-library/blocks/spacer/editor.css 302 B 0 B
build/block-library/blocks/spacer/style-rtl.css 48 B 0 B
build/block-library/blocks/spacer/style.css 48 B 0 B
build/block-library/blocks/subhead/editor-rtl.css 99 B 0 B
build/block-library/blocks/subhead/editor.css 99 B 0 B
build/block-library/blocks/subhead/style-rtl.css 80 B 0 B
build/block-library/blocks/subhead/style.css 80 B 0 B
build/block-library/blocks/table/editor-rtl.css 489 B 0 B
build/block-library/blocks/table/editor.css 489 B 0 B
build/block-library/blocks/table/style-rtl.css 386 B 0 B
build/block-library/blocks/table/style.css 386 B 0 B
build/block-library/blocks/tag-cloud/editor-rtl.css 118 B 0 B
build/block-library/blocks/tag-cloud/editor.css 118 B 0 B
build/block-library/blocks/tag-cloud/style-rtl.css 94 B 0 B
build/block-library/blocks/tag-cloud/style.css 94 B 0 B
build/block-library/blocks/template-part/editor-rtl.css 557 B 0 B
build/block-library/blocks/template-part/editor.css 556 B 0 B
build/block-library/blocks/text-columns/editor-rtl.css 95 B 0 B
build/block-library/blocks/text-columns/editor.css 95 B 0 B
build/block-library/blocks/text-columns/style-rtl.css 166 B 0 B
build/block-library/blocks/text-columns/style.css 166 B 0 B
build/block-library/blocks/verse/editor-rtl.css 62 B 0 B
build/block-library/blocks/verse/editor.css 62 B 0 B
build/block-library/blocks/verse/style-rtl.css 87 B 0 B
build/block-library/blocks/verse/style.css 87 B 0 B
build/block-library/blocks/video/editor-rtl.css 504 B 0 B
build/block-library/blocks/video/editor.css 503 B 0 B
build/block-library/blocks/video/style-rtl.css 193 B 0 B
build/block-library/blocks/video/style.css 193 B 0 B
build/block-library/common-rtl.css 1.01 kB 0 B
build/block-library/common.css 1.01 kB 0 B
build/block-library/editor-rtl.css 9.04 kB 0 B
build/block-library/editor.css 9.03 kB 0 B
build/block-library/index.js 145 kB 0 B
build/block-library/style-rtl.css 8.8 kB 0 B
build/block-library/style.css 8.8 kB 0 B
build/block-library/theme-rtl.css 748 B 0 B
build/block-library/theme.css 748 B 0 B
build/block-serialization-default-parser/index.js 1.88 kB 0 B
build/block-serialization-spec-parser/index.js 3.06 kB 0 B
build/blocks/index.js 48.2 kB 0 B
build/components/index.js 270 kB 0 B
build/components/style-rtl.css 15.5 kB 0 B
build/components/style.css 15.5 kB 0 B
build/compose/index.js 11 kB 0 B
build/core-data/index.js 16.8 kB 0 B
build/customize-widgets/index.js 4.08 kB 0 B
build/customize-widgets/style-rtl.css 168 B 0 B
build/customize-widgets/style.css 168 B 0 B
build/data-controls/index.js 827 B 0 B
build/data/index.js 8.86 kB 0 B
build/date/index.js 31.8 kB 0 B
build/deprecated/index.js 769 B 0 B
build/dom-ready/index.js 570 B 0 B
build/dom/index.js 4.93 kB 0 B
build/edit-navigation/index.js 10.5 kB 0 B
build/edit-navigation/style-rtl.css 1.11 kB 0 B
build/edit-navigation/style.css 1.11 kB 0 B
build/edit-post/index.js 307 kB 0 B
build/edit-post/style-rtl.css 6.79 kB 0 B
build/edit-post/style.css 6.78 kB 0 B
build/edit-site/index.js 24.8 kB 0 B
build/edit-site/style-rtl.css 4.04 kB 0 B
build/edit-site/style.css 4.04 kB 0 B
build/edit-widgets/index.js 20 kB 0 B
build/edit-widgets/style-rtl.css 3.2 kB 0 B
build/edit-widgets/style.css 3.2 kB 0 B
build/editor/editor-styles-rtl.css 543 B 0 B
build/editor/editor-styles.css 545 B 0 B
build/editor/index.js 41.8 kB 0 B
build/editor/style-rtl.css 3.89 kB 0 B
build/editor/style.css 3.89 kB 0 B
build/element/index.js 4.61 kB 0 B
build/escape-html/index.js 735 B 0 B
build/format-library/index.js 6.76 kB 0 B
build/format-library/style-rtl.css 637 B 0 B
build/format-library/style.css 639 B 0 B
build/hooks/index.js 2.28 kB 0 B
build/html-entities/index.js 622 B 0 B
build/i18n/index.js 4.01 kB 0 B
build/is-shallow-equal/index.js 699 B 0 B
build/keyboard-shortcuts/index.js 2.53 kB 0 B
build/keycodes/index.js 1.93 kB 0 B
build/list-reusable-blocks/index.js 3.15 kB 0 B
build/list-reusable-blocks/style-rtl.css 629 B 0 B
build/list-reusable-blocks/style.css 628 B 0 B
build/media-utils/index.js 5.35 kB 0 B
build/notices/index.js 1.85 kB 0 B
build/nux/index.js 3.41 kB 0 B
build/nux/style-rtl.css 731 B 0 B
build/nux/style.css 727 B 0 B
build/plugins/index.js 2.54 kB 0 B
build/primitives/index.js 1.42 kB 0 B
build/priority-queue/index.js 791 B 0 B
build/redux-routine/index.js 2.84 kB 0 B
build/reusable-blocks/index.js 2.92 kB 0 B
build/rich-text/index.js 13.4 kB 0 B
build/server-side-render/index.js 2.77 kB 0 B
build/shortcode/index.js 1.7 kB 0 B
build/token-list/index.js 1.27 kB 0 B
build/url/index.js 3.01 kB 0 B
build/viewport/index.js 1.86 kB 0 B
build/warning/index.js 1.14 kB 0 B
build/wordcount/index.js 1.22 kB 0 B

compressed-size-action

Copy link
Contributor

@youknowriad youknowriad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. The more stable the test is, the better.

@talldan talldan merged commit 5401bf2 into master Feb 11, 2021
@talldan talldan deleted the fix/navigation-editor-tests branch February 11, 2021 09:27
@github-actions github-actions bot added this to the Gutenberg 10.1 milestone Feb 11, 2021
noisysocks pushed a commit that referenced this pull request Mar 1, 2021
* Wait for loading to finish in e2e test

* Simplify and add some more assertions
noisysocks pushed a commit that referenced this pull request Mar 1, 2021
* Wait for loading to finish in e2e test

* Simplify and add some more assertions
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
[Type] Automated Testing Testing infrastructure changes impacting the execution of end-to-end (E2E) and/or unit tests.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

"Navigation editor > allows creation of a menu" e2e test failing inconsistently
2 participants