Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

E2E Tests: Verify that we can publish even if some changes haven't been saved #37408

Draft
wants to merge 2 commits into
base: trunk
Choose a base branch
from

Conversation

ockham
Copy link
Contributor

@ockham ockham commented Dec 15, 2021

Description

Based on #36096 (comment). Provides test coverage for the issue described at #36096 (comment), which has a fix at #37383.

Note that this is still a WIP: The test is currently not failing, even though the tested functionality is.

For background, see discussion starting at #36096 (comment).

How has this been tested?

npm run test-e2e -- packages/e2e-tests/specs/site-editor/multi-entity-saving.test.js

Types of changes

Code quality

@ockham
Copy link
Contributor Author

ockham commented Dec 15, 2021

Current test output (snippet):

> gutenberg@12.1.0 test-e2e src/gutenberg
> wp-scripts test-e2e --config packages/e2e-tests/jest.config.js "packages/e2e-tests/specs/site-editor/multi-entity-saving.test.js"

Chromium is already in src/gutenberg/node_modules/puppeteer-core/.local-chromium/mac-901912; skipping download.
[Error: TypeError: t.props.setEntitiesSavedStatesCallback is not a function
    at http://localhost:8889/wp-content/plugins/gutenberg/build/editor/index.min.js?ver=7186e4882f68b3d6c56f3ddaf14409d6:15:2966
    at Object.vi (http://localhost:8889/wp-content/plugins/gutenberg/vendor/react-dom.min.edd8b7a6.js:202:330)
    at ui (http://localhost:8889/wp-content/plugins/gutenberg/vendor/react-dom.min.edd8b7a6.js:32:27)
    at xi (http://localhost:8889/wp-content/plugins/gutenberg/vendor/react-dom.min.edd8b7a6.js:32:81)
    at zg (http://localhost:8889/wp-content/plugins/gutenberg/vendor/react-dom.min.edd8b7a6.js:55:403)
    at rg (http://localhost:8889/wp-content/plugins/gutenberg/vendor/react-dom.min.edd8b7a6.js:56:317)
    at http://localhost:8889/wp-content/plugins/gutenberg/vendor/react-dom.min.edd8b7a6.js:66:168
    at http://localhost:8889/wp-content/plugins/gutenberg/vendor/react-dom.min.edd8b7a6.js:240:292
    at ti (http://localhost:8889/wp-content/plugins/gutenberg/vendor/react-dom.min.edd8b7a6.js:30:406)
    at hg (http://localhost:8889/wp-content/plugins/gutenberg/vendor/react-dom.min.edd8b7a6.js:59:148)]
 FAIL  packages/e2e-tests/specs/site-editor/multi-entity-saving.test.js (99.286 s)
  Multi-entity save flow
    Post Editor
      ✕ Save flow should work as expected. (41029 ms)
      ✓ Site blocks should save individually (11108 ms)
      ✓ Can publish even if some changes haven't been saved (10173 ms)
    Site Editor
      ✓ Save flow should work as expected (6586 ms)
      ✓ Save flow should allow re-saving after changing the same block attribute (7594 ms)

Note that the TypeError shows up in the console, but doesn't cause the "Can publish even if some changes haven't been saved" test to fail. (The "Save flow should work as expected." failure is unrelated.)

This, along some possible strategies to make it fail the test, is discussed over at #36096 (comment)

@github-actions
Copy link

github-actions bot commented Dec 15, 2021

Size Change: +2.17 kB (0%)

Total Size: 1.13 MB

Filename Size Change
build/block-editor/index.min.js 140 kB +310 B (0%)
build/block-editor/style-rtl.css 14.6 kB +63 B (0%)
build/block-editor/style.css 14.6 kB +62 B (0%)
build/block-library/blocks/columns/style-rtl.css 502 B -1 B (0%)
build/block-library/blocks/columns/style.css 501 B -1 B (0%)
build/block-library/blocks/comments-pagination/style-rtl.css 242 B +7 B (+3%)
build/block-library/blocks/comments-pagination/style.css 237 B +6 B (+3%)
build/block-library/blocks/navigation/editor-rtl.css 1.91 kB +4 B (0%)
build/block-library/blocks/navigation/editor.css 1.92 kB +8 B (0%)
build/block-library/blocks/navigation/style-rtl.css 1.8 kB +96 B (+6%) 🔍
build/block-library/blocks/navigation/style.css 1.8 kB +97 B (+6%) 🔍
build/block-library/blocks/navigation/view.min.js 2.82 kB +29 B (+1%)
build/block-library/blocks/site-logo/editor-rtl.css 744 B -28 B (-4%)
build/block-library/blocks/site-logo/editor.css 744 B -28 B (-4%)
build/block-library/blocks/site-logo/style-rtl.css 181 B +16 B (+10%) ⚠️
build/block-library/blocks/site-logo/style.css 181 B +16 B (+10%) ⚠️
build/block-library/editor-rtl.css 10 kB -15 B (0%)
build/block-library/editor.css 10 kB -16 B (0%)
build/block-library/index.min.js 164 kB +1.11 kB (+1%)
build/block-library/style-rtl.css 10.9 kB +111 B (+1%)
build/block-library/style.css 10.9 kB +112 B (+1%)
build/block-library/theme-rtl.css 675 B +3 B (0%)
build/block-library/theme.css 679 B +2 B (0%)
build/components/index.min.js 215 kB -68 B (0%)
build/components/style-rtl.css 15.5 kB +44 B (0%)
build/components/style.css 15.5 kB +44 B (0%)
build/compose/index.min.js 11.2 kB +247 B (+2%)
build/edit-post/index.min.js 29.4 kB +4 B (0%)
build/edit-post/style-rtl.css 7.13 kB +35 B (0%)
build/edit-post/style.css 7.13 kB +35 B (0%)
build/edit-site/index.min.js 35.6 kB +22 B (0%)
build/edit-site/style-rtl.css 6.61 kB -28 B (0%)
build/edit-site/style.css 6.6 kB -29 B (0%)
build/edit-widgets/index.min.js 16.5 kB -2 B (0%)
build/editor/index.min.js 37.9 kB -33 B (0%)
build/editor/style-rtl.css 3.75 kB -32 B (-1%)
build/editor/style.css 3.74 kB -29 B (-1%)
build/redux-routine/index.min.js 2.65 kB +1 B (0%)
ℹ️ View Unchanged
Filename Size
build/a11y/index.min.js 960 B
build/admin-manifest/index.min.js 1.1 kB
build/annotations/index.min.js 2.75 kB
build/api-fetch/index.min.js 2.21 kB
build/autop/index.min.js 2.12 kB
build/blob/index.min.js 459 B
build/block-directory/index.min.js 6.28 kB
build/block-directory/style-rtl.css 1.01 kB
build/block-directory/style.css 1.01 kB
build/block-editor/default-editor-styles-rtl.css 378 B
build/block-editor/default-editor-styles.css 378 B
build/block-library/blocks/archives/editor-rtl.css 61 B
build/block-library/blocks/archives/editor.css 60 B
build/block-library/blocks/archives/style-rtl.css 65 B
build/block-library/blocks/archives/style.css 65 B
build/block-library/blocks/audio/editor-rtl.css 58 B
build/block-library/blocks/audio/editor.css 58 B
build/block-library/blocks/audio/style-rtl.css 111 B
build/block-library/blocks/audio/style.css 111 B
build/block-library/blocks/audio/theme-rtl.css 125 B
build/block-library/blocks/audio/theme.css 125 B
build/block-library/blocks/block/editor-rtl.css 161 B
build/block-library/blocks/block/editor.css 161 B
build/block-library/blocks/button/editor-rtl.css 470 B
build/block-library/blocks/button/editor.css 470 B
build/block-library/blocks/button/style-rtl.css 560 B
build/block-library/blocks/button/style.css 560 B
build/block-library/blocks/buttons/editor-rtl.css 291 B
build/block-library/blocks/buttons/editor.css 291 B
build/block-library/blocks/buttons/style-rtl.css 275 B
build/block-library/blocks/buttons/style.css 275 B
build/block-library/blocks/calendar/style-rtl.css 207 B
build/block-library/blocks/calendar/style.css 207 B
build/block-library/blocks/categories/editor-rtl.css 84 B
build/block-library/blocks/categories/editor.css 83 B
build/block-library/blocks/categories/style-rtl.css 79 B
build/block-library/blocks/categories/style.css 79 B
build/block-library/blocks/code/style-rtl.css 90 B
build/block-library/blocks/code/style.css 90 B
build/block-library/blocks/code/theme-rtl.css 134 B
build/block-library/blocks/code/theme.css 134 B
build/block-library/blocks/columns/editor-rtl.css 210 B
build/block-library/blocks/columns/editor.css 208 B
build/block-library/blocks/comment-template/style-rtl.css 127 B
build/block-library/blocks/comment-template/style.css 127 B
build/block-library/blocks/comments-pagination-numbers/editor-rtl.css 123 B
build/block-library/blocks/comments-pagination-numbers/editor.css 121 B
build/block-library/blocks/comments-pagination/editor-rtl.css 222 B
build/block-library/blocks/comments-pagination/editor.css 209 B
build/block-library/blocks/cover/editor-rtl.css 546 B
build/block-library/blocks/cover/editor.css 547 B
build/block-library/blocks/cover/style-rtl.css 1.22 kB
build/block-library/blocks/cover/style.css 1.22 kB
build/block-library/blocks/embed/editor-rtl.css 488 B
build/block-library/blocks/embed/editor.css 488 B
build/block-library/blocks/embed/style-rtl.css 417 B
build/block-library/blocks/embed/style.css 417 B
build/block-library/blocks/embed/theme-rtl.css 124 B
build/block-library/blocks/embed/theme.css 124 B
build/block-library/blocks/file/editor-rtl.css 300 B
build/block-library/blocks/file/editor.css 300 B
build/block-library/blocks/file/style-rtl.css 255 B
build/block-library/blocks/file/style.css 255 B
build/block-library/blocks/file/view.min.js 322 B
build/block-library/blocks/freeform/editor-rtl.css 2.44 kB
build/block-library/blocks/freeform/editor.css 2.44 kB
build/block-library/blocks/gallery/editor-rtl.css 966 B
build/block-library/blocks/gallery/editor.css 970 B
build/block-library/blocks/gallery/style-rtl.css 1.63 kB
build/block-library/blocks/gallery/style.css 1.62 kB
build/block-library/blocks/gallery/theme-rtl.css 122 B
build/block-library/blocks/gallery/theme.css 122 B
build/block-library/blocks/group/editor-rtl.css 159 B
build/block-library/blocks/group/editor.css 159 B
build/block-library/blocks/group/style-rtl.css 57 B
build/block-library/blocks/group/style.css 57 B
build/block-library/blocks/group/theme-rtl.css 78 B
build/block-library/blocks/group/theme.css 78 B
build/block-library/blocks/heading/style-rtl.css 114 B
build/block-library/blocks/heading/style.css 114 B
build/block-library/blocks/html/editor-rtl.css 332 B
build/block-library/blocks/html/editor.css 333 B
build/block-library/blocks/image/editor-rtl.css 810 B
build/block-library/blocks/image/editor.css 809 B
build/block-library/blocks/image/style-rtl.css 507 B
build/block-library/blocks/image/style.css 511 B
build/block-library/blocks/image/theme-rtl.css 124 B
build/block-library/blocks/image/theme.css 124 B
build/block-library/blocks/latest-comments/style-rtl.css 284 B
build/block-library/blocks/latest-comments/style.css 284 B
build/block-library/blocks/latest-posts/editor-rtl.css 137 B
build/block-library/blocks/latest-posts/editor.css 137 B
build/block-library/blocks/latest-posts/style-rtl.css 528 B
build/block-library/blocks/latest-posts/style.css 527 B
build/block-library/blocks/list/style-rtl.css 94 B
build/block-library/blocks/list/style.css 94 B
build/block-library/blocks/media-text/editor-rtl.css 266 B
build/block-library/blocks/media-text/editor.css 263 B
build/block-library/blocks/media-text/style-rtl.css 493 B
build/block-library/blocks/media-text/style.css 490 B
build/block-library/blocks/more/editor-rtl.css 431 B
build/block-library/blocks/more/editor.css 431 B
build/block-library/blocks/navigation-link/editor-rtl.css 649 B
build/block-library/blocks/navigation-link/editor.css 650 B
build/block-library/blocks/navigation-link/style-rtl.css 94 B
build/block-library/blocks/navigation-link/style.css 94 B
build/block-library/blocks/navigation-submenu/editor-rtl.css 299 B
build/block-library/blocks/navigation-submenu/editor.css 299 B
build/block-library/blocks/navigation-submenu/view.min.js 343 B
build/block-library/blocks/nextpage/editor-rtl.css 395 B
build/block-library/blocks/nextpage/editor.css 395 B
build/block-library/blocks/page-list/editor-rtl.css 377 B
build/block-library/blocks/page-list/editor.css 377 B
build/block-library/blocks/page-list/style-rtl.css 172 B
build/block-library/blocks/page-list/style.css 172 B
build/block-library/blocks/paragraph/editor-rtl.css 157 B
build/block-library/blocks/paragraph/editor.css 157 B
build/block-library/blocks/paragraph/style-rtl.css 273 B
build/block-library/blocks/paragraph/style.css 273 B
build/block-library/blocks/post-author/style-rtl.css 175 B
build/block-library/blocks/post-author/style.css 176 B
build/block-library/blocks/post-comments-form/style-rtl.css 446 B
build/block-library/blocks/post-comments-form/style.css 446 B
build/block-library/blocks/post-comments/style-rtl.css 507 B
build/block-library/blocks/post-comments/style.css 507 B
build/block-library/blocks/post-excerpt/editor-rtl.css 73 B
build/block-library/blocks/post-excerpt/editor.css 73 B
build/block-library/blocks/post-excerpt/style-rtl.css 69 B
build/block-library/blocks/post-excerpt/style.css 69 B
build/block-library/blocks/post-featured-image/editor-rtl.css 721 B
build/block-library/blocks/post-featured-image/editor.css 721 B
build/block-library/blocks/post-featured-image/style-rtl.css 153 B
build/block-library/blocks/post-featured-image/style.css 153 B
build/block-library/blocks/post-template/editor-rtl.css 99 B
build/block-library/blocks/post-template/editor.css 98 B
build/block-library/blocks/post-template/style-rtl.css 391 B
build/block-library/blocks/post-template/style.css 392 B
build/block-library/blocks/post-terms/style-rtl.css 73 B
build/block-library/blocks/post-terms/style.css 73 B
build/block-library/blocks/post-title/style-rtl.css 80 B
build/block-library/blocks/post-title/style.css 80 B
build/block-library/blocks/preformatted/style-rtl.css 103 B
build/block-library/blocks/preformatted/style.css 103 B
build/block-library/blocks/pullquote/editor-rtl.css 198 B
build/block-library/blocks/pullquote/editor.css 198 B
build/block-library/blocks/pullquote/style-rtl.css 389 B
build/block-library/blocks/pullquote/style.css 388 B
build/block-library/blocks/pullquote/theme-rtl.css 167 B
build/block-library/blocks/pullquote/theme.css 167 B
build/block-library/blocks/query-pagination-numbers/editor-rtl.css 122 B
build/block-library/blocks/query-pagination-numbers/editor.css 121 B
build/block-library/blocks/query-pagination/editor-rtl.css 221 B
build/block-library/blocks/query-pagination/editor.css 211 B
build/block-library/blocks/query-pagination/style-rtl.css 273 B
build/block-library/blocks/query-pagination/style.css 269 B
build/block-library/blocks/query/editor-rtl.css 131 B
build/block-library/blocks/query/editor.css 132 B
build/block-library/blocks/quote/style-rtl.css 187 B
build/block-library/blocks/quote/style.css 187 B
build/block-library/blocks/quote/theme-rtl.css 223 B
build/block-library/blocks/quote/theme.css 226 B
build/block-library/blocks/rss/editor-rtl.css 202 B
build/block-library/blocks/rss/editor.css 204 B
build/block-library/blocks/rss/style-rtl.css 289 B
build/block-library/blocks/rss/style.css 288 B
build/block-library/blocks/search/editor-rtl.css 165 B
build/block-library/blocks/search/editor.css 165 B
build/block-library/blocks/search/style-rtl.css 397 B
build/block-library/blocks/search/style.css 398 B
build/block-library/blocks/search/theme-rtl.css 64 B
build/block-library/blocks/search/theme.css 64 B
build/block-library/blocks/separator/editor-rtl.css 99 B
build/block-library/blocks/separator/editor.css 99 B
build/block-library/blocks/separator/style-rtl.css 245 B
build/block-library/blocks/separator/style.css 245 B
build/block-library/blocks/separator/theme-rtl.css 172 B
build/block-library/blocks/separator/theme.css 172 B
build/block-library/blocks/shortcode/editor-rtl.css 474 B
build/block-library/blocks/shortcode/editor.css 474 B
build/block-library/blocks/site-tagline/editor-rtl.css 86 B
build/block-library/blocks/site-tagline/editor.css 86 B
build/block-library/blocks/site-title/editor-rtl.css 84 B
build/block-library/blocks/site-title/editor.css 84 B
build/block-library/blocks/social-link/editor-rtl.css 177 B
build/block-library/blocks/social-link/editor.css 177 B
build/block-library/blocks/social-links/editor-rtl.css 670 B
build/block-library/blocks/social-links/editor.css 669 B
build/block-library/blocks/social-links/style-rtl.css 1.32 kB
build/block-library/blocks/social-links/style.css 1.32 kB
build/block-library/blocks/spacer/editor-rtl.css 307 B
build/block-library/blocks/spacer/editor.css 307 B
build/block-library/blocks/spacer/style-rtl.css 48 B
build/block-library/blocks/spacer/style.css 48 B
build/block-library/blocks/table/editor-rtl.css 471 B
build/block-library/blocks/table/editor.css 472 B
build/block-library/blocks/table/style-rtl.css 481 B
build/block-library/blocks/table/style.css 481 B
build/block-library/blocks/table/theme-rtl.css 188 B
build/block-library/blocks/table/theme.css 188 B
build/block-library/blocks/tag-cloud/style-rtl.css 146 B
build/block-library/blocks/tag-cloud/style.css 146 B
build/block-library/blocks/template-part/editor-rtl.css 560 B
build/block-library/blocks/template-part/editor.css 559 B
build/block-library/blocks/template-part/theme-rtl.css 101 B
build/block-library/blocks/template-part/theme.css 101 B
build/block-library/blocks/text-columns/editor-rtl.css 95 B
build/block-library/blocks/text-columns/editor.css 95 B
build/block-library/blocks/text-columns/style-rtl.css 166 B
build/block-library/blocks/text-columns/style.css 166 B
build/block-library/blocks/verse/style-rtl.css 87 B
build/block-library/blocks/verse/style.css 87 B
build/block-library/blocks/video/editor-rtl.css 569 B
build/block-library/blocks/video/editor.css 570 B
build/block-library/blocks/video/style-rtl.css 173 B
build/block-library/blocks/video/style.css 173 B
build/block-library/blocks/video/theme-rtl.css 124 B
build/block-library/blocks/video/theme.css 124 B
build/block-library/common-rtl.css 910 B
build/block-library/common.css 908 B
build/block-library/reset-rtl.css 474 B
build/block-library/reset.css 474 B
build/block-serialization-default-parser/index.min.js 1.09 kB
build/block-serialization-spec-parser/index.min.js 2.79 kB
build/blocks/index.min.js 46.3 kB
build/core-data/index.min.js 13.2 kB
build/customize-widgets/index.min.js 11.4 kB
build/customize-widgets/style-rtl.css 1.5 kB
build/customize-widgets/style.css 1.49 kB
build/data-controls/index.min.js 631 B
build/data/index.min.js 7.49 kB
build/date/index.min.js 31.9 kB
build/deprecated/index.min.js 485 B
build/dom-ready/index.min.js 304 B
build/dom/index.min.js 4.5 kB
build/edit-navigation/index.min.js 16 kB
build/edit-navigation/style-rtl.css 3.76 kB
build/edit-navigation/style.css 3.76 kB
build/edit-post/classic-rtl.css 492 B
build/edit-post/classic.css 494 B
build/edit-widgets/style-rtl.css 4.18 kB
build/edit-widgets/style.css 4.18 kB
build/element/index.min.js 3.29 kB
build/escape-html/index.min.js 517 B
build/format-library/index.min.js 6.58 kB
build/format-library/style-rtl.css 571 B
build/format-library/style.css 571 B
build/hooks/index.min.js 1.63 kB
build/html-entities/index.min.js 424 B
build/i18n/index.min.js 3.71 kB
build/is-shallow-equal/index.min.js 501 B
build/keyboard-shortcuts/index.min.js 1.8 kB
build/keycodes/index.min.js 1.39 kB
build/list-reusable-blocks/index.min.js 1.72 kB
build/list-reusable-blocks/style-rtl.css 838 B
build/list-reusable-blocks/style.css 838 B
build/media-utils/index.min.js 2.92 kB
build/notices/index.min.js 925 B
build/nux/index.min.js 2.08 kB
build/nux/style-rtl.css 747 B
build/nux/style.css 743 B
build/plugins/index.min.js 1.84 kB
build/primitives/index.min.js 924 B
build/priority-queue/index.min.js 582 B
build/react-i18n/index.min.js 671 B
build/react-refresh-entry/index.min.js 8.44 kB
build/react-refresh-runtime/index.min.js 7.31 kB
build/reusable-blocks/index.min.js 2.22 kB
build/reusable-blocks/style-rtl.css 256 B
build/reusable-blocks/style.css 256 B
build/rich-text/index.min.js 11 kB
build/server-side-render/index.min.js 1.57 kB
build/shortcode/index.min.js 1.49 kB
build/token-list/index.min.js 639 B
build/url/index.min.js 1.9 kB
build/viewport/index.min.js 1.05 kB
build/warning/index.min.js 248 B
build/widgets/index.min.js 7.15 kB
build/widgets/style-rtl.css 1.16 kB
build/widgets/style.css 1.16 kB
build/wordcount/index.min.js 1.04 kB

compressed-size-action

@ockham
Copy link
Contributor Author

ockham commented Dec 16, 2021

Wondering if the issue could be due to something like jestjs/jest#11054 (essentially console output buffering). IIUC, the fix was released as part of Jest 27.2.x, and we have yet to upgrade to Jest 27.

Might be worth trying this PR on top of the Jest upgrade PR.

@ockham
Copy link
Contributor Author

ockham commented Dec 17, 2021

Wondering if the issue could be due to something like facebook/jest#11054 (essentially console output buffering). IIUC, the fix was released as part of Jest 27.2.x, and we have yet to upgrade to Jest 27.

Might be worth trying this PR on top of the Jest upgrade PR.

Tried that -- that's also not it.

@ockham
Copy link
Contributor Author

ockham commented Dec 17, 2021

@gziolo had a few suggestions over in #36096 (React error boundaries, onError handlers). While I see a few of those across the codebase, I'm not totally sure which of them are relevant.

What I don't really get is where this (i.e. expect( console ).not.toHaveErrored()) seems to be working for other tests that don't seem that different.


I'll probably punt this for now, I've wasted too much time on it already. If anyone has an idea on how to get this to fail, feel free to pick it up. (Note that I've purposefully not rebased this PR yet on #37383, since we need to make sure the test really fails without that fix.)

@ockham
Copy link
Contributor Author

ockham commented Dec 20, 2021

Giving this one more try 😅 It seems like I can catch the error usingpage.on ( 'pageerror', ( err ) => { /* ... */ } ).

It seems like packages/scripts/config/jest-environment-puppeteer/index.js modifies the pageerror handler; currently looking into that.

@ockham
Copy link
Contributor Author

ockham commented Dec 20, 2021

With 2a64a02 (and .skip()ping the other tests in the same file), I've managed to get the test to fail locally. Still not failing in CI 🙄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
[Package] E2E Tests /packages/e2e-tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

1 participant