[RNMobile] - Update tests with paragraph blocks to work on both environments and platforms #40404
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jostnes
added
Mobile App - i.e. Android or iOS
|
Size Change: +111 B (0%) Total Size: 1.23 MB
ℹ️ View Unchanged
|
jostnes
commented
Apr 18, 2022
packages/react-native-editor/__device-tests__/gutenberg-editor-paragraph.test.js
Show resolved
Hide resolved
packages/react-native-editor/__device-tests__/gutenberg-editor-paragraph.test.js
Show resolved
Hide resolved
packages/react-native-editor/__device-tests__/gutenberg-editor-paragraph.test.js
Outdated
Show resolved
Hide resolved
packages/react-native-editor/__device-tests__/pages/editor-page.js
Outdated
Show resolved
Hide resolved
fluiddot
approved these changes
Apr 26, 2022
There was a problem hiding this comment.
LGTM 🎊, great job @jostnes 🏅.
The changes look good and the PR checks pass in wordpress-mobile/gutenberg-mobile#4765, so I consider this PR as ready to merge. However, I'd like to share a consistent failure I only had locally and while running Android tests.
Failure in test gutenberg-editor-block-insertion-2:
Gutenberg Editor tests for Block insertion 2
✓ adds new block at the end of post (8717 ms)
✓ inserts between 2 existing blocks (5826 ms)
✕ inserts block before selected block (28684 ms)
✕ inserts block at the end of post when no block is selected (4509 ms)
✕ creates a new Paragraph block tapping on the empty area below the last block (3299 ms)
● Gutenberg Editor tests for Block insertion 2 › inserts block before selected block
"//android.widget.Button[@content-desc="Add Block Before"]" is still not visible after 25 retries!
at _callee21$ (gutenberg/packages/react-native-editor/__device-tests__/helpers/utils.js:972:17)
at tryCatch (gutenberg/node_modules/regenerator-runtime/runtime.js:63:40)
at Generator.invoke [as _invoke] (gutenberg/node_modules/regenerator-runtime/runtime.js:293:22)
at Generator.next (gutenberg/node_modules/regenerator-runtime/runtime.js:118:21)
at tryCatch (gutenberg/node_modules/regenerator-runtime/runtime.js:63:40)
at invoke (gutenberg/node_modules/regenerator-runtime/runtime.js:154:20)
at gutenberg/node_modules/regenerator-runtime/runtime.js:189:11
at callInvokeWithMethodAndArg (gutenberg/node_modules/regenerator-runtime/runtime.js:188:16)
at AsyncIterator.enqueue (gutenberg/node_modules/regenerator-runtime/runtime.js:211:13)
● Gutenberg Editor tests for Block insertion 2 › inserts block at the end of post when no block is selected
expect(received).toBe(expected) // Object.is equality
- Expected - 8
+ Received + 0
@@ -1,17 +1,9 @@
<!-- wp:heading -->
<h2></h2>
<!-- /wp:heading -->
- <!-- wp:image -->
- <figure class="wp-block-image"><img alt=""/></figure>
- <!-- /wp:image -->
-
- <!-- wp:separator -->
- <hr class="wp-block-separator has-alpha-channel-opacity"/>
- <!-- /wp:separator -->
-
<!-- wp:list -->
<ul><li></li></ul>
<!-- /wp:list -->
<!-- wp:more -->
76 |
77 | const html = await editorPage.getHtmlContent();
> 78 | expect( html.toLowerCase() ).toBe( expectedHtml );
| ^
79 | } );
80 |
81 | it( 'creates a new Paragraph block tapping on the empty area below the last block', async () => {
at _callee4$ (gutenberg/packages/react-native-editor/__device-tests__/gutenberg-editor-block-insertion-2.test.js:78:32)
at tryCatch (gutenberg/node_modules/regenerator-runtime/runtime.js:63:40)
at Generator.invoke [as _invoke] (gutenberg/node_modules/regenerator-runtime/runtime.js:293:22)
at Generator.next (gutenberg/node_modules/regenerator-runtime/runtime.js:118:21)
at tryCatch (gutenberg/node_modules/regenerator-runtime/runtime.js:63:40)
at invoke (gutenberg/node_modules/regenerator-runtime/runtime.js:154:20)
at gutenberg/node_modules/regenerator-runtime/runtime.js:164:13
● Gutenberg Editor tests for Block insertion 2 › creates a new Paragraph block tapping on the empty area below the last block
expect(received).toBe(expected) // Object.is equality
- Expected - 8
+ Received + 0
@@ -1,17 +1,9 @@
<!-- wp:heading -->
<h2></h2>
<!-- /wp:heading -->
- <!-- wp:image -->
- <figure class="wp-block-image"><img alt=""/></figure>
- <!-- /wp:image -->
-
- <!-- wp:separator -->
- <hr class="wp-block-separator has-alpha-channel-opacity"/>
- <!-- /wp:separator -->
-
<!-- wp:list -->
<ul><li></li></ul>
<!-- /wp:list -->
<!-- wp:more -->
92 |
93 | const html = await editorPage.getHtmlContent();
> 94 | expect( html.toLowerCase() ).toBe( expectedHtml );
| ^
95 | } );
96 | } );
97 |
at _callee5$ (gutenberg/packages/react-native-editor/__device-tests__/gutenberg-editor-block-insertion-2.test.js:94:32)
at tryCatch (gutenberg/node_modules/regenerator-runtime/runtime.js:63:40)
at Generator.invoke [as _invoke] (gutenberg/node_modules/regenerator-runtime/runtime.js:293:22)
at Generator.next (gutenberg/node_modules/regenerator-runtime/runtime.js:118:21)
at tryCatch (gutenberg/node_modules/regenerator-runtime/runtime.js:63:40)
at invoke (gutenberg/node_modules/regenerator-runtime/runtime.js:154:20)
at gutenberg/node_modules/regenerator-runtime/runtime.js:164:13
The error doesn't look related to these changes but at least locally, I confirm it failed consistently after re-running the test several times.
| @@ -9,7 +9,11 @@ describe( 'Gutenberg Editor tests', () => { | |||
| it( 'should be able to create a post with heading and paragraph blocks', async () => { | |||
There was a problem hiding this comment.
Not related to this PR but I noticed this test case doesn't have expect statements 🤔, it will fail for some error cases but I think we're not fully covering the logic without actually checking the HTML output.
There was a problem hiding this comment.
You are right, this test can potentially be a false positive. I'll update the assert for this in the next PR where I'm planning to update heading and lists blocks.
packages/react-native-editor/__device-tests__/gutenberg-editor-paragraph.test.js
Outdated
Show resolved
Hide resolved
packages/react-native-editor/__device-tests__/pages/editor-page.js
Outdated
Show resolved
Hide resolved
packages/react-native-editor/__device-tests__/pages/editor-page.js
Outdated
Show resolved
Hide resolved
|
Re: I was not able to reproduce it on my local Android: That particular set of tests is interdependent, so once one starts failing subsequent tests would follow. Based on the error it looks like it couldn't find the I suspect that the test was "too fast" on your local (I'm comparing the timings for the first two tests), can you try adding the |
Yep, I have the impression that this might be related to the fact that running them locally goes faster. I'll try adding the |
SiobhyB
pushed a commit
that referenced
this pull request
Apr 28, 2022
…onments and platforms (#40404) * add fixes for tests with paragraph blocks * lint fixes * update missed paragraph blocks to use getTextBlockLocatorAtPosition * lint fixes * remove unused code and revert ios coordinate condition * lint fix * remove extra backspace press * rename for clarity Co-authored-by: jos <17252150+jostnes@users.noreply.github.com>
4 tasks
|
I tried using the Note how at 0:22 the Separator block disappears 🙃 . android-e2e-test.mp4 |
|
That is... very strange. I looked at the code, there is no backspace keypress to have that block removed. I usually use the same simulator/emulator used on CI on my local (in this case, Though I can now see a future challenge when we decide to expand the tests to test on multiple devices... |
|
Yep, it's a bit odd, probably it's related to my emulator configuration 🤷♂️ . Since it's working on CI and other emulators, I wouldn't worry too much about this, let's see when we perform the E2E tests on new devices if this can be reproduced. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What?
This is a follow up to #40078. However, instead of just updating
gutenberg-editor-paragraph.test.js, this PR updates all tests that are using the paragraph block. The reason is because adding a condition to make it only work on thegutenberg-editor-paragraph.test.jsfile seems to be more complicated as the functions are used on multiple tests. So in the end, decided that it made more sense to update it this way. This PR does not fix tests that are not using paragraph blocks. That will be handled separately in a separate PR(s).Why?
This PR will attempt to fix the flakiness that we see on tests related to the paragraph block. This PR will also make the tests work in both local and CI environments. Previously, the tests only work on CI with some flakiness every once in a while.
How?
getTextBlockLocatorAtPosition()for paragraph block which will now wait for the block to be visible and return the locator for the block at the chosen position.getNumberOfParagraphBlocks()that will check the number of paragraph blocks that exist on the screen.waitForVisible()Testing Instructions
Ensure that the affected test files work locally and in CI.
For local testing, use the same simulator/emulator that CI is using:
iOS - iPhone 11, iOS 14.5
Android - Google Pixel 3 XL GoogleAPI Emulator, Android 9.0
This was also tested here and all updated tests are passing 6 out of 6 times. There are a few failing test cases in the test runs that will be handled separately. Failing tests from the test runs:
Screenshots or screencast