Add UI Tests for Simple Text Formatting #382

rachelmcr · 2017-06-13T23:36:06Z

This PR adds a suite of UI tests for simple text formatting in the Aztec demo app, using the Android Espresso testing framework.

Before running the tests

Espresso advises disabling system animations on devices used for testing:

On your device, under Settings->Developer options disable the following 3 settings:

Window animation scale

Transition animation scale

Animator duration scale

One additional setup step is also required to handle an Espresso issue with clicks (see the caveats below):

On your device, under Settings -> Accessibility -> Touch & hold delay, set the delay to Long.

Test Details

Each test performs the following steps:

Enter text (unformatted).
Select a formatting button in the toolbar.
Enter more text (with formatting).
Verify the correct HTML appears in HTML mode for the unformatted and formatted text.

A couple caveats:

The Espresso framework sometimes performs a long click instead of a single click (see the Espresso code for an explanation). In the Aztec toolbar, this displays a hint instead of selecting the formatting button and causes the test to fail.
Espresso also has trouble with the editor's link dialog. Although it can identify the fields and buttons in the dialog (and TalkBack works with the dialog as expected) it clicks in the wrong position. This seems to be a bug with the coordinates select for the click action for a custom AlertDialog and so I reversed the link formatting test. (It enters the link HTML and then verifies that the link dialog displays the expected values.)

cc @catehstn @lancewillett @hoverduck

rachelmcr · 2017-06-13T23:50:08Z

I just edited the original description to add a setup step for changing the "Touch & hold delay" setting to Long on the device used for testing. I'd forgotten this step earlier, but it should resolve most issues with the click turning into a long click in these tests (phones and tablets).

I do want to note that I observed a persistent problem with the click turning into a long click if the formatting button was selected as the first step in the test (before adding any text to the editor). That could indicate a potential performance issue when Aztec is first launched, and I went ahead and opened an issue to track that: #383

catehstn

This looks great!

catehstn · 2017-06-15T09:28:35Z

app/src/androidTest/java/org/wordpress/aztec/demo/BetterClickAction.java

+    @Override
+    @SuppressWarnings("unchecked")
+    public Matcher<View> getConstraints() {
+        Matcher<View> standardConstraint = isDisplayingAtLeast(65);


What's the 65 for? Does this relate to the comment at the top?

Yes, the 65 here is a percentage: The default ClickAction in Espresso checks to make sure at least 90% of the element is displayed on-screen before clicking on it. The 65 here checks to make sure at least 65% of the element is displayed on-screen instead.

catehstn · 2017-06-15T09:28:35Z

app/src/androidTest/java/org/wordpress/aztec/demo/SimpleTextFormattingTests.java

+        toggleHTMLView();
+        sourceText.check(matches(withText(unformattedText + "\n<pre>" + formattedText + "</pre>")));
+    }
+}


catehstn · 2017-06-15T09:28:35Z

app/src/androidTest/java/org/wordpress/aztec/demo/TestUtils.java

+public class TestUtils {
+
+    // Strings
+    public static String unformattedText = "hello";


Nice that we can have proper strings in Android - unlike iOS! :)

catehstn · 2017-06-15T09:28:35Z

app/src/androidTest/java/org/wordpress/aztec/demo/SimpleTextFormattingTests.java

+
+        // Check that HTML formatting tags were correctly added
+        toggleHTMLView();
+        sourceText.check(matches(withText(unformattedText + "\n<ol>\n\t<li>" + formattedText + "</li>\n</ol>")));


why is there a \t here?

The \t indents the list item on screen.

catehstn · 2017-06-15T09:28:35Z

app/src/androidTest/java/org/wordpress/aztec/demo/SimpleTextFormattingTests.java

+    }
+
+    /*
+    * This test is disabled because Espresso does not click in the correct position for the link dialog's OK button.


I think don't comment out a test if you've replaced it - instead probably add a comment inside the other one noting why it's tested that way and not this.

catehstn · 2017-06-15T09:28:52Z

@0nko ptal - if you're happy let's merge it.

hoverduck · 2017-06-16T18:23:57Z

@rachelmcr I finally got a chance to get my Android dev environment set back up and take a look. Everything checks out 👍 !

My only comment is that you should probably include an update to the README to provide the "Before running the tests" setup steps from the PR details. Is there maybe a way to automate those steps via a set of ADB commands prior to starting the tests?

0nko · 2017-06-19T11:45:53Z

After following the device configuration steps the tests pass. I'll add these to the README, like @hoverduck suggested.

Quickly searching for the adb commands I did find something for disabling the animations:

adb shell content update --uri content://settings/system --bind value:s:0.0 --where 'name="window_animation_scale"'
adb shell content update --uri content://settings/system --bind value:s:0.0 --where 'name="transition_animation_scale"'
adb shell content update --uri content://settings/system --bind value:s:0.0 --where 'name="animator_duration_scale"'

And for setting a long tap delay:

adb shell settings put secure long_press_timeout <time in ms>

I haven't tested them, though. @rachelmcr, thank you so much for setting up the test environment and writing these!

0nko · 2017-06-19T12:02:11Z

rachelmcr added 8 commits May 15, 2017 22:26

Add bold text formatting test

46726c6

Reverse test to enter HTML and check for enabled format button

f885132

Add helper methods

f8af68d

Add check for disabled format button at start of test

92e7a84

Add simple formatting tests

3ca2d85

Update paragraph style selectors

728446c

Add custom actions to fix page break test

3d117ee

Replace failing link formatting test

41c85d2

rachelmcr added enhancement test labels Jun 13, 2017

rachelmcr mentioned this pull request Jun 13, 2017

Possible performance issue when Aztec is first launched #383

Open

catehstn reviewed Jun 15, 2017

View changes

0nko self-assigned this Jun 15, 2017

rachelmcr added 2 commits June 15, 2017 13:28

Remove commented out link formatting test

c044ad1

Add explanatory comments to click and scroll actions

3f26f95

0nko merged commit a22b37b into develop Jun 19, 2017

0nko deleted the add/ui-text-formatting-tests branch June 19, 2017 12:02

0nko modified the milestone: Open beta Jun 22, 2017

Add UI Tests for Simple Text Formatting #382

Add UI Tests for Simple Text Formatting #382

rachelmcr commented Jun 13, 2017 •

edited

rachelmcr commented Jun 13, 2017

catehstn left a comment

catehstn Jun 15, 2017

rachelmcr Jun 15, 2017

catehstn Jun 15, 2017

catehstn Jun 15, 2017

catehstn Jun 15, 2017

catehstn Jun 15, 2017

rachelmcr Jun 15, 2017

catehstn Jun 15, 2017

catehstn commented Jun 15, 2017

hoverduck commented Jun 16, 2017

0nko commented Jun 19, 2017

0nko commented Jun 19, 2017

Add UI Tests for Simple Text Formatting #382

Add UI Tests for Simple Text Formatting #382

Conversation

rachelmcr commented Jun 13, 2017 • edited

Before running the tests

Test Details

rachelmcr commented Jun 13, 2017

catehstn left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

catehstn commented Jun 15, 2017

hoverduck commented Jun 16, 2017

0nko commented Jun 19, 2017

0nko commented Jun 19, 2017

rachelmcr commented Jun 13, 2017 •

edited