PLANET-5167: PoC for BDD tests

[lithrel]

Refs:

• https://jira.greenpeace.org/browse/PLANET-5167
• https://jira.greenpeace.org/browse/PLANET-5168
• https://jira.greenpeace.org/browse/PLANET-5169

PoC of tests written in Gherkin as a way to facilitate communication and implementation of functional tests.
After trying to use Selenium IDE in Gutenberg editor, I realized that recording a scenario and replaying it is not a straightforward task; it requires multiple tries to find the right selectors, adding intermediate steps to wait for elements, and finding proper assertions. Although it is a great tool to automate repetitive tasks, it feels like a very technical thing to do in the context of Gutenberg, which is why I've pushed more the topic of textual tests that can be written by non-developers and carefully implemented by developers to be reusable.

Features

This PR allows to write tests in a feature format, a textual format readable and writable by non-developers:

  Scenario: Create a new post
    Given I am on a new post page
    When I add a title "Test title"
    And I add a paragraph "Test paragraph"
    And I publish the post
    Then a message "Post published." is displayed
    And the post is visible on the website

In this feature, codeception will directly interact with the editor, as a user would do.
The functionality already exists in Codeception, using it is a matter of implementing sentences into reusable bits of code.

To match a sentence with a method, annotations are used, @Given, @When and @Then:

	/**
	 * Login to WordPress as the admin user saving the session as snapshot to make
	 * subsequent admin logins reuse the session to save time.
	 *
	 * @Given I am logged in as administrator
	 */
	public function loginAsAdminCached()

In this example, I just used an existing method, and made it run againt the sentence Given I am logged in as administrator.
Sentences can become regular expression, to capture parameters or match variations of a sentence:

/**
 * @When /I publish the (post|page)/
 * will match `When I publish the post` and `When I publish the page`
 * @Then I see a message :message
 * will match `Then I see a message "any message"`, and propagate "any message" as first parameter for the method
 */

End goal

By implementing reusable sentences and making available a documented list of those, we can try to reach a point where a tester writes a feature file that is playable as is, without any technical intervention. It is in that spirit that the Jira tickets were written.
As the documentation says, not every test should be described as a feature. Features are slow to run, it is important to group them efficiently and use them to test valuable features of our code.

What could be done next

• Writing tests by groups of 10, to see how the code evolves and have time to refactor and define reusable vocabulary with testers
• Get a better sense of the difference between PageObject and StepObject (still confused :|)
• Extract selectors and page names to constant classes, make them available for all tests
  I like PHP-enum because it allows to type hint constants, but we can also do without it
• Make a better use of WPBrowser functionalities
• Solve the copy/paste problem 😭

Tests

Run in your make php-shell

$ tests/vendor/bin/codecept run -vv acceptance editor.feature

You will see all the steps and intermediate actions executed.

To get a list of all available steps:

$ tests/vendor/bin/codecept gherkin:steps acceptance
Steps from default context:
+------------------------------------+---------------------------------------------------------------+
| Step                               | Implementation                                                |
+------------------------------------+---------------------------------------------------------------+
| I am logged in as administrator    | AcceptanceTester::loginAsAdminCached                          |
| I open dashboard page              | Page\Acceptance\Admin\GutenbergEditor::openDashboardPage      |
| I am on a new post page            | Page\Acceptance\Admin\GutenbergEditor::newPostPage            |
| I add a title :arg1                | Page\Acceptance\Admin\GutenbergEditor::addTitle               |
| I add a paragraph :arg1            | Page\Acceptance\Admin\GutenbergEditor::addParagraph           |
| /I publish the (post|page)/        | Page\Acceptance\Admin\GutenbergEditor::publishPost            |
| I paste a video link :link         | Page\Acceptance\Admin\GutenbergEditor::pasteYoutubeLink       |
| I paste :text                      | Page\Acceptance\Admin\GutenbergEditor::pasteText              |
| I add a block :blockpath           | Page\Acceptance\Admin\GutenbergEditor::addBlock               |
| I open block selector              | Page\Acceptance\Admin\GutenbergEditor::openBlockSelector      |
| a message :message is displayed    | Page\Acceptance\Admin\GutenbergEditor::messageIsDisplayed     |
| I see a message :message           | Page\Acceptance\Admin\GutenbergEditor::messageIsDisplayed     |
| the post is visible on the website | Page\Acceptance\Admin\GutenbergEditor::postIsVisibleOnWebsite |
| I see the video in the editor      | Page\Acceptance\Admin\GutenbergEditor::videoIsEmbedded        |
| I see the video in the post        | Page\Acceptance\Admin\GutenbergEditor::videoVisibleOnWebsite  |
| I am on admin page :location       | Page\Acceptance\Admin\Navigation::navigateTo                  |
+------------------------------------+---------------------------------------------------------------+

To check if a step is not yet implemented (not matched by any existing annotation):

$ tests/vendor/bin/codecept dry-run acceptance editor.feature

Resources

• Codeception - Behavior Driven Development
• Behat - Defining reusable actions
• Codeception - Webdriver
• Chrome - Unblocking Clipboard Access
• Chrome - Deprecating Powerful Features on Insecure Origins
• XPath cheatsheet

[Inwerpsel]

Nice, this can really help us make our tests more readable for the entire team 👍 Left a few minor remarks relating to string handling, but in general it looks pretty solid.

Maybe one thing that's a bit unfortunate about Codeception's implementation is that, even though the Given/When/Then annotations are authoritative, since it's on instance methods we still need to give the method a name as well, even though we don't call it with its name in our code. That name can be quite different from what is in the annotation, which is a bit ironic as the idea is to make testers and coders speak a common language.

But I don't really see a way around this. Maybe we can mitigate somewhat by determining some rules to transform the @Given into a method name (including params/regex) so that it's not free-form anymore and we can always generate it without having to think about it?

[Inwerpsel]

If this function will always split the argument in two, isn't it more straightforward to give the function 2 parameters?

[Inwerpsel]

That way we get documentation for each part in our IDE, instead of a long documentation that needs to explain that there is 2 pieces of information in a single parameter.

[lithrel]

Good point, I have all the regex I want and I don't think about it 🤦

[Inwerpsel]

Constants are definitely a better fit here, as the value won't change.

Another option could be to create a function, which might be easier to work with if the selector has a parameter, like in $blockSelectorSection. Then we can avoid having to call sprintf whenever using the selector, which adds cognitive overhead as you need to locate the %s in the string without any color or hint from your IDE. That gets especially hard to read and easy to make mistakes if there is more than 1 parameter.

We can then also use string interpolation to make the selector more readable.

function selectBlock ( $name ) {
    return "//button[contains(@class, 'components-button')][text()='${name}']"
}

[Inwerpsel]

Moving these constants to a separate class indeed seems a good idea, as it looks like we'll have a lot of them.

[lithrel]

Maybe we can mitigate somewhat by determining some rules to transform the @Given into a method name (including params/regex) so that it's not free-form anymore and we can always generate it without having to think about it?

Codeception has a command to generate snippets based on unimplemented sentences, so theses sentences

When I add "some stuff"
Then I see a rainbow

will give

$ tests/vendor/bin/codecept gherkin:snippets acceptance
 Snippets found in:
  - editor.feature
 Generated Snippets:
 -----------------------------------------
    /**
     * @When I add :arg1
     */
     public function iAdd($arg1)
     {
         throw new \PHPUnit\Framework\IncompleteTestError("Step `I add :arg1` is not defined");
     }

    /**
     * @Then I see a rainbow
     */
     public function iSeeARainbow()
     {
         throw new \PHPUnit\Framework\IncompleteTestError("Step `I see a rainbow` is not defined");
     }

 -----------------------------------------
 2 snippets proposed
 Copy generated snippets to AcceptanceTester or a specific Gherkin context

so maybe we could follow this generated format. I didn't use it because those method name felt weird, but I guess it makes sense to keep code consistent :)

Also I recently saw that wp-cli uses features for tests, even for things like stderr output, so there's a bunch of examples there.

[lithrel]

Next PR in #81