Make BDD end2end tests faster

[toofar]

The Behaviour Driven Design / gherkin / .feature based end2end tests have proven to be useful as a low barrier to entry way to get test coverage for new features and fixes. While there are plans (#3319) to reduce our reliance on these tests (and some progress toward that scattered about the place) while we are still reliant on them it would be great if we could make them run faster somehow. I think the average time for them is like 1s per test, which makes the full test suite quite slow.

This will probably involve a lot of tedious investigation into what types of patterns are slowest and options to speed them up. And it may end up to be largely intractable and we have to live with them as-is (until we replace them).

I've looked at the IPC mechanism and regex matching in the past and don't believe they are slow, a lot of the slowness seems to be in the browser under test itself (;_;) (for example, opening a new window seems to show up as a common slow step), but don't let that vague assertion bias your investigation. Doing a profile of the test process should help get more visibility on that.

Speeding up the browser itself is out of scope for this investigation, so if you see an opportunity to do that please raise a separate issue for it.

One opportunity to speed things up I've seen is to batch commands where possible, instead of running them one by one. Opening tabs all at once in each step of a feature file with just four tests saved about 25% of the total run time (see example below). For instead of doing:

And I open about:blank?2.txt in a new related tab
And I open about:blank?3.txt in a new related tab
And I open about:blank?4.txt in a new tab

We might be able to do:

And run the following steps in parallel
    And I open about:blank?2.txt in a new related tab
    And I open about:blank?3.txt in a new related tab
    And I open about:blank?4.txt in a new tab
And I wait for
    about:blank?2.txt is loaded
    about:blank?3.txt is loaded
    about:blank?4.txt is loaded

With the "wait for" step being optional, for example currently with "And I open" you don't need to specify your own wait line, you only have to do that when you run raw commands like And I run :open -t about:blank?3.txt.

This may increase the flakyness of tests, since we don't have a way of strictly specifying command ordering (see #3007) so there should be some thought put into 1) how to highlight ordering issues if a test fails 2) how to disable the parallelization either per test (with a fixture or even just deleting the "in parallel" line) 3) for a whole test run. In practice though I suspect there are a few common cases where it would work fine.

Another thing I saw in the same example below is that opening about:blank tabs seems to be faster than opening tiny files in the local webserver. Not sure if that's expected or it's something to do with the webserver itself.

Changing the "----> found it" log message to be f"----> found it ({elapsed_timer.elapsed()}) ({match.message})" could help highlight slow pieces to be investigated, same with pytest's --duration=<num> command.

example

This is a small BDD scenario I was looking at today. I tested batching up tab open commands and switching from opening a file from the web server vs opening about:blank. The times I got across four runs of each attempt were:

# webserver and one-by-one
5.45
5.49
5.73
5.94

# about blank and one-by-one
5.16
4.91
5.32
4.68

# webserver and batched
5.67
5.47
5.52
5.06

# about blank and batched
4.65
4.96
4.91
4.88

The initial version of the file was:

Feature: Tree tab management
    Tests for various :tree-tab-* commands.

    Background:
        # Open a new tree tab enabled window, close everything else
        Given I set tabs.tabs_are_windows to false
        And I set tabs.tree_tabs to true
        And I open about:blank?starting%20page in a new window
        And I clean up open tabs
        And I clear the log

    Scenario: :tab-close --recursive
        When I open data/numbers/1.txt
        And I open data/numbers/2.txt in a new related tab
        And I open data/numbers/3.txt in a new related tab
        And I open data/numbers/4.txt in a new tab
        And I run :tab-focus 1
        And I run :tab-close --recursive
        Then the following tabs should be open:
            - data/numbers/4.txt

    Scenario: Open a child tab
        When I open data/numbers/1.txt
        And I open data/numbers/2.txt in a new related tab
        Then the following tabs should be open:
            - data/numbers/1.txt
              - data/numbers/2.txt (active)

    Scenario: Collapse a subtree
        When I open data/numbers/1.txt
        And I open data/numbers/2.txt in a new related tab
        And I open data/numbers/3.txt in a new related tab
        And I run :tab-focus 2
        And I run :tree-tab-toggle-hide
        Then the following tabs should be open:
            - data/numbers/1.txt
              - data/numbers/2.txt (active) (collapsed)
                - data/numbers/3.txt

    # Same as a test in sessions.feature but tree tabs.
    Scenario: TreeTabs: Loading a session with tabs.new_position.related=prev
        When I open data/numbers/1.txt
        And I open data/numbers/2.txt in a new related tab
        And I open data/numbers/3.txt in a new related tab
        And I open data/numbers/4.txt in a new tab
        And I run :tab-focus 2
        And I run :tree-tab-toggle-hide
        And I run :session-save foo
        And I set tabs.new_position.related to prev
        And I run :session-load -c foo
        And I wait until data/numbers/4.txt is loaded
        Then the following tabs should be open:
          - data/numbers/1.txt
            - data/numbers/2.txt (active) (collapsed)
              - data/numbers/3.txt
          - data/numbers/4.txt

And with about:blank:

Feature: Tree tab management
    Tests for various :tree-tab-* commands.

    Background:
        # Open a new tree tab enabled window, close everything else
        Given I set tabs.tabs_are_windows to false
        And I set tabs.tree_tabs to true
        And I open about:blank?starting%20page in a new window
        And I clean up open tabs
        And I clear the log

    Scenario: :tab-close --recursive
        When I open about:blank?1.txt
        And I open about:blank?2.txt in a new related tab
        And I open about:blank?3.txt in a new related tab
        And I open about:blank?4.txt in a new tab
        And I run :tab-focus 1
        And I run :tab-close --recursive
        Then the following tabs should be open:
            - about:blank?4.txt

    Scenario: Open a child tab
        When I open about:blank?1.txt
        And I open about:blank?2.txt in a new related tab
        Then the following tabs should be open:
            - about:blank?1.txt
              - about:blank?2.txt (active)

    Scenario: Collapse a subtree
        When I open about:blank?1.txt
        And I open about:blank?2.txt in a new related tab
        And I open about:blank?3.txt in a new related tab
        And I run :tab-focus 2
        And I run :tree-tab-toggle-hide
        Then the following tabs should be open:
            - about:blank?1.txt
              - about:blank?2.txt (active) (collapsed)
                - about:blank?3.txt

    # Same as a test in sessions.feature but tree tabs.
    Scenario: TreeTabs: Loading a session with tabs.new_position.related=prev
        When I open about:blank?1.txt
        And I open about:blank?2.txt in a new related tab
        And I open about:blank?3.txt in a new related tab
        And I open about:blank?4.txt in a new tab
        And I run :tab-focus 2
        And I run :tree-tab-toggle-hide
        And I run :session-save foo
        And I set tabs.new_position.related to prev
        And I run :session-load -c foo
        And I wait until about:blank?4.txt is loaded
        Then the following tabs should be open:
          - about:blank?1.txt
            - about:blank?2.txt (active) (collapsed)
              - about:blank?3.txt
          - about:blank?4.txt

And back to loading from the webserver but batched (using ;; to chain :open commands):

Feature: Tree tab management
    Tests for various :tree-tab-* commands.

    Background:
        # Open a new tree tab enabled window, close everything else
        Given I set tabs.tabs_are_windows to false
        And I set tabs.tree_tabs to true
        And I open about:blank?starting%20page in a new window
        And I clean up open tabs
        And I clear the log

    Scenario: :tab-close --recursive
        When I run :open http://localhost:(port)/data/numbers/1.txt ;; open -tr http://localhost:(port)/data/numbers/2.txt ;; open -tr http://localhost:(port)/data/numbers/3.txt ;; open -t http://localhost:(port)/data/numbers/4.txt
        And I wait until http://localhost:(port)/data/numbers/4.txt is loaded
        And I run :tab-focus 1
        And I run :tab-close --recursive
        Then the following tabs should be open:
            - http://localhost:(port)/data/numbers/4.txt

    Scenario: Open a child tab
        When I open http://localhost:(port)/data/numbers/1.txt
        And I open http://localhost:(port)/data/numbers/2.txt in a new related tab
        Then the following tabs should be open:
            - http://localhost:(port)/data/numbers/1.txt
              - http://localhost:(port)/data/numbers/2.txt (active)

    Scenario: Collapse a subtree
        When I run :open http://localhost:(port)/data/numbers/1.txt ;; open -tr http://localhost:(port)/data/numbers/2.txt ;; open -tr http://localhost:(port)/data/numbers/3.txt
        And I wait until http://localhost:(port)/data/numbers/3.txt is loaded
        And I run :tab-focus 2
        And I run :tree-tab-toggle-hide
        Then the following tabs should be open:
            - http://localhost:(port)/data/numbers/1.txt
              - http://localhost:(port)/data/numbers/2.txt (active) (collapsed)
                - http://localhost:(port)/data/numbers/3.txt

    # Same as a test in sessions.feature but tree tabs.
    Scenario: TreeTabs: Loading a session with tabs.new_position.related=prev
        When I run :open http://localhost:(port)/data/numbers/1.txt ;; open -tr http://localhost:(port)/data/numbers/2.txt ;; open -tr http://localhost:(port)/data/numbers/3.txt ;; open -t http://localhost:(port)/data/numbers/4.txt
        And I wait until http://localhost:(port)/data/numbers/4.txt is loaded
        And I run :tab-focus 2
        And I run :tree-tab-toggle-hide
        And I run :session-save foo
        And I set tabs.new_position.related to prev
        And I run :session-load -c foo
        And I wait until http://localhost:(port)/data/numbers/4.txt is loaded
        Then the following tabs should be open:
          - http://localhost:(port)/data/numbers/1.txt
            - http://localhost:(port)/data/numbers/2.txt (active) (collapsed)
              - http://localhost:(port)/data/numbers/3.txt
          - http://localhost:(port)/data/numbers/4.txt

And now about:blank again but batched:

Feature: Tree tab management
    Tests for various :tree-tab-* commands.

    Background:
        # Open a new tree tab enabled window, close everything else
        Given I set tabs.tabs_are_windows to false
        And I set tabs.tree_tabs to true
        And I open about:blank?starting%20page in a new window
        And I clean up open tabs
        And I clear the log

    Scenario: :tab-close --recursive
        When I run :open about:blank?1.txt ;; open -tr about:blank?2.txt ;; open -tr about:blank?3.txt ;; open -t about:blank?4.txt
        And I wait until about:blank?4.txt is loaded
        And I run :tab-focus 1
        And I run :tab-close --recursive
        Then the following tabs should be open:
            - about:blank?4.txt

    Scenario: Open a child tab
        When I open about:blank?1.txt
        And I open about:blank?2.txt in a new related tab
        Then the following tabs should be open:
            - about:blank?1.txt
              - about:blank?2.txt (active)

    Scenario: Collapse a subtree
        When I run :open about:blank?1.txt ;; open -tr about:blank?2.txt ;; open -tr about:blank?3.txt
        And I wait until about:blank?3.txt is loaded
        And I run :tab-focus 2
        And I run :tree-tab-toggle-hide
        Then the following tabs should be open:
            - about:blank?1.txt
              - about:blank?2.txt (active) (collapsed)
                - about:blank?3.txt

    # Same as a test in sessions.feature but tree tabs.
    Scenario: TreeTabs: Loading a session with tabs.new_position.related=prev
        When I run :open about:blank?1.txt ;; open -tr about:blank?2.txt ;; open -tr about:blank?3.txt ;; open -t about:blank?4.txt
        And I wait until about:blank?4.txt is loaded
        And I run :tab-focus 2
        And I run :tree-tab-toggle-hide
        And I run :session-save foo
        And I set tabs.new_position.related to prev
        And I run :session-load -c foo
        And I wait until about:blank?4.txt is loaded
        Then the following tabs should be open:
          - about:blank?1.txt
            - about:blank?2.txt (active) (collapsed)
              - about:blank?3.txt
          - about:blank?4.txt