Speeding up Testaro tests with Promise.all

[tgrushka]

Hi Jonathan,

Thank you and CVS for such a great test suite! I'd like to make a small contribution in terms of test speed on some of the Testaro tests.

The caveat is that this improvement can only be made on tests that are not affected by:

• order of execution;
• concurrent execution;
• interaction within the each locator loop using the Playwright API such as await page.keyboard.press (e.g. as in focAll.js).

Taking focOp.js as an example, this test inspects each focusable element (with tabIndex === 0 ... another question: why not tabIndex >= 0? I know, setting tabIndex to a value > 0 is bad practice but another test could be added for focus order ... at least tabIndex > 0 elements I believe are still keyboard focusable...). The isOperable(loc) just inspects the computedStyle and other props of the element. Therefore, we should be able to run these in parallel.

We can simply replace:

  // For each locator:
  for (const loc of all.allLocs) {
  ...
  }

with:

  // For each locator:
  await Promise.all(
    all.allLocs.map(async loc => {
    ...
    })
  )

On a page with a sample size of 100, running only the loop 20 times, I got the following averages:

• 4295 ms running consecutively
• 413 ms running concurrently

which is roughly 10.4 times faster.

Another example, focInd:
On a page with a sample size of 100, running only the loop 20 times, I got the following averages:

• 4120 ms running consecutively
• 880 ms running concurrently

which is roughly 4.7 times faster.

Should add up significantly over multiple tests and thousands of pages.

A more involved step would be to bypass the allLocs altogether, and thus serialization/deserialization overhead between node and chromium, and just put the entire test loop into a single page.evaluate call with a document.querySelectorAll.

Any reason this could be a problem that I haven't considered?

[jrpool]

Thank you, @tgrushka. This seems like a superb idea! It is also possible to manage Testaro work by splitting tests and tools among multiple agents and combining their reports, but obviously we should also make any testing by any agent efficient. I don’t foresee any complications, but testing will reveal them, if any. Would you like to create a pull request for one increment? Testing should show that it is faster but otherwise identical.

Changing tabIndex to >= 0 also seems right.

I have been avoiding page-level evaluations that need to use shared functions because it seems impossible to call such functions without converting them to text and then, inside the evaluation, converting the text to a function with a Function() constructor.

In general, Testaro mainly seeks to facilitate multi-tool ensemble testing, and efficiency has so far been a secondary concern. Some demos using it update the user agent while the user waits a few minutes for completion; others tell the user to wait for an email message that will contain a link to a report. So, if you can contribute some efficiency improvements, that will nicely complement the work that I have done.

By the way, I was a happy and appreciative user of Cyberduck for several years.