An Interface For Running Tests Headlessly

[jameshopkins]

The three main use cases I envisage for running tests are:

1. using the UI and being provided visual feedback as part of a one-off process
2. integration into a CI/CD pipeline, where the process would be run as a build step to catch potential regressions introduced by new code in the release. It should have the ability to abort the deploy process if the SpeedIndex score dropped below a certain threshold, for example.
3. execution as part of a continuous cycle (at a user defined frequency) to check for sporadic latency introduced by external resources, such as JS and CSS files.

The last two 'headless' concepts haven't really been discussed before, but I think will likely be the most common use cases.

This ticket is a place to discuss both feature requirements and the interface we want to provide to allow for this.

Requirements

• A visual interface in which to view and analyse previous test runs - this should be a cross-cutting concern of both the headless and visual interface.
• Parity of test configuration options relative to the visual interface - i.e scripting, etc.

Lets addd some more!

[charypar]

I believe 2. and 3. can be merged - i.e. I would leave the repeated execution on the consumer.

I would almost say the CI/CD story is basically an API story - provide a good public REST API to configure and run a test, then get results streamed / fetch them by polling. We can even provide a CLI consuming this API for use in CI automation (and write it in Go, mwahahaha!). But the rest of it - success thresholds, etc., the consumers can do with the API on their own.

The tricky thing about testing the current release and catching regressions is you'd need to first deploy it to production environment (or one that has comparable performance characteristics) in order to have a URL to give to WebAppTest.. this seems almost impossible for most projects, so the use-case as a regression test to stop a build seems relatively minor...

[lpil]

and write it in Go

I vote Rust because monads and types <3