Iterating on e2e tests locally

[jhchabran]

Right now, it's not very practical to iterate on e2e tests locally, for two reasons:

• Most of the tests requires precise environment variables which are only available in CI or stored in secrets. Unless you're the owner of the test in question, we recommend you to iterate in CI if you're fixing a test or adding a new one.
• For the moment, it's not always possible to run such tests on a MacOS machine. Building the server image, which runs a linux/amd64 OS, requires cross-compiling the code. This is mostly fine, but some test binaries also requires compilation (the backend integration tests for example, are written in Go), which in turns requires to build those binaries targeting the host platform and not linux/amd64.

Still, writing e2e test is a critical to ensure we can ship non breaking code. Because we're not able to do so without a lot of friction, teammates tend to avoid writing new e2e tests or fall back to disabling the flakiest ones.

We want to flip that around and provide a nice experience, where teammates doesn't have to do the plumbing themselves, and while it may not be fast as such tests are always slow, at least make it so it's a pleasant experience.

Plan

Tracked issues

Legend

• 👩 Customer issue
• 🐛 Bug
• 🧶 Technical debt
• 🎩 Quality of life
• 🛠️ Roadmap
• 🕵️ Spike
• 🔒 Security issue
• 🙆 Stretch goal