Tetragon e2e Test: Run e2e tests in parallel on multiple runners

[Trung-DV]

E2e tests currently have 3 packages that can run independently with each other

By split the test into 3 runners, we can reduce about ~50% CI time: https://github.com/Trung-DV/tetragon/actions/runs/8771741177

[kkourt]

Hi,

Thanks for the PR, but it's not clear to me that this will work as expected.

Each e2e test requires its own Tetragon instance, and having multiple Tetragon instances at the same time can be problematic. This is why we generally running the tests sequentially.

[Trung-DV]

Hi,

Thanks for the PR, but it's not clear to me that this will work as expected.

Each e2e test requires its own Tetragon instance, and having multiple Tetragon instances at the same time can be problematic. This is why we generally running the tests sequentially.

Hi @kkourt
As I understand, currently, we run e2e test sequentially, we will init a Tetragon instance for each test package at a time.
So I just split out the runners, init a Tetragon instance for each runner, and each runner will a single test package.
All are independent environments (runners).
Is my approach reasonable?

[mtardy]

That's a good idea thanks! The only downside I could see is that your workflow might require lots of runners and sometimes it's a bottleneck when GitHub or actuated doesn't provide us with enough runners. But that should be fine.

Is there a way we could make this easier to see which is which?

Perhaps you could use jobs.<job_id>.name with the last folder name of the matrix path, or the entire path.

Like:

run-e2e-test:
  name: ${{ <your path> }}

[netlify]

✅ Deploy Preview for tetragon ready!

Name	Link
🔨 Latest commit	87ed230
🔍 Latest deploy log	https://app.netlify.com/sites/tetragon/deploys/662798b8add0fc0008d07d84
😎 Deploy Preview	https://deploy-preview-2354--tetragon.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[Trung-DV]

I have updated
now it looks like https://github.com/Trung-DV/tetragon/actions/runs/8799449115

[mtardy]

Thanks I like it, I think it's a great improvement!

[mtardy]

In addition to my previous comments, if you can squash your commits maybe! Thanks!

[mtardy]

Please @willfindlay take a look and feel free to merge this!

[willfindlay]

Looks amazing, I just have one nitpick.

[mtardy]

Thanks!!

[alexellis]

How many more jobs would you expect this to generate @mtardy?

Anything that could generate dozens or hundreds of jobs will probably be misusing "the commons" resource, if it's a couple more that is unlikely to make much difference.

[mtardy]

How many more jobs would you expect this to generate @mtardy?

Anything that could generate dozens or hundreds of jobs will probably be misusing "the commons" resource, if it's a couple more that is unlikely to make much difference.

You are everywhere 👀 , it just transforms a 20min job into three jobs that are running significantly faster (something around 10min each) so we consume a bit more generally but it's also faster in a way. It should be reasonable as it asks for 4 CPUs and 16GB. We have been seeing bottlenecks on some occasions on the providing on actuated side, we hope our consumption is reasonable otherwise, we can find solutions.

[alexellis]

We've had a couple of users starting matrix builds with 250+ jobs .. this of course causes some issues for the shared service.

What you're doing sounds fine.

You can right-size VMs by running vmmeter - https://actuated.dev/blog/right-sizing-vms-github-actions