Refactor workload-lifecycle e2e to be more fault tolerant

[damemi]

The workload-lifecycel e2e can flake because it only runs the traffic generation job once, then loops checking for traces. If auto-instrumentation isn't actually running for a service before the job is generated, the loop is doomed to fail.

This changes how the workload-lifecycle e2e generates test traces and counts them:

• Instead of creating the traffic job once, create it on every loop that checks for traces
• When checking for traces, use a new jq filter to count the unique occurance of each service name

Unlike the source e2e, which can just check for a minimum number of spans, we have to aggregate the unique services in these checks. That's because in the source e2e, all of the services are tied in a single trace. In this test, the services are separate, so the loop could generate multiple traces for the same service while waiting for others to be ready. This means we can't rely on just checking minimum, because we might hit that minimum even if all services haven't sent traces yet.

We might be able to eventually drop the custom_jq field and just make that the default, but because other jobs might be using the traceql_runner.sh script I'm not doing that yet to not break anything else.

[BenElferink]

Very nice! 🏆
Do you think this loop can be implemented for cli-upgrade as well?

[RonFed]

The reason we are generating the traffic once is since (at leas in theory) we should have made all the necessary asserts that make sure that all the services are instrumented and Odigos is ready. If that is not the case (which is probably what happens today) - it means we either missing an assert or we have bugs.
Those asserts also reflect what we present to the users - if all the InstrumentationInstances are healthy etc' they should expect traces.
This change, although reducing the flakiness - might hide some of the bugs from us IMO.

[RonFed]

see my comment above

removed approval due to requested changes

[github-actions]

This PR is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[github-actions]

This PR was closed because it has been stale for 30 days with no activity.