Run benchmark from CI once a week

[trask]

Run both coldstart and runtime benchmark from:

https://github.com/trask/agent-benchmarking

These spin up a number of Azure App Service VMs and report average and median in order to get a more stable result than you can get on a single cloud machine.

On each VM, the tests alternate enabling/disabling the agent, capturing the overhead that enabling the agent incurs.

[iNikem]

Are you sure about running this on every merge? My concern is the cost :) May be run once a week?

[trask]

Agree, will plan to start with once a week. Then we can run the benchmark for longer across more machines and get a more stable result.

[anuraaga]

Should we start by exporting to a collector within GitHub actions itself? Just spam the spring-boot smoke test container? I don't know if the numbers are reliable or not, but it seems like an easier start than provisioning cloud.

[anuraaga]

For reference, I've written this benchmark with testcontainers before for Zipkin, it's basically awk + zipkin + backend + prometheus. https://github.com/openzipkin/zipkin/blob/master/benchmarks/src/test/java/zipkin2/server/ServerIntegratedBenchmark.java

[trask]

cc: @breedx-splk ready made issue 😁

[iNikem]

We now have https://github.com/open-telemetry/opentelemetry-java-instrumentation/actions/workflows/nightly-benchmark-overhead.yml, should we close this issue @trask ?