A quick scraper program that let you export builds on BuildKite as OpenTelemetry data and then send them to honeycomb.io for slice-n-dice high cardinality analysis.
This program is using 100% OpenTelemetry SDK so Honeycomb can be swapped out with any Tracing/Metrics/APM providers that support OpenTelemetry protocol (DataDog, Grafana Tempo etc...)
This was built as a quick POC / MVP for my daily use cases but PRs/Issues are more than welcome.
It's definitely more efficient to push traces on each pipeline run than to poll for build result and creating cache manually each run. There are several BuildKite plugins that enable generating traces via different hooks.
However, I picked polling approach instead for these reasons:
-
Keep It Simple and Stupid(KISS): This allows us to keep the complexity of creating traces away from our pipeline logic and thus, reduce the complexity in maintaining an already complex system.
-
Being able to get past/old data: This polling approach allow us to send traces for old data as well as new. Having old data is valuable to use as base line of comparision.
-
Being able to create traces without wasting CI compute power: In the future, we might want to analyze and add our profiling artifacts as traces. These artifacts can be heavy and take time to analyze. Decoupling the creation of profile-traces way from CI pipeline keep the pipeline fast and performance.
There are definitely some cons to this approach:
-
Developers cannot freely add more traces to the pipeline.
-
Require separate compute to run this exporter.
Feel free to pick the tradeoffs that is right for your use case.
Totally inspired by https://github.com/zoidbergwill/gitlab-honeycomb-buildevents-webhooks-sink