Skip to content

fix: separate OTel provider provisioning from span generation#3880

Merged
ss1909 merged 1 commit intomainfrom
sneha/seperate-otel-functionality
May 4, 2026
Merged

fix: separate OTel provider provisioning from span generation#3880
ss1909 merged 1 commit intomainfrom
sneha/seperate-otel-functionality

Conversation

@ss1909
Copy link
Copy Markdown
Contributor

@ss1909 ss1909 commented May 4, 2026

Description

Problem

setupLoggerAndConfig only initialised the OpenTelemetry TracerProvider when TRACEPARENT was set in the environment. This was unreliable: TRACEPARENT is injected into every subprocess the agent spawns (git, bootstrap, hooks) so its presence is not a reliable signal that the current process is a traced subcommand.

This also caused a double-initialisation bug in the bootstrap flow: setupLoggerAndConfig created one provider, and startTracingOpenTelemetry created a second, loosing the first one.

Fix

Separate the two concerns:

  • setupLoggerAndConfig always initialises the provider when BUILDKITE_TRACING_BACKEND=opentelemetry, and owns its lifecycle (ForceFlush + Shutdown in done())
  • startTracingOpenTelemetry uses otel.GetTracerProvider() to obtain the already-registered provider and only creates the root job span, adding job-specific attributes (pipeline slug, job ID, etc.) at span level rather than resource level since they are not available at provider init time

Context

Changes

Testing

  • Tests have run locally (with go test ./...). Buildkite employees may check this if the pipeline has run automatically.
  • Code is formatted (with go tool gofumpt -extra -w .)

screenshots on grafana to show the subcommands:
image

Disclosures / Credits

@ss1909 ss1909 force-pushed the sneha/seperate-otel-functionality branch from 0d31195 to 6bf593e Compare May 4, 2026 01:22
@ss1909 ss1909 marked this pull request as ready for review May 4, 2026 01:30
@ss1909 ss1909 requested review from a team as code owners May 4, 2026 01:30
@ss1909 ss1909 force-pushed the sneha/seperate-otel-functionality branch 2 times, most recently from eddea5b to 31b350b Compare May 4, 2026 03:33
Copy link
Copy Markdown
Contributor

@zhming0 zhming0 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM ✅ with some nits. Well done 👍🏿

Comment thread internal/job/tracing.go Outdated
Comment thread internal/job/tracing.go Outdated
@ss1909 ss1909 force-pushed the sneha/seperate-otel-functionality branch from 0758991 to ca8ddc5 Compare May 4, 2026 04:17
@ss1909 ss1909 merged commit 691418d into main May 4, 2026
3 checks passed
@ss1909 ss1909 deleted the sneha/seperate-otel-functionality branch May 4, 2026 05:21
@buildkate buildkate mentioned this pull request May 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants