Provide a Honeycomb context to applications run inside the workflow

[kvrhdn]

Applications run during the workflow might be "Honeycomb-aware", i.e. they can send traces their own traces to Honeycomb. By providing these application additional context from gha-buildevents, it might be possible to create one detailed trace.

Prior art:

• puppetlabs/rspec_honeycomb_formatter checks for an environment variable HTTP_X_HONEYCOMB_TRACE.
  See: rspec_honeycomb_formatter.rb
  Post on slack: https://honeycombpollinators.slack.com/archives/CKWR1DRMM/p1588255603033700?thread_ts=1588179870.032600&cid=CKWR1DRMM

[DavidS]

to record my answers from slack:

I'm using the default serialisation from beeline(-ruby):
https://github.com/honeycombio/beeline-ruby/blob/344928cbfb44269aed23ab9be099cda79a7bab95/lib/honeycomb/propagation.rb#L54-L64

for the dataset and api key config, I'm using the default environment variables used in
https://github.com/honeycombio/beeline-ruby/blob/344928cbfb44269aed23ab9be099cda79a7bab95/lib/honeycomb/configuration.rb#L15-L21

[kvrhdn]

I've been able to attach the traces from a command to the trace from buildevents. All the data is there, we just have to polish this a bit / define a standard way to propagate context through the environment.

The API key and dataset are already present in the environment since buildevents needs this as well. API key = BUILDEVENT_APIKEY, dataset = BUILDEVENT_DATASET.
The trace ID corresponds to the build ID, which is exported by gha-buildevents as BUILD_ID.
The parent ID corresponds to the step ID.

My PoC implementation: https://github.com/kvrhdn/tfe-run/blob/honeycomb/main.go#L29

End result, this workflow run: https://github.com/kvrhdn/tfe-run/runs/639591915?check_suite_focus=true
Created this trace 🎉

[kvrhdn]

I've iterated a bit more on this, I now create a 'trace propagation context' string in gha-buildevents. This is a string the contains the trace ID, the dataset and more. Example:

1;trace_id=725,parent_id=725,dataset=gha-buildevents_integration,context=

I export this to the environment as HONEYCOMB_TRACE_CONTEXT. (this is basically the same thing as HTTP_X_HONEYCOMB_TRACE)

You can import it really easy using most beelines since this normally parse from HTTP headers.
Example in Go: https://github.com/kvrhdn/tfe-run/blob/honeycomb/main.go#L51

Advantages of this system:

• It's fairly standardized, every beeline should be able to parse this.
  Besides, I think it should be possible to extend most beelines with a function GetTraceFromEnvironment.
• It's high bandwidth: one string contains all data (except for the API key, but I'd argue it's best to inject this per step)

Disadvantages:

• This is not really compatible with how buildevents works right now. The commands cmd and step expect you to manually define a STEP_ID. It does not check for an environment variable nor does it set one.
  We could make buildevents cmd propagation context-aware, but I'm not sure how this would work with step...

A workaround would be to not use buildevents step anymore, in favour of a custom action like gha-buildevent-step #5

[kvrhdn]

I'm closing this issue here since I'm in favor of implementing this in buildevents itself, see honeycombio/buildevents#74.