Skip to content

Conversation

Copy link
Contributor

Copilot AI commented Jan 19, 2026

The deferred export mechanism in workflow instrumentation was broken because OTLP headers weren't properly formatted when forwarding telemetry data to the actual endpoint.

Root Cause

OTLP headers are provided as comma-separated key=value pairs (e.g., Authorization=Bearer token,X-Custom=value), but curl's -H @file syntax expects each header on a separate line in key: value format. The workflow instrumentation was writing the raw comma-separated string to the headers file, which curl couldn't parse correctly.

Changes

actions/instrument/workflow/main.sh - Transform headers to proper format before passing to curl:

# Before (broken):
echo "$OTEL_EXPORTER_OTLP_HEADERS" >> "$headers"
echo "$OTEL_EXPORTER_OTLP_LOGS_HEADERS" >> "$headers"

# After (fixed):
echo "$OTEL_EXPORTER_OTLP_HEADERS","$OTEL_EXPORTER_OTLP_LOGS_HEADERS" | tr ',' '\n' | grep -v '^$' | sed 's/=/: /g' >> "$headers"

The transformation:

  1. Combines general (OTEL_EXPORTER_OTLP_HEADERS) and signal-specific headers (e.g., OTEL_EXPORTER_OTLP_LOGS_HEADERS)
  2. Splits comma-separated headers into separate lines (tr ',' '\n')
  3. Filters out empty lines (grep -v '^$')
  4. Converts key=value to key: value format (sed 's/=/: /g')

This matches the exact pattern used in the job-level instrumentation for the collector configuration, ensuring consistency across the codebase.

Impact

When export_deferred_signal_file forwards deferred telemetry data via curl, it now correctly passes headers in the format curl expects, allowing the data to be successfully sent to the OTLP endpoint with proper authentication and configuration.

Original prompt

This section details on the original issue you should resolve

<issue_title>Deferred export is broken</issue_title>
<issue_description>Somehow the deferred export in the workflow instrumentaiton is broken</issue_description>

<agent_instructions>analayze both the code in the job level instrumentaiton and the forwarding code in the workflow instrumentation carefully, find the problem and fix it. the purprose of the deferred export is that it takes the payload, and then sends it later with proper credentials.</agent_instructions>

Comments on the Issue (you are @copilot in this section)


💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.

Copilot AI and others added 2 commits January 19, 2026 19:13
Co-authored-by: plengauer <100447901+plengauer@users.noreply.github.com>
Co-authored-by: plengauer <100447901+plengauer@users.noreply.github.com>
Copilot AI changed the title [WIP] Fix broken deferred export in workflow instrumentation Fix deferred export by exporting environment variables for parallel execution Jan 19, 2026
Copilot AI requested a review from plengauer January 19, 2026 19:18
Copy link
Owner

@plengauer plengauer left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@copilot that cant be it. they are already exported from the start, so unless you can show me a specific place where they are unexported, they are available. re-analyze and find the real issue

Co-authored-by: plengauer <100447901+plengauer@users.noreply.github.com>
Copy link
Contributor Author

Copilot AI commented Jan 19, 2026

@copilot that cant be it. they are already exported from the start, so unless you can show me a specific place where they are unexported, they are available. re-analyze and find the real issue

You're absolutely right - the variables were already exported. The real issue was in the header formatting.

The OTLP headers are comma-separated key=value pairs, but curl's -H @file syntax expects each header on a separate line in key: value format. The workflow code was just echoing the raw comma-separated string, which curl couldn't parse.

Fixed in commit 41df900 by applying the same header transformation used in the job-level instrumentation:

echo "$OTEL_EXPORTER_OTLP_HEADERS" | tr ',' '\n' | grep -v '^$' | sed 's/=/: /g' >> "$headers"

This splits by comma, filters empty lines, and converts = to : format.

Co-authored-by: plengauer <100447901+plengauer@users.noreply.github.com>
Copilot AI changed the title Fix deferred export by exporting environment variables for parallel execution Fix deferred export by properly formatting OTLP headers for curl Jan 19, 2026
Copilot AI requested a review from plengauer January 19, 2026 19:43
@plengauer plengauer marked this pull request as ready for review January 19, 2026 19:51
@plengauer plengauer requested a review from moflwi as a code owner January 19, 2026 19:51
Copilot AI review requested due to automatic review settings January 19, 2026 19:51
@plengauer plengauer enabled auto-merge (squash) January 19, 2026 19:51
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR fixes the deferred export mechanism in workflow instrumentation by properly formatting OTLP headers for curl. The issue was that headers were provided as comma-separated key=value pairs but curl requires key: value format with each header on a separate line.

Changes:

  • Transform OTLP headers from comma-separated format to curl-compatible format with proper key: value syntax
  • Combine signal-specific headers with general headers before transformation
  • Remove the redundant separate echo of OTEL_EXPORTER_OTLP_HEADERS

@plengauer plengauer merged commit 6afeca6 into main Jan 22, 2026
557 checks passed
@plengauer plengauer deleted the copilot/fix-deferred-export-issue branch January 22, 2026 17:28
plengauer added a commit that referenced this pull request Jan 22, 2026
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Deferred export is broken

3 participants