Skip to content

fix(opentelemetry): inject additional_attributes in log phase#13265

Merged
Baoyuantop merged 3 commits into
apache:masterfrom
janiussyafiq:fix/ot-add-attr
Apr 30, 2026
Merged

fix(opentelemetry): inject additional_attributes in log phase#13265
Baoyuantop merged 3 commits into
apache:masterfrom
janiussyafiq:fix/ot-add-attr

Conversation

@janiussyafiq
Copy link
Copy Markdown
Contributor

@janiussyafiq janiussyafiq commented Apr 20, 2026

Description

The opentelemetry plugin supports additional_attributes to inject APISIX variables (e.g. consumer_name, balancer_ip) as span attributes. However, these attributes were silently missing from traces when the variable is set by a lower-priority plugin.

Root cause: additional_attributes were collected during the rewrite phase, where the opentelemetry plugin runs first (priority 12009). At that point, authentication plugins like jwt-auth (priority 2510) have not yet executed, so variables such as consumer_name are still nil and silently skipped by inject_attributes. The same applies to balancer_ip, which is only set after the balancer phase.

Fix: Move additional_attributes and additional_header_prefix_attributes injection to the log phase, where all variables are fully populated. This reuses the existing span:set_attributes() pattern already present in the log
phase for http.status_code and apisix.response_source.

Which issue(s) this PR fixes:

Fixes #12873

Checklist

  • I have explained the need for this PR and the problem it solves
  • I have explained the changes or the new features added to this PR
  • I have added tests corresponding to this change
  • I have updated the documentation to reflect this change
  • I have verified that this change is backward compatible (If not, please discuss on the APISIX mailing list first)

@dosubot dosubot Bot added size:L This PR changes 100-499 lines, ignoring generated files. bug Something isn't working labels Apr 20, 2026
Comment thread t/plugin/opentelemetry.t
Comment on lines +496 to +498
plugins:
- opentelemetry
- key-auth
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

uses key-auth instead of jwt-auth for easier testing setup since the logic is the same (opentelemetry's priority > key-auth's priority)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good

Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Fixes missing OpenTelemetry span attributes sourced from APISIX variables that are only populated after the rewrite phase (e.g., auth-derived consumer_name, balancer-derived vars), by moving attribute injection later in the request lifecycle.

Changes:

  • Move additional_attributes and additional_header_prefix_attributes injection from rewrite to log phase in the opentelemetry plugin.
  • Add a regression test ensuring consumer_name is exported as a span attribute when set by an auth plugin.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.

File Description
apisix/plugins/opentelemetry.lua Shifts additional attribute/header-derived attribute injection to log phase and applies them via span:set_attributes() before finishing the span.
t/plugin/opentelemetry.t Adds tests that create a consumer + key-auth protected route and validates consumer_name appears in exported OTLP span data.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread apisix/plugins/opentelemetry.lua Outdated
Comment thread apisix/plugins/opentelemetry.lua
Copy link
Copy Markdown

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 2 comments.


💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread t/plugin/opentelemetry.t

=== TEST 25: check consumer_name in span attributes
--- exec
tail -n 1 ci/pod/otelcol-contrib/data-otlp.json
Copy link

Copilot AI Apr 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using tail -n 1 plus a broad regex can make the test flaky if another span is exported after the request (or if exports are reordered). To make this deterministic, match the exported record for the triggering request (e.g., grep/filter for the specific X-Request-Id value in the JSON or parse and assert on the span whose request-id attribute matches 01010101010101010101010101010102).

Suggested change
tail -n 1 ci/pod/otelcol-contrib/data-otlp.json
grep '01010101010101010101010101010102' ci/pod/otelcol-contrib/data-otlp.json

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

followed test convention from previous test cases

Comment thread t/plugin/opentelemetry.t
Comment on lines +590 to +603
"plugins": {
"serverless-pre-function": {
"phase": "rewrite",
"functions": ["return function(conf, ctx) ngx.req.set_header('x-injected-by-plugin', 'test-value') end"]
},
"opentelemetry": {
"sampler": {
"name": "always_on"
},
"additional_header_prefix_attributes": [
"x-injected-*"
]
}
},
Copy link

Copilot AI Apr 28, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The test intent is “header added by lower-priority plugin”, but serverless-pre-function priority may not be obviously lower (and could change), which weakens the regression signal. Consider using a plugin with a well-known, clearly lower priority than opentelemetry that can set request headers (e.g., proxy-rewrite), or otherwise ensure the chosen plugin is definitively executed after opentelemetry in rewrite.

Copilot uses AI. Check for mistakes.
Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

serverless-pre-function has lower priority then opentelemetry

Copy link
Copy Markdown
Member

@nic-6443 nic-6443 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@Baoyuantop Baoyuantop merged commit 6577cfa into apache:master Apr 30, 2026
23 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working size:L This PR changes 100-499 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bug: using consumer_name additional_attributes not work in opentelemetry plugins

6 participants