value and from_context not working for header_setter otel extension.

[kadhamecha-conga]

Component(s)

exporter/loki, extension/headerssetter

What happened?

hi team,

i'm not able to set value of resource attributes which are passes in header from agent.
we have agents and collector architecture, where agent is configured to set header level values as X-Scope-OrgID as env name and those are used at collector level to looked up from the value property of the extension configuration.

I tried with value and from_context in collector level , but folder are not creating in s3 as per values set in tenant_id , i'm trying to achieve multi tenant for whole (LGTM) as per dynamic header value?

how to can we check if header values are set? ( i'm using logging exporter with verbosity: detailed ) but it is not showing any header level values.

let me know, if you need more info.

thanks.

Collector version

0.75.0

Environment information

Environment

kubernetes 1.24
otel agent and collector version: 0.75.0

OpenTelemetry Collector configuration

agent config: 

  extensions:
    headers_setter:
      headers:
        - action: upsert
          key: X-Scope-OrgID
          value: tenant_id

  processors:
    batch:
      timeout: 10s  
    resource/loki:
      attributes:
        - action: insert
          key: tenant_id
          value: test-env

  service:
    telemetry:
      metrics:
        address: 0.0.0.0:8888
    extensions:  
      - headers_setter
    pipelines:
      logs:
        exporters:
        - otlphttp
        - logging
        processors:
        - memory_limiter
        - resource/loki
        receivers:
        - otlp
        - filelog

collector config :

  exporters:
    otlphttp:
      endpoint: http://loki-loki-distributed-gateway/loki/api/v1/push
      auth:
        authenticator: headers_setter
      tls:
        insecure: false
        insecure_skip_verify: true

  extensions:
    headers_setter:
      headers:
        - action: upsert
          key: X-Scope-OrgID
          value: tenant_id ( from_context: X-Scope-OrgID ) << tried with both things.....
          
  receivers:
      otlp:
        protocols:
          grpc:
            include_metadata: true
          http:
            include_metadata: true
    extensions:
      - headers_setter
      - memory_ballast
    pipelines:
      logs:
        exporters:
        - otlphttp
        - logging
        processors:
        - memory_limiter
        receivers:
        - otlp

Log output

No response

Additional context

No response

[github-actions]

Pinging code owners:

• exporter/loki: @gramidt @gouthamve @jpkrohling @mar4uk
• extension/headerssetter: @jpkrohling

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[mar4uk]

Did I understand it correctly: you have two collectors running. The first collector sends data to the second collector. Is it correct?

If so, the configuration of the first collector could look like this:

receivers:
  otlp:
    protocols:
      http:
extensions:
  headers_setter:
    headers:
      - action: insert
        key: X-Scope-OrgID
        value: test-env
exporters:
  otlphttp:
    endpoint: "http://otelcol:4318"
    auth:
      authenticator: headers_setter
service:
  extensions: [headers_setter]
  pipelines:
    logs:
      receivers: [otlp]
      exporters: [otlphttp]

The configuration of the second collector:

receivers:
  otlp:
    protocols:
      http:
        include_metadata: true
extensions:
  headers_setter:
    headers:
      - action: insert
        key: X-Scope-OrgID
        from_context: X-Scope-OrgID
exporters:
  loki:
    endpoint: "http://loki:3100/loki/api/v1/push"
    auth:
      authenticator: headers_setter
service:
  extensions: [headers_setter]
  pipelines:
    logs:
      receivers: [otlp]
      exporters: [loki]

The first collector receives data in otlp format, sets header "X-Scope-OrgID: test-env" and exports data to the second collector. The second collector receives "X-Scope-OrgID: test-env" and propagates this header to Loki. Is it what you want to achieve? Let me know if it helps

[kadhamecha-conga]

hi @mar4uk ,

let me clarify again for you, there is agent and collector scenario, where agent send data to collector and collector process the data.

different agents installed on different kubernetes cluster, set X-Scope-OrgID as env value and collector receives data from all those agents , decode it and send it to respective env folder in loki remote storage (s3).

but value and from_context is not working to get value from header.

let me know , if you need more info.

thanks

[mar4uk]

Thank you for the clarification. The agent is the collector, right? You just call it an agent to distinguish two collectors, right?

In the example configs I posted value and from_context are working to get value from the header. The first config is for an agent. Isn't it what you try to achieve? header propagation works for that example

[kadhamecha-conga]

hi,

the shared config is not working to set value from resource attributes, which I passed as test-env. I'm expecting the value to be fetch from resource attributes not the field itself.

` headers_setter:
headers:
- action: insert
key: X-Scope-OrgID
value: tenant_id << value from resource attributes.

resource/loki:
  attributes:
    - action: insert
      key: tenant_id
      value: test-env

`
instead of test-env , it should set value from resource attribute tenant_id and same should be passed to loki to create per tenant folder in s3 .

[mar4uk]

I'm not sure we can use the resource attribute that is set in the processor in the extension. Need to double-check it. But the docs say:

Secondarily, there are extensions, which provide capabilities that can be added to the Collector, but which do not require direct access to telemetry data and are not part of pipelines. They are also enabled within the service section.

I assume headers_setter extension doesn't have access to the resource attributes.
@jpkrohling has more insight maybe

[jpkrohling]

Extensions don't have direct access to pipeline data. It can come only as connection metadata (ie, request header), or static values.

[crobert-1]

Closing based on @jpkrohling's comment. Please let us know if there's anything else we can help with!