[BUG] OpenTelemetry logging does not seem to work

[Daniel-Vaz]

📋 Prerequisites

• I have searched the existing issues to avoid creating a duplicate
• By submitting this issue, you agree to follow our Code of Conduct
• I am using the latest version of the software
• I have tried to clear cache/cookies or used incognito mode (if ui-related)
• I can consistently reproduce this issue

🎯 Affected Service(s)

Multiple services / System-wide issue

🚦 Impact/Severity

Minor inconvenience

🐛 Bug Description

When deploying kagent v0.7.13 with OpenTelemetry (OTel) tracing and logging enabled, tracing is successfully exported to the configured OTLP endpoint, but logs are not being sent to the OpenTelemetry collector.

No errors appear in the OTel collector logs, yet logs never arrive in the collector (and therefore never reach Loki).

Additionally, the kagent runtime repeatedly emits the following warning in the agent logs:

/.kagent/.venv/lib/python3.13/site-packages/opentelemetry/sdk/_events/__init__.py:53: 
LogDeprecatedInitWarning: LogRecord init with `trace_id`, `span_id`, and/or `trace_flags` is deprecated since 1.35.0. 
Use `context` instead.
  log_record = LogRecord(

This suggests that kagent may be using a deprecated OpenTelemetry logging API that could be preventing proper log export.

---

Configuration Used

otel:
  tracing:
    enabled: true
    exporter:
      otlp:
        endpoint: http://obs-agent-ai-signals.observability.svc.cluster.local:4317
        timeout: 15
        insecure: true
  logging:
    enabled: true
    exporter:
      otlp:
        endpoint: http://obs-agent-ai-signals.observability.svc.cluster.local:4317
        timeout: 15
        insecure: true

$ kubectl get svc obs-agent-ai-signals -n observability

NAME                   TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                       AGE
obs-agent-ai-signals   ClusterIP   172.17.236.35   <none>        12345/TCP,4317/TCP,4318/TCP   60m

---

🔄 Steps To Reproduce

1. Deploy kagent v0.7.13 in a Kubernetes cluster.

2. Enable OpenTelemetry tracing and logging using the configuration above.

3. Confirm that:

   • Traces are successfully received by the OpenTelemetry collector.

4. Interact with the agent and observe kagent logs in the pod.

5. Check the OpenTelemetry collector and downstream Loki for logs from kagent.

🤔 Expected Behavior

OTel Logs should work as intended.

📱 Actual Behavior

OTel Tracing works, but not Logs.

💻 Environment

• Kagent version v0.7.13
• Kubernetes cluster version: v1.34.3
• Kubernetes provider: kubeadm

🔧 CLI Bug Report

No response

🔍 Additional Context

I followed the following docs - https://kagent.dev/docs/kagent/observability/audit-prompts

The main differences were:

• We already had a Tempo and Loki installation up and running;
• We didn't use a otel collector like the example, instead we used Grafana Alloy with the otelcol.receiver.otlp component (which is basically a wrapper around the OTel Collector).
  • For reference the full Alloy config is the following:

    otelcol.receiver.otlp "otlp" {
      http {}
      grpc {}

      output {
        logs= [otelcol.processor.attributes.default.input]
        traces= [otelcol.processor.filter.drop_health_checks.input]
      }
    }

    otelcol.exporter.otlphttp "loki" {
      client {
        endpoint = "http://obs-agent-logs.monitoring-system.svc.cluster.local:4317"
        tls {
          insecure             = true
          insecure_skip_verify = true
        }
      }
    }

    otelcol.exporter.otlp "traces" {
      client {
        endpoint = "http://obs-agent-traces.monitoring-system.svc.cluster.local:4317"
        tls {
          insecure             = true
          insecure_skip_verify = true
        }
      }
    }

📋 Logs

/.kagent/.venv/lib/python3.13/site-packages/opentelemetry/sdk/_events/__init__.py:53: 
LogDeprecatedInitWarning: LogRecord init with `trace_id`, `span_id`, and/or `trace_flags` is deprecated since 1.35.0. 
Use `context` instead.
  log_record = LogRecord(

📷 Screenshots

No response

🙋 Are you willing to contribute?

• I am willing to submit a PR to fix this issue

[Daniel-Vaz]

Changed one agent logging to debug mode and noticed this during the initialization:

2026-02-09 11:58:54,600 - root - INFO - Enabling tracing
2026-02-09 11:58:54,600 - root - INFO - Trace endpoint: http://obs-agent-ai-signals.observability.svc.cluster.local:4317
2026-02-09 11:58:54,602 - root - INFO - Created new TracerProvider
2026-02-09 11:58:54,605 - root - INFO - Enabling logging for GenAI events
2026-02-09 11:58:54,605 - root - INFO - Log endpoint configured: None
2026-02-09 11:58:54,605 - root - INFO - Log provider configured with OTLP
2026-02-09 11:58:54,605 - root - INFO - OpenAI instrumentation configured with event logging capability

But the controller configmap seems to have the needed values there properly passed through the helm values:

data:
  A2A_BASE_URL: http://observability-kagent-controller.observability.svc.cluster.local:8083
  DATABASE_TYPE: postgres
  DEFAULT_MODEL_CONFIG_NAME: default-model-config
  IMAGE_PULL_POLICY: IfNotPresent
  IMAGE_REGISTRY: cr.kagent.dev
  IMAGE_REPOSITORY: kagent-dev/kagent/app
  IMAGE_TAG: 0.7.13
  KAGENT_CONTROLLER_NAME: observability-kagent-controller
  LEADER_ELECT: "true"
  OTEL_EXPORTER_OTLP_ENDPOINT: http://obs-agent-ai-signals.observability.svc.cluster.local:4317
  OTEL_EXPORTER_OTLP_LOGS_ENDPOINT: http://obs-agent-ai-signals.observability.svc.cluster.local:4317
  OTEL_EXPORTER_OTLP_LOGS_INSECURE: "true"
  OTEL_EXPORTER_OTLP_LOGS_TIMEOUT: "15"
  OTEL_EXPORTER_OTLP_TRACES_INSECURE: "true"
  OTEL_EXPORTER_OTLP_TRACES_TIMEOUT: "15"
  OTEL_LOGGING_ENABLED: "true"
  OTEL_TRACING_ENABLED: "true"
  OTEL_TRACING_EXPORTER_OTLP_ENDPOINT: http://obs-agent-ai-signals.observability.svc.cluster.local:4317
  POSTGRES_DATABASE_URL: postgres://postgres:kagent@pgsql-postgresql.kagent.svc.cluster.local:5432/postgres
  STREAMING_INITIAL_BUF_SIZE: 4Ki
  STREAMING_MAX_BUF_SIZE: 1Mi
  STREAMING_TIMEOUT: 600s
  WATCH_NAMESPACES: ""
  ZAP_LOG_LEVEL: info

[Daniel-Vaz]

Seems that you're checking for the wrong env var.

Here, the env var OTEL_LOGGING_EXPORTER_OTLP_ENDPOINT is checked, yet the helm templates populate the OTEL_EXPORTER_OTLP_LOGS_ENDPOINT var instead

[krisztianfekete]

Hi @Daniel-Vaz, thanks for opening the issue!
The deprecation warning shouldn't be causing any problems, but we will be cleaning that up!

• Logging input/output as log events is only supported for OpenAI and Anthropic models at the moment. Can you confirm what models are you using when you're not seeing logs passing though?

• If you're using supported models then as the next step, I'd check the agent logs as those should be logging errors if exporting fails for whatever reason. Can you confirm that you have no such logs?

• Finally, I'd add a debugexporter (or its Alloy equivalent) into your Alloy pipeline to validate that logs are reaching Alloy.

[Daniel-Vaz]

Hello @krisztianfekete

Thank you for the feedback 😄
To answer your question, I'm using Azure OpenAi gpt-5-mini.

Did you notice my latest comment where I've identified what I think is the root issue on the kagent helm chart ?

[Daniel-Vaz]

Uhmm but now I see I'm also the one not paying enough attention to your comment...

I was wrongly assuming compatibility between Azure OpenAI and OpenAI perhaps. Could that be the reason ?
Either way the env var differences I mentioned are strange. Could you comment on it ?

[krisztianfekete]

It shouldn't matter as it's falling back to the correct endpoint as you can also see in your env vars (would you be interested in cleaning up the code so others won't get confused?). I just tried with the same version as you, and can see logs via OpenAI models in debug logs in a standard OTel collector.

LogRecord #1
ObservedTimestamp: 2026-02-10 11:12:14.465858034 +0000 UTC
Timestamp: 2026-02-10 11:12:14.465849451 +0000 UTC
SeverityText:
SeverityNumber: Info(9)
Body: Map({"content":"testing logs\n"})
Attributes:
     -> gen_ai.system: Str(openai)
     -> event.name: Str(gen_ai.user.message)
Trace ID:
Span ID:
Flags: 0
LogRecord #2
ObservedTimestamp: 2026-02-10 11:12:16.288339382 +0000 UTC
Timestamp: 2026-02-10 11:12:16.288311631 +0000 UTC
SeverityText:
SeverityNumber: Info(9)
Body: Map({"finish_reason":"stop","index":0,"message":{"content":"You mentioned \"testing logs.\" Could you please clarify which Kubernetes resource or pod logs you want to test or inspect? \n\nFor example: \n- Are you looking to retrieve logs from a specific pod or container?\n- Is there a particular namespace or application for which you want to see logs?\n- Are you troubleshooting an issue related to logs?\n\nPlease provide more details so I can assist you effectively."}})
Attributes:
     -> gen_ai.system: Str(openai)
     -> event.name: Str(gen_ai.choice)
Trace ID:

Have you had a chance to look exporting errors in agents and in Alloy? Azure OpenAI and OpenAI should be the same instrumentation-wise. Would you care to share you're ModelConfig too?

[Daniel-Vaz]

I stand corrected, indeed It's a issue on my side and when I started troubleshooting things I noticed those warning logs and helm values differences and mistakenly thought it could be something on the kagent side.

With Alloy live debug mode I noticed that indeed we were receiving Otel log entries and writing them to Loki.

Regarding your question if I could contribute with code changes, I would honestly prefer not to, because I don't have much programming experience. I'll explore this a bit further and let you know.

Either way I really appreciate you taking your time explaining things and helping me along this troubleshooting journey. You guys have here a very promising project. Thank you very much. I'll close this issue for now.

[Daniel-Vaz]

@krisztianfekete
Gave it a try addressing the env vars confusion in #1292 .
Could I get your feedback on it, when you get a chance ? Thanks in advance.