How to verify if Triton is really sending OTel traces to collector?

[ggiuffre]

I'm trying to instrument Triton to send OpenTelemetry traces to an OpenTelemetry-enabled DataDog agent.

I run Triton as a Docker container from nvcr.io/nvidia/tritonserver:25.02-py3 with the following command:

/opt/tritonserver/bin/tritonserver \
    --model-repository=${MODEL_REPO_S3_PATH} \
    --trace-config mode=opentelemetry \
    --trace-config rate=1 \
    --trace-config level=TIMESTAMPS \
    --trace-config opentelemetry,url=http://172.17.0.1:4318

Triton answers inference requests correctly with the above setup.

In order to confirm that Triton is sending traces I temporarily set the option --trace-config opentelemetry,url=http://localhost:4318 (notice localhost instead of 172.17.0.1) and tried to run a minimal OTel collector inside the same container where Triton is running:

# install OTel collector:
wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.126.0/otelcol_0.126.0_linux_amd64.deb
dpkg -i otelcol_0.126.0_linux_amd64.deb

# declare minimal OTel collector config:
cat > collector-dump.yaml <<-EOF
receivers:
  otlp:
    protocols:
      grpc:    # default localhost:4317
      http:    # default localhost:4318
exporters:
  debug:
service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [debug]
EOF

# start OTel collector:
otelcol --config collector-dump.yaml

With both Triton and this minimal OTel collector running inside the same container, I then sent an inference request from a Python Triton client. The request was served correctly but did not produce any output in the otelcol output.

Why do I not see any output from otelcol when Triton processes an inference request? Is it possible that Triton is not sending traces at all with my configuration?

I looked at another very similar discussion in this repo, but my problem is likely different.

[ggiuffre]

When pointing Triton to http://172.17.0.1:4318 I get this log about tracing:

[Error] File: /tmp/tritonbuild/tritonserver/build/_deps/repo-third-party-build/opentelemetry-cpp/src/opentelemetry-cpp/exporters/otlp/src/otlp_http_client.cc:103 [OTLP HTTP Client] Export failed, Status:404, Header:	Content-Length: 19
,	Content-Type: text/plain; charset=utf-8
,	Date: Fri, 23 May 2025 08:35:08 GMT
,	X-Content-Type-Options: nosniff
,Body:404 page not found

But when I point Triton to http://172.17.0.1:4318/v1/traces I don't get any logs at all about tracing.

[ggiuffre]

The problem was just that I had to add /v1/traces to the URL in the options. See this other issue.

If anybody comes across this post, remember to also add your deployment environment to the OpenTelemetry tracing options: --trace-config opentelemetry,resource=deployment.environment.name=${ENVIRONMENT}. The issue for me was that the DataDog trace explorer UI automatically added env:dev to my search terms, but Triton did not have an env set. Just removing env:dev from the search terms in the UI (or setting the above option in Triton with ENVIRONMENT=dev) solved the mystery.

One more thing that might be useful to future readers: Triton seems to only emit logs at startup. So if you don't see logs, in theory that's a good sign 😅