Populating Langfuse trace input/output fields in the Langfuse UI

[edeandrea]

Describe your question

In a distributed system, traces and spans can come in from different applications. Each of those applications could be interacting as part of the same trace. I've noticed in the trace/session view in the Langfuse UI. I've noticed that the input & output fields of a trace in the Langfuse UI are only populated if the input/output span attributes are populated into the root span of the entire trace.

How is that supposed to work in practice? In a distributed system, one application may begin the trace, but the actual LLM operation may occur in another application where the parent trace/span is propagated. That application will add the appropriate spans to the trace with the proper attributes.

But the data won't surface in the main Traces view in Langfuse because those gen_ai.* attributes don't exist on the root span. And it would be pretty impossible to "copy" up the attributes to the root level, since the application computing the attribute values wouldn't have access to the entire trace tree, since each application is exporting the spans on their own.

Especially in an agentic system - none of the agents (that actually deal with the llm) are generally ever the system that creates the root span.

One way around this would be to start a new trace at each llm operation, but then you would lose the overall visibility of where the llm operation occurred in the overall request flow. We'd still want to export data to Tempo/Loki/Prometheus as well, since Langfuse is really only interested in the trace data - not exemplars/logs/metrics. We'd still need/want a way to correlate any operation in a trace to its metrics/logs, something that those other tools do, and do well.

Any thoughts/suggestions?

I asked Langfuse AI and it didn't have a good answer.

Langfuse Cloud or Self-Hosted?

Self-Hosted

If Self-Hosted

3.172.1 OSS

If Langfuse Cloud

No response

SDK and integration versions

OpenTelemetry 1.60.1 (Java SDK)

Pre-Submission Checklist

• I have checked for existing issues/discussions and consulted Langfuse AI.

[inkeep-ai-assistant[bot]]

You've identified a real architectural challenge with Langfuse's current trace input/output model in distributed systems. By default, trace input/output is copied from the observation at the top level (the "root observation"), which means if your root observation has no input/output, your trace will be empty unless you set it explicitly.(1)

The Core Problem

In distributed agentic systems, the root span is typically created by an orchestrator or API gateway, while the actual LLM operations happen in downstream services that have no access to modify the root span. The documentation acknowledges this limitation exists.(1)

Available Workarounds

1. Propagate Attributes via OpenTelemetry Baggage

For trace-level attributes like userId, sessionId, metadata, version, release, and tags, you can use OpenTelemetry Baggage with a BaggageSpanProcessor to propagate these across all spans in a trace.(2)

However, there's a critical security consideration: OpenTelemetry baggage is propagated across service boundaries and to third-party APIs. Do not include sensitive information (passwords, API keys, personal data, etc.) in baggage when using this approach.(2)

2. Set Trace Input/Output Explicitly

For cases where the trace input/output should be different from the root observation, you can use set_trace_io() from within a child span:(1)

root_span.set_trace_io(
    input={"user_question": user_input},
    output={"answer": result}
)

(1)

However, this method is deprecated and only exists for backward compatibility with trace-level LLM-as-a-judge evaluators.(1)

3. Use Langfuse-Specific Span Attributes

For OpenTelemetry integrations, you can set trace-level attributes using the langfuse.* namespace on any span in the trace:(2)

with tracer.start_as_current_span("Langfuse Attributes") as span:
    span.set_attribute("langfuse.trace.input", json.dumps(input_data))
    span.set_attribute("langfuse.trace.output", json.dumps(output_data))

(2)

The trace-level attributes langfuse.trace.input and langfuse.trace.output can be set on any span in the trace and will be applied to the trace record in Langfuse.(2)

Limitations

The fundamental tension you've identified remains: Langfuse's model assumes the service performing LLM operations can access and modify trace-level context. The documentation doesn't provide a clean solution for distributed scenarios where meaningful trace-level attributes are generated by non-root services.

For dual-exporting to Tempo/Loki/Prometheus alongside Langfuse, you can configure an OpenTelemetry Collector to export to multiple backends, with Langfuse receiving traces via its /api/public/otel endpoint.(2)

This appears to be a known architectural limitation. Consider opening a feature request on GitHub to discuss potential solutions for distributed agentic systems.

📚 Sources:

1. Why are the input and output of my trace empty?
2. OpenTelemetry (OTEL) for LLM Observability

---

Have another question? Just tag @inkeep.

[edeandrea]

@inkeep

Even without using OpenTelemetry the same scenario exists. The application at the "front" creates the trace, then other applications (or agents) all have their own llm invocation(s). There could be many LLM invocations within a single request.

Not only would the root need the input/output, but if you truly wanted to do cost calculation, you'd have to aggregate all the child llm spans, count tokens (model-dependent), and then aggregate that up to the root too.

The challenge is that the application that creates the root doesn't know anything about the data (input/output/tokens/etc) from the children, nor do the children have access to the root.

So how would one expect this whole scenario to work?

[inkeep-ai-assistant[bot]]

You've identified a fundamental architectural challenge with Langfuse's current model. The documentation confirms that trace input/output defaults to mirroring the root observation, and the available workarounds assume the service doing the LLM work has access to the trace context.(1)

How Langfuse Handles This Currently

Cost and token aggregation is automatic at the trace level. Langfuse aggregates usage details (tokens, cost) from all child generations within a trace—you don't need to manually propagate these up to the root. Each generation records its own usage_details, and Langfuse's dashboard and analytics aggregate these across the entire trace.(1)

For trace input/output specifically, the challenge you've described is real. The documentation shows you can use set_trace_io() from within a child span to override trace-level input/output:(1)

with langfuse.start_as_current_observation(as_type="span", name="child-operation") as span:
    # Child service can set trace-level input/output
    span.set_trace_io(
        input={"user_question": user_input},
        output={"answer": result}
    )

(1)

However, this requires the child service to explicitly call this method, and it assumes the child knows what the "meaningful" trace-level input/output should be.

Cross-Service Propagation

For distributed tracing, you can propagate attributes via HTTP headers using as_baggage=True:(1)

from langfuse import get_client, propagate_attributes

langfuse = get_client()
with langfuse.start_as_current_observation(as_type="span", name="api-request"):
    with propagate_attributes(
        user_id="user_123",
        session_id="session_abc",
        as_baggage=True,
    ):
        requests.get("https://service-b.example.com/api")

(1)

This propagates userId, sessionId, metadata, version, tags, and traceName across services.(1)

The Gap

The documentation doesn't provide a clean solution for the scenario where:

1. Multiple distributed agents each perform LLM operations
2. The root span creator doesn't know the aggregate input/output
3. No single child "owns" the trace-level input/output

This appears to be an architectural limitation. The cost/token aggregation works automatically, but trace-level input/output for distributed agentic systems remains a challenge without explicit coordination between services.

📚 Sources:

1. Instrumentation

---

Have another question? Just tag @inkeep.

[edeandrea]

@inkeep I'm using Java, not Python, and therefore am using the Langfuse REST API. It doesn't seem that the Langfuse REST API has that capability.

[inkeep-ai-assistant[bot]]

You're correct that the Python/JS SDK-specific methods like set_trace_io() aren't directly available when using the REST API or OpenTelemetry from Java.

OpenTelemetry Approach for Java

Since you're using the Java OpenTelemetry SDK (1.60.1), you can set trace-level input/output using Langfuse-specific span attributes. Langfuse maps these OTEL span attributes to trace input/output:(1)

• langfuse.trace.input / langfuse.trace.output - These can be set on any span in the trace and will be applied to the trace record in Langfuse

For observation-level input/output, Langfuse checks these attributes in priority order:(1)

• langfuse.observation.input, gen_ai.prompt, input.value, mlflow.spanInputs
• langfuse.observation.output, gen_ai.completion, output.value, mlflow.spanOutputs

So from your Java service that performs the LLM operation, you could set:

span.setAttribute("langfuse.trace.input", jsonInputString);
span.setAttribute("langfuse.trace.output", jsonOutputString);

(1)

This should populate the trace-level input/output in Langfuse, even though the span isn't the root span.

The Distributed Tracing Challenge

A related scenario was discussed where traces from distributed services need to share trace context. The approach involves checking incoming headers for an existing trace ID and creating spans with that trace context.(2)

However, the fundamental limitation remains: if multiple agents each set langfuse.trace.input/langfuse.trace.output, the behavior for which one "wins" isn't clearly documented. This architectural tension between distributed systems and trace-level attributes persists regardless of SDK or API choice.

📚 Sources:

1. Why are the input and output of my trace empty?
2. bug: distributed tracing replaces trace name and input

---

Have another question? Just tag @inkeep.

[edeandrea]

@inkeep What if the trace contains multiple llm calls?

[inkeep-ai-assistant[bot]]

When a trace contains multiple LLM calls, each LLM call should be recorded as a separate generation observation within the trace hierarchy.(1) The nesting and parent-child relationships are handled automatically by OpenTelemetry's context propagation.(1)

How Multiple LLM Calls Work

Each LLM operation creates its own generation span with its own input, output, and usage_details (tokens, cost). These are captured at the observation level, not the trace level.(1)

with langfuse.start_as_current_observation(as_type="span", name="outer-process") as outer_span:
    with langfuse.start_as_current_observation(as_type="generation", name="llm-step-1") as gen1:
        gen1.update(output="LLM 1 output")
    with langfuse.start_as_current_observation(as_type="generation", name="llm-step-2") as gen2:
        gen2.update(output="LLM 2 output")

(1)

The Trace Input/Output Problem with Multiple LLM Calls

The trace-level input and output fields default to mirroring the root observation only.(1) If you have multiple LLM calls, there's no automatic aggregation of their inputs/outputs into a single trace-level field.

If multiple spans set langfuse.trace.input or langfuse.trace.output, the behavior for which value appears at the trace level isn't explicitly documented in the available sources. This is the architectural gap you've identified—there's no built-in mechanism to aggregate inputs/outputs from multiple child LLM operations into a coherent trace-level summary.

The cost and token aggregation across all generations within a trace is handled by Langfuse's analytics layer, but the trace-level input/output fields remain tied to either the root observation or explicit overrides.

📚 Sources:

1. Instrumentation

---

Have another question? Just tag @inkeep.