Providing Parent in X-Amzn-Trace-Id results in no spans being exported

[andrew-matteson]

There's a good chance this is user error. If so, I'd appreciate a pointer to the relevant doc.

Describe your environment

• Python 3.9
• fastapi==0.65.3
• opentelemetry-api==1.4.1
• opentelemetry-exporter-otlp==1.4.1
• opentelemetry-exporter-otlp-proto-grpc==1.4.1
• opentelemetry-instrumentation==0.23b2
• opentelemetry-instrumentation-asgi==0.23b2
• opentelemetry-instrumentation-fastapi==0.23b2
• opentelemetry-proto==1.4.1
• opentelemetry-sdk==1.4.1
• opentelemetry-sdk-extension-aws==0.23b2
• opentelemetry-semantic-conventions==0.23b2
• opentelemetry-util-http==0.23b2

Steps to reproduce
Using this sample application:

import fastapi
import uvicorn
from opentelemetry import propagate, trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.sdk.extension.aws.trace import AwsXRayIdGenerator
from opentelemetry.sdk.extension.aws.trace.propagation.aws_xray_format import AwsXRayFormat
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor

app = fastapi.FastAPI()

@app.get("/foo")
async def foo():
    return {"message": "foo"}


# Setup AWS X-Ray Propagator
propagate.set_global_textmap(AwsXRayFormat())

# Setup Tracer
otlp_exporter = OTLPSpanExporter()
span_processor = BatchSpanProcessor(otlp_exporter)

tracer_provider = TracerProvider(id_generator=AwsXRayIdGenerator())
tracer_provider.add_span_processor(span_processor)

trace.set_tracer_provider(tracer_provider)

FastAPIInstrumentor.instrument_app(app)

uvicorn.run(app)

Calling: curl 'http://localhost:8000/foo' produces a span that is exported by my collector to X-Ray.

Calling: curl 'http://localhost:8000/foo' -H 'X-Amzn-Trace-Id: Root=1-612fa749-271fa48e4c544863a13425d5;Parent=86153bfee2237b3b' does not export a span.

What is the expected behavior?
My frontend application is producing requests with X-Amzn-Trace-Id in the format above. The frontend is separately sending X-Ray data. I'm expecting the trace ID generated by the frontend to be the root of the of the server segment so I can correlate the frontend activity and the backend activity.

What is the actual behavior?
When providing the header from the frontend, no span is generated by the backend. The frontend succeeds in generating spans that I see in the X-Ray console. No backend segments are seen.

Additional context
Unsure if #445 is related. It's the only thing that I could find in the issues that might be.

[owais]

FYI @NathanielRN

[andrew-matteson]

I continued playing with this last night and was able to get this to work by adding ;Sampled=1 to the header. My team is exploring moving from using the aws xray sdk to opentelemetry. The behavior of our stack in the past was to not provide Sampled from our frontend (we get a small number of requests, so just sample all). The defaults from our instrumentation libraries and the xray sdk seem to use that as a signal to sample.

I'd refine my bug report, as

• Not providing Sampled should result in the span being recorded (equivalent of Sampled=1)
• The behavior in the absence of Sampled should be consistent for Parent supplied or not.

[NathanielRN]

Thanks for providing this additional context @andrew-matteson ! It's very helpful 😄

Not providing Sampled should result in the span being recorded (equivalent of Sampled=1)

I looks like you are correct, if Sampled= is not provided, the trace is assumed to not be sampled by the AwsXrayFormat propagator when it extracts the context from the header:

opentelemetry-python-contrib/sdk-extension/opentelemetry-sdk-extension-aws/src/opentelemetry/sdk/extension/aws/trace/propagation/aws_xray_format.py

Lines 154 to 158 in 6f96577

	@staticmethod
	def _extract_span_properties(trace_header):
	trace_id = trace.INVALID_TRACE_ID
	span_id = trace.INVALID_SPAN_ID
	sampled = False

This behavior is consistent with what is done in other languages such as in the OTel Java AWS X-Ray Propagator or in OTel JS AWS X-Ray Propagator.

Your use case makes sense, however I'm not sure if sampling by default is what most users would want. Maybe we can make documentation about this better in the meantime.

The behavior in the absence of Sampled should be consistent for Parent supplied or not.

I think based on your configuration, the behavior you saw was expected. In your code you are using TracerProvider without a custom Sampler, that means you get the default Sampler:

class TracerProvider(trace_api.TracerProvider):
    """See `opentelemetry.trace.TracerProvider`."""

    def __init__(
        self,
        sampler: sampling.Sampler = _TRACE_SAMPLER,

The default Sampler is the parentbased_alwayson sampler:

def _get_from_env_or_default() -> Sampler:
    trace_sampler = os.getenv(
        OTEL_TRACES_SAMPLER, "parentbased_always_on"
    ).lower()

which will always sample unless the Parent is present to say otherwise:

DEFAULT_ON = ParentBased(ALWAYS_ON)
"""Sampler that respects its parent span's sampling decision, but otherwise always samples."""

So in your 2nd case, when you add the Parent header, your TracerProvider chooses to respect the Parent. Maybe if you want the trace to be Sampled without setting Sampled=1 explicitly we could make the AwsXrayFormat propagator read from the currently configured TraceProvider's Sampler to make the default sampling decision (instead of assuming sampled = false.

But to unblock you for now I would suggest using Sampled=1 🙂

Please let me know if that makes sense!

[andrew-matteson]

@NathanielRN Thank you! Makes complete sense. I'm unblocked at this point.

Your doc suggestion sounds like the way to go.