Span's add_event and record_exception not working with X-Ray

[Kharvi-Code]

Span's add_event() and record_exception() are not working when exporter is X-Ray. For other exporters like Jaeger I could see Logs as a separate field in UI for add_event() and record_exception(). Looks like set_attribute() works fine as I can see the attributes in Metadata section of the span X-Ray.

Steps to reproduce:

Run the aws-otel-collector in docker as below
docker run --rm -p 4317:4317 -p 55680:55680 -p 8889:8888 -e "AWS_ACCESS_KEY_ID=" -e "AWS_SECRET_ACCESS_KEY=" -e AWS_REGION= -v :/otel-local-config.yaml --name awscollector public.ecr.aws/aws-observability/aws-otel-collector:latest --config otel-local-config.yaml
Create a simple python script with 2 spans as below and run:
"""OTLP exporter"""
from opentelemetry import trace
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import (
OTLPSpanExporter,
)
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.sdk.extension.aws.trace import AwsXRayIdGenerator

span_exporter = OTLPSpanExporter(
endpoint="localhost:4317",
insecure=True,
)
span_processor = BatchSpanProcessor(span_exporter)
tracer_provider = TracerProvider(id_generator=AwsXRayIdGenerator())
tracer_provider.add_span_processor(span_processor)
trace.set_tracer_provider(tracer_provider)

tracer = trace.get_tracer(name)

try:
s1 = tracer.start_span("foo")
s1.add_event("Tracing.........Foo...........", {})
s1.set_attribute("example", "value")
time.sleep(2)
s2 = tracer.start_span("bar", context=trace.set_span_in_context(s1))
try:
1 / 0
except Exception as e:
s2.record_exception(e)
finally:
s2.end()
finally:
s1.end()

Note: I have to use tracer.start_span instead of tracer.start_as_current_span as per the requirement

Expected:
s1.add_event("Tracing.........Foo...........", {}) - has to send logs/event to X-Ray (foo Segment)
s2.record_exception(e) - has to send Exception trace to X-Ray (bar Subsegment)

Actual:
Dont know where the events/logs will be for foo Segment in X-Ray.
Did not get Exception trace in Exception tab of bar Subsegment

[anuraaga]

Hi @Kharvi-Code - unfortunately X-Ray does not have a concept of events like Jaeger. It would be a cool feature but there isn't any plan for it yet - events get dropped during conversion to X-Ray.

Exceptions should be supported though so it's surprising to not have anything show. Are you able to share the JSON for the trace which can be downloaded from X-Ray? Also if you haven't, it would be good to run docker pull public.ecr.aws/aws-observability/aws-otel-collector:latest once just to make sure you are using the latest version of the collector.

[Kharvi-Code]

@anuraaga I have pulled the latest aws-collector and tried again. Still no exception recorded.
Json data:

{
    "Id": "1-615d85af-121d49032ccc229d6d1029f6",
    "Duration": 2.006,
    "LimitExceeded": false,
    "Segments": [
        {
            "Id": "91d067549cb2789c",
            "Document": {
                "id": "91d067549cb2789c",
                "name": "foo",
                "start_time": 1633519023.622719,
                "trace_id": "1-615d85af-121d49032ccc229d6d1029f6",
                "end_time": 1633519025.6289365,
                "fault": false,
                "error": false,
                "throttle": false,
                "aws": {
                    "xray": {
                        "auto_instrumentation": false,
                        "sdk_version": "1.5.0",
                        "sdk": "opentelemetry for python"
                    }
                },
                "metadata": {
                    "default": {
                        "otel.resource.telemetry.sdk.name": "opentelemetry",
                        "otel.resource.service.name": "unknown_service",
                        "otel.resource.telemetry.sdk.language": "python",
                        "example": "value",
                        "otel.resource.telemetry.sdk.version": "1.5.0"
                    }
                },
                "subsegments": [
                    {
                        "id": "05696b5e2028be9f",
                        "name": "bar",
                        "start_time": 1633519025.6289365,
                        "end_time": 1633519025.6289365,
                        "fault": false,
                        "error": false,
                        "throttle": false,
                        "aws": {
                            "xray": {
                                "auto_instrumentation": false,
                                "sdk_version": "1.5.0",
                                "sdk": "opentelemetry for python"
                            }
                        }
                    }
                ]
            }
        }
    ]
}

[anuraaga]

Thanks @Kharvi-Code - indeed the exception is not being exported to X-Ray. I realize it's because you don't call set_status to mark the span as a failure - exceptions are only exported for failed spans. Was it your intention to not set the status?

@willarmiros Do you know if the backend would generally expect exceptions to be exported even for successful spans?

[Kharvi-Code]

Thanks @anuraaga - Since I was using Jaeger before this where I was not setting status for failed spans, I was not aware that I have to explicitly set status for failed spans.
It started recording exception once I gave set_status(). I could see the exception in XRay now.
Thanks for the help.

[willarmiros]

@anuraaga I'm not sure if it's a requirement of the X-Ray backend to have a non-OK span for an exception to be recorded, but I do know that with the X-Ray SDKs calling add_exception would always also call set_fault, so the state of recording an exception with a successful status wasn't really documented. We could do an experiment to see if it's allowed by the backend and change the awsxrayexporter if so.

Glad to see it worked as expected for now though.

[Kharvi-Code]

Closing issue as there is no further discussion on this. Thanks for the help.