Collector on ECS Fargate not showing the correct environment in X-Ray

[CarmenAPuccio]

I'm trying to spin up a custom app in ECS Fargate with the latest version of the aws-otel-collector (v11) and I'm seeing interesting results in X-Ray depending on the X-Ray SDK used (Go vs .NET). Obviously this leads me to believe it's an SDK specific issue but I'm opening the issue here first to see if there are any suggestions. I'm expecting to see information about the environment in X-Ray such as AWS::ECS::Container in the service map and corresponding traces but I'm getting mixed results. When i deploy the CloudFormation template as outlined in the aws-otel docs I'm noticing that there is no information about the environment as seen below.

{
	"Id": "1-60ff091d-6a8796d4709a6a1472b9deb6",
	"Duration": 0,
	"LimitExceeded": false,
	"Segments": [{
		"Id": "b57a66add9f1197f",
		"Document": {
			"id": "b57a66add9f1197f",
			"name": "SampleServer",
			"start_time": 1627326749.672193,
			"trace_id": "1-60ff091d-6a8796d4709a6a1472b9deb6",
			"end_time": 1627326749.672249,
			"fault": false,
			"error": false,
			"throttle": false,
			"http": {
				"request": {
					"url": "http://localhost:8000/",
					"method": "GET",
					"user_agent": "Go-http-client/1.1",
					"client_ip": "127.0.0.1",
					"x_forwarded_for": true
				},
				"response": {
					"status": 200,
					"content_length": 0
				}
			},
			"aws": {
				"xray": {
					"auto_instrumentation": false,
					"sdk_version": "1.1.0",
					"sdk": "X-Ray for Go for Go"
				}
			},
			"metadata": {
				"default": {
					"otel.resource.telemetry.sdk.name": "X-Ray for Go",
					"otel.resource.telemetry.sdk.language": "Go",
					"otel.resource.cloud.provider": "aws",
					"otel.resource.telemetry.sdk.version": "1.1.0"
				}
			}
		}
	}]
}

My own sample application is written in .NET 5.0.302 with the latest packages (listed below) but interestingly enough when I deploy to ECS Fargate instead of no origin/environment it's listed as "origin": "AWS::ElasticBeanstalk::Environment" as shown below. The .NET SDK obviously exposes more data but it's obviously still incorrect.

In both cases, I'm running the aws-otel-collector with the config that comes in the image --config=/etc/ecs/container-insights/otel-task-metrics-config.yaml

{
    "Id": "1-60ff0ef9-69b7f7c2497cf80f8ea659c6",
    "Duration": 0.009,
    "LimitExceeded": false,
    "Segments": [
        {
            "Id": "5c06663e843c564c",
            "Document": {
                "id": "5c06663e843c564c",
                "name": "GeoLocationAPI",
                "start_time": 1627328249.489951,
                "trace_id": "1-60ff0ef9-69b7f7c2497cf80f8ea659c6",
                "end_time": 1627328249.4993875,
                "fault": false,
                "error": false,
                "throttle": false,
                "http": {
                    "request": {
                        "url": "http://cdkge-lb8a1-1b4rhmo4ueypd-651829293.us-east-1.elb.amazonaws.com/api/v1/geolocation",
                        "method": "GET",
                        "user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36"
                    },
                    "response": {
                        "status": 200,
                        "content_length": 0
                    }
                },
                "aws": {
                    "elastic_beanstalk": {
                        "version_label": null,
                        "deployment_id": 0,
                        "environment_name": null
                    },
                    "xray": {
                        "auto_instrumentation": false,
                        "sdk_version": "1.1.0.174",
                        "sdk": "opentelemetry for dotnet"
                    }
                },
                "metadata": {
                    "default": {
                        "otel.resource.telemetry.sdk.name": "opentelemetry",
                        "http.route": "api/v{version:apiVersion}/GeoLocation",
                        "http.path": "/api/v1/geolocation",
                        "otel.resource.service.instance.id": "41c9ff05-035c-46a4-938d-8b1d4fbd728c",
                        "otel.resource.service.name": "GeoLocationAPI",
                        "otel.resource.telemetry.sdk.language": "dotnet",
                        "otel.resource.telemetry.sdk.version": "1.1.0.174"
                    }
                },
                "origin": "AWS::ElasticBeanstalk::Environment"
            }
        }
    ]
}

Package References:

    <PackageReference Include="OpenTelemetry" Version="1.1.0" />
    <PackageReference Include="OpenTelemetry.Contrib.Extensions.AWSXRay" Version="1.0.1" />
    <PackageReference Include="OpenTelemetry.Contrib.Instrumentation.AWS" Version="1.0.1" />
    <PackageReference Include="OpenTelemetry.Exporter.Console" Version="1.1.0" />
    <PackageReference Include="OpenTelemetry.Exporter.OpenTelemetryProtocol" Version="1.1.0" />
    <PackageReference Include="OpenTelemetry.Instrumentation.AspNetCore" Version="1.0.0-rc7" />
    <PackageReference Include="OpenTelemetry.Instrumentation.Http" Version="1.0.0-rc7" />

[CarmenAPuccio]

Also opened an issue on open-telemetry/opentelemetry-dotnet-contrib#146 for further investigation in the .NET portion.

[lupengamzn]

Hey @CarmenAPuccio ,

Thanks for bring this up

OTel .NET SDK by default sets an instance id in the resource attributes when you apply AddService API, which causes the collector to add ElasticBeanstalk as origin.

When you're applying extension API AddService to the ResourceBuilder, make sure to set autoGenerateServiceInstanceId as false to mitigate this issue if you're not running on ElasticBeanstalk.

Also, OTel .NET SDK currently doesn't have public resource detector API provided yet, you may switch to use the ones on collector side.

[CarmenAPuccio]

@lupengamzn - Thanks for the heads up. Setting up a resource detection processor on the collector worked perfect for ECS/Fargate. Unrelated to the collector but still digging - I'm still noticing that the client-ip isn't making it into the x-ray traces with the latest .NET packages though. I see you are working on this on open-telemetry/opentelemetry-dotnet#1546 it would be awesome if that could be merged.

{
	"Id": "1-610d91e3-20b7a3ae569efc3500817ef2",
	"Duration": 0.011,
	"LimitExceeded": false,
	"Segments": [{
		"Id": "d6252887edd12045",
		"Document": {
			"id": "d6252887edd12045",
			"name": "GeoLocationAPI",
			"start_time": 1628279267.963049,
			"trace_id": "1-610d91e3-20b7a3ae569efc3500817ef2",
			"end_time": 1628279267.9744542,
			"fault": false,
			"error": false,
			"throttle": false,
			"http": {
				"request": {
					"url": "http://xxxxxxxxxxx.us-east-1.elb.amazonaws.com/api/v1/geolocation",
					"method": "GET",
					"user_agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.107 Safari/537.36"
				},
				"response": {
					"status": 200,
					"content_length": 0
				}
			},
			"aws": {
				"account_id": "123456789",
				"xray": {
					"auto_instrumentation": false,
					"sdk_version": "1.1.0.174",
					"sdk": "opentelemetry for dotnet"
				}
			},
			"metadata": {
				"default": {
					"otel.resource.aws.ecs.task.arn": "arn:aws:ecs:us-east-1:123456789:task/CdkGeoLocationApiStack-GeoLocationAPICluster885F1E42-AIZm1Zfdtn19/6d5163d78641444386a898590ea1e555",
					"otel.resource.telemetry.sdk.name": "opentelemetry",
					"otel.resource.cloud.availability_zone": "us-east-1b",
					"http.path": "/api/v1/geolocation",
					"otel.resource.aws.ecs.cluster.arn": "arn:aws:ecs:us-east-1:123456789:cluster/CdkGeoLocationApiStack-GeoLocationAPICluster885F1E42-AIZm1Zfdtn19",
					"otel.resource.aws.ecs.task.family": "GeoLocationAPI",
					"otel.resource.aws.ecs.launchtype": "fargate",
					"otel.resource.service.namespace": "GeoLocation",
					"otel.resource.aws.ecs.task.revision": "49",
					"otel.resource.cloud.account.id": "123456789",
					"otel.resource.service.name": "GeoLocationAPI",
					"otel.resource.telemetry.sdk.version": "1.1.0.174",
					"http.route": "api/v{version:apiVersion}/GeoLocation",
					"otel.resource.cloud.platform": "aws_ecs",
					"otel.resource.cloud.region": "us-east-1",
					"otel.resource.telemetry.sdk.language": "dotnet",
					"otel.resource.cloud.provider": "aws"
				}
			}
		}
	}]
}

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[github-actions]

This issue was closed because it has been marked as stall for 30 days with no activity.