How to configure Lambda to send traces to custom OTEL endpoint?

[jangaraj]

I use Python 3.8 Lambda with the config:

• layer: arn:aws:lambda:us-east-1:901920570463:layer:aws-otel-python38-ver-1-5-0:2

• collector.yaml:

receivers:
  otlp:
    protocols:
      grpc:
      http:

exporters:
  logging:
    loglevel: debug
  otlphttp:
    endpoint: https://<ALB-DNS-NAME>
        
service:
  pipelines:
    traces:
      receivers:
        - otlp
      exporters: 
        - logging
        - otlphttp
    metrics:
      receivers:
        - otlp
      exporters: 
        - logging
        - otlphttp

• env variables:

AWS_LAMBDA_EXEC_WRAPPER: /opt/otel-instrument
OPENTELEMETRY_COLLECTOR_CONFIG_FILE: /var/task/collector.yaml
OTEL_EXPORTER_OTLP_ENDPOINT: 'https://<ALB-DNS-NAME>'

Lambda logs:

...
2021-09-15T08:22:30.178Z	info	service/service.go:143	Starting extensions...
2021-09-15T08:22:30.178Z	info	service/service.go:188	Starting exporters...
2021-09-15T08:22:30.178Z	info	builder/exporters_builder.go:93	Exporter is starting...	{"kind": "exporter", "name": "logging"}
2021-09-15T08:22:30.178Z	info	builder/exporters_builder.go:98	Exporter started.	{"kind": "exporter", "name": "logging"}
2021-09-15T08:22:30.178Z	info	builder/exporters_builder.go:93	Exporter is starting...	{"kind": "exporter", "name": "otlphttp"}
2021-09-15T08:22:30.180Z	info	builder/exporters_builder.go:98	Exporter started.	{"kind": "exporter", "name": "otlphttp"}
2021-09-15T08:22:30.180Z	info	service/service.go:193	Starting processors...
2021-09-15T08:22:30.180Z	info	builder/pipelines_builder.go:52	Pipeline is starting...	{"pipeline_name": "traces", "pipeline_datatype": "traces"}
2021-09-15T08:22:30.180Z	info	builder/pipelines_builder.go:63	Pipeline is started.	{"pipeline_name": "traces", "pipeline_datatype": "traces"}
2021-09-15T08:22:30.180Z	info	builder/pipelines_builder.go:52	Pipeline is starting...	{"pipeline_name": "metrics", "pipeline_datatype": "metrics"}
2021-09-15T08:22:30.180Z	info	builder/pipelines_builder.go:63	Pipeline is started.	{"pipeline_name": "metrics", "pipeline_datatype": "metrics"}
2021-09-15T08:22:30.180Z	info	service/service.go:198	Starting receivers...
2021-09-15T08:22:30.180Z	info	builder/receivers_builder.go:71	Receiver is starting...	{"kind": "receiver", "name": "otlp"}
2021-09-15T08:22:30.180Z	info	otlpreceiver/otlp.go:75	Starting GRPC server on endpoint 0.0.0.0:4317	{"kind": "receiver", "name": "otlp"}
2021-09-15T08:22:30.180Z	info	otlpreceiver/otlp.go:137	Setting up a second GRPC listener on legacy endpoint 0.0.0.0:55680	{"kind": "receiver", "name": "otlp"}
2021-09-15T08:22:30.180Z	info	otlpreceiver/otlp.go:75	Starting GRPC server on endpoint 0.0.0.0:55680	{"kind": "receiver", "name": "otlp"}
2021-09-15T08:22:30.180Z	info	otlpreceiver/otlp.go:93	Starting HTTP server on endpoint 0.0.0.0:4318	{"kind": "receiver", "name": "otlp"}
2021-09-15T08:22:30.180Z	info	otlpreceiver/otlp.go:159	Setting up a second HTTP listener on legacy endpoint 0.0.0.0:55681	{"kind": "receiver", "name": "otlp"}
2021-09-15T08:22:30.180Z	info	otlpreceiver/otlp.go:93	Starting HTTP server on endpoint 0.0.0.0:55681	{"kind": "receiver", "name": "otlp"}
2021-09-15T08:22:30.180Z	info	builder/receivers_builder.go:76	Receiver started.	{"kind": "receiver", "name": "otlp"}
2021-09-15T08:22:30.180Z	info	service/collector.go:206	Setting up own telemetry...
2021-09-15T08:22:30.185Z	info	service/telemetry.go:99	Serving Prometheus metrics	{"address": ":8888", "level": 0, "service.instance.id": "65433d74-271d-419e-b5ee-7c3b111baaac"}
2021-09-15T08:22:30.185Z	info	service/collector.go:218	Everything is ready. Begin running and processing data.
2021/09/15 08:22:30 Registered extension ID: "9a51a80c-5bcd-4942-9c14-dcb7e9db5093"
2021/09/15 08:22:30 [collector] Register response: {
	"functionName": "demo-lambda",
	"functionVersion": "$LATEST",
	"handler": "index.handler"
}
2021/09/15 08:22:30 [collector] Waiting for event...
EXTENSION	Name: collector	State: Ready	Events: [INVOKE,SHUTDOWN]
2021/09/15 08:22:31 [collector] Received event: {
	"eventType": "INVOKE",
	"deadlineMs": 1631694161138,
	"requestId": "e7811555-3202-4f18-b5c6-e66ddbce6150",
	"invokedFunctionArn": "arn:aws:lambda:us-east-1:570626500822:function:demo-lambda",
	"tracing": {
		"type": "X-Amzn-Trace-Id",
		"value": "Root=1-6141ad45-5007b3231301d2fe12ec27d3;Parent=1734683a5b7cd4f7;Sampled=0"
	}
}
2021/09/15 08:22:31 [collector] Waiting for event...
END RequestId: e7811555-3202-4f18-b5c6-e66ddbce6150
REPORT RequestId: e7811555-3202-4f18-b5c6-e66ddbce6150	Duration: 33.84 ms	Billed Duration: 34 ms	Memory Size: 128 MB	Max Memory Used: 121 MB	Init Duration: 1167.83 ms

But my custom OTEL collector (https://<ALB-DNS-NAME>) is not receiving any traces. How to configure AWS OTEL python layer to send traces to the custom OTLP collector endpoint properly?

For the record: I have also AWS support ticket for this problem (#8887184821).

[jangaraj]

I got it working with:

receivers:
  otlp:
    protocols:
      grpc:
      http:

exporters:
  otlp:
    endpoint: ALB:4317
        
service:
  pipelines:
    traces:
      receivers:
        - otlp
      exporters:
        - otlp
    metrics:
      receivers:
        - otlp
      exporters:
        - otlp

I guess: only OTLP GRPC exporter is working.

Problem: any change of Enable active tracing removes evn variable AWS_LAMBDA_EXEC_WRAPPER, so it must be recreated again.

Exporter is working only when Active tracing is enabled - that mean traces are duplicated/pushed into AWS X-Ray and my AWS OTEL collector. How to configure layer, so traces will be pushed only to my AWS OTEL collector?

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[kolanos]

@jangaraj Thank you for posting a follow up solution. Much appreciated.

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.

[github-actions]

This issue was closed because it has been marked as stall for 30 days with no activity.