GRPC Receiver Timeout in Python Lambda running in a Container

[brett-bim]

I have a Python lambda deployed and running in a container, similar to this issue "publish Docker images". I have managed to compile and place the aws-otel-lambda extension in my container correctly as far as I can tell. However, when I execute the function, the lambda times out waiting for the gRPC receiver to start. The timeout is set to 30 seconds.

2022-01-28T20:31:58.661Z	info	builder/receivers_builder.go:68	Receiver is starting...	{"kind": "receiver", "name": "otlp"}
2022-01-28T20:31:58.661Z	info	otlpreceiver/otlp.go:68	Starting GRPC server on endpoint 0.0.0.0:4317	{"kind": "receiver", "name": "otlp"}
EXTENSION	Name: collector	State: Ready	Events: [INVOKE,SHUTDOWN]
2022/01/28 20:32:28 [collector] Received event: {
	"eventType": "SHUTDOWN",
	"deadlineMs": 1643401950589,
	"requestId": "",

etc

My collector config:

# collector.yaml in the root directory
# Set an environemnt variable 'OPENTELEMETRY_COLLECTOR_CONFIG_FILE' to
# '/var/task/collector.yaml'

receivers:
  otlp:
    protocols:
      grpc:
      http:

exporters:
  logging:
    loglevel: debug
  # awsxray:

#enables output for traces
service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [logging]

I have a hunch I need to do something else with my container to enable this but not sure what. Any thoughts would be appreciated.

[brett-bim]

Is it possible that a role needs to be assigned to the lambda to allow the GRPC receiver to start up/connect? I haven't seen any thing confirming that but want to rule it out.

NOTE: I'm not trying to call any AWS services yet, just get the layer working inside a container to do basic logging.

[brett-bim]

Circling back on this, I've discovered that if the AWS_LAMBDA_EXEC_WRAPPER environment variable is set, the exhibited behavior is the timeout starting up the receiver. I deleted this env variable on a whim, leaving the OPENTELEMETRY_COLLECTOR_CONFIG_FILE env var intact and I now have logs being written.

However, I still do not have any events going to external exporters (Honeycomb specifically for some reason. I'm going to try and set up xray and see if the behavior is the same.

[brett-bim]

I've confirmed that the extension is running now in the container but logs are never sent to external exporters. Logging and x-ray work fine but my Honeycomb exporter does not send events. I'm sort of at a loss for how to debug this and welcome any insight.

#collector.yaml in the root directory
#Set an environemnt variable 'OPENTELEMETRY_COLLECTOR_CONFIG_FILE' to '/var/task/collector.yaml'

receivers:
  otlp:
    protocols:
      grpc:
      http:
exporters:
  logging:
    loglevel: debug
  awsxray:
  otlp:
    endpoint: api.honeycomb.io:443
    headers:
      "x-honeycomb-team": "someapikey"
      "x-honeycomb-dataset": "some-dataset"
#enables output for traces to xray
service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [otlp, logging, awsxray]

Dockerfile

FROM alpine:latest as layer-copy 

#ARG AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION:-"us-east-1"}
#ARG AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID:-""}
#ARG AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY:-""}
ENV AWS_DEFAULT_REGION=us-east-1
ENV AWS_ACCESS_KEY_ID=mykeyid
ENV AWS_SECRET_ACCESS_KEY=secret_access_key

RUN apk add aws-cli curl unzip

RUN mkdir -p /opt

RUN curl $(aws lambda get-layer-version-by-arn --arn arn:aws:lambda:\us-east-1:901920570463:layer:aws-otel-python38-ver-1-7-1:1 --query 'Content.Location' --output text) --output layer.zip
RUN unzip layer.zip -d /opt
RUN rm layer.zip

FROM public.ecr.aws/lambda/python:3.8 as application
RUN mkdir -p ${LAMBDA_TASK_ROOT}
WORKDIR ${LAMBDA_TASK_ROOT}

COPY hello_world/requirements.txt ${LAMBDA_TASK_ROOT}/requirements.txt
RUN pip3 install -r ${LAMBDA_TASK_ROOT}/requirements.txt

COPY hello_world/ ${LAMBDA_TASK_ROOT}/hello_world
COPY collector.yaml ${LAMBDA_TASK_ROOT}

WORKDIR /opt
COPY --from=layer-copy /opt .
EXPOSE 55680
EXPOSE 4317
EXPOSE 9464
EXPOSE 55681

CMD [ "hello_world.app.lambda_handler" ]

Lambda is built and deployed via SAM.

[osiris43]

Turns out, I had a mistake in the value for AWS_LAMBDA_EXEC_WRAPPER. This was pointing to a non-existent location/script. Once I pointed this at /opt/otel-instrument, traces started flowing. Sorry for the trouble. You can close this issue.

[anuraaga]

Thanks for confirming @osiris43