Extension crashing and terminating lambda

[git-rzanelli]

Expected Behavior

Datadog lambda extension should fail gracefully with an error message in the logs, but without causing the entire lambda to crash.

Actual Behavior

Logs show the following:
DD_EXTENSION \| ERROR \| Error while trying to read an API Key from Secrets Manager: Secrets Manager read error: ResourceNotFoundException: Secrets Manager can't find the specified secret. DD_EXTENSION \| ERROR \| No API key configured, exiting ... DD_EXTENSION | ERROR | Unable to load trace agent config: you must specify an API Key, either via a configuration file or the DD_API_KEY env var ... RequestId: 68533268-b573-4133-8059-f0a7eeb4387b Error: exit status 255 Extension.Crash
And the lambda terminates

Steps to Reproduce the Problem

1.Setup datadog-lambda-python as a lambda layer
2.Specify an invalid AWS Secrets manager ARN for DD_API_KEY_SECRET_ARN

Specifications

• Datadog Lambda Layer version: Datadog-Extension:22
• Python version: Datadog-Python39:58

Stacktrace

  | 2022-06-22T10:56:51.400+01:00 | 2022-06-22 09:56:51 UTC \| DD_EXTENSION \| INFO \| Features detected from environment:
-- | --
  | 2022-06-22T10:56:51.400+01:00 | 2022-06-22 09:56:51 UTC \| DD_EXTENSION \| ERROR \| Unable to load trace agent config: you must specify an API Key, either via a configuration file or the DD_API_KEY env var
  | 2022-06-22T10:56:51.401+01:00 | LOGS Name: datadog-agent State: Subscribed Types: [platform,function,extension]
  | 2022-06-22T10:56:51.404+01:00 | 2022-06-22 09:56:51 UTC \| DD_EXTENSION \| INFO \| Starting a serverless logs-agent...
  | 2022-06-22T10:56:51.406+01:00 | EXTENSION Name: datadog-agent State: Registered Events: [INVOKE,SHUTDOWN]
  | 2022-06-22T10:56:51.549+01:00 | START RequestId: 68533268-b573-4133-8059-f0a7eeb4387b Version: $LATEST
  | 2022-06-22T10:56:51.924+01:00 | 2022-06-22 09:56:51 UTC \| DD_EXTENSION \| DEBUG \| Datadog extension version : 22\|Datadog environment variables: DD_API_KEY_SECRET_ARN=***\|DD_LOG_LEVEL=debug\|
  | 2022-06-22T10:56:51.924+01:00 | 2022-06-22 09:56:51 UTC \| DD_EXTENSION \| DEBUG \| Starting daemon to receive messages from runtime...
  | 2022-06-22T10:56:51.925+01:00 | 2022-06-22 09:56:51 UTC \| DD_EXTENSION \| DEBUG \| Unable to restore the state from file
  | 2022-06-22T10:56:51.926+01:00 | 2022-06-22 09:56:51 UTC \| DD_EXTENSION \| DEBUG \| Found DD_API_KEY_SECRET_ARN value, trying to use it.
  | 2022-06-22T10:56:52.102+01:00 | 2022-06-22 09:56:52 UTC \| DD_EXTENSION \| ERROR \| Error while trying to read an API Key from Secrets Manager: Secrets Manager read error: ResourceNotFoundException: Secrets Manager can't find the specified secret.
  | 2022-06-22T10:56:52.102+01:00 | 2022-06-22 09:56:52 UTC \| DD_EXTENSION \| ERROR \| No API key configured, exiting
  | 2022-06-22T10:56:52.102+01:00 | 2022-06-22 09:56:52 UTC \| DD_EXTENSION \| DEBUG \| Using a SyncForwarder with a 5s timeout
  | 2022-06-22T10:56:52.102+01:00 | 2022-06-22 09:56:52 UTC \| DD_EXTENSION \| INFO \| No API key provided for domain "https://app.datadoghq.com", removing domain from endpoints
  | 2022-06-22T10:56:52.102+01:00 | 2022-06-22 09:56:52 UTC \| DD_EXTENSION \| INFO \| Retry queue storage on disk is disabled
  | 2022-06-22T10:56:52.102+01:00 | 2022-06-22 09:56:52 UTC \| DD_EXTENSION \| INFO \| Creating TimeSampler #0
  | 2022-06-22T10:56:52.102+01:00 | 2022-06-22 09:56:52 UTC \| DD_EXTENSION \| DEBUG \| 'telemetry.dogstatsd.aggregator_channel_latency_buckets' is empty, falling back to default values
  | 2022-06-22T10:56:52.103+01:00 | 2022-06-22 09:56:52 UTC \| DD_EXTENSION \| DEBUG \| 'telemetry.dogstatsd.listeners_latency_buckets' is empty, falling back to default values
  | 2022-06-22T10:56:52.103+01:00 | 2022-06-22 09:56:52 UTC \| DD_EXTENSION \| DEBUG \| 'telemetry.dogstatsd.listeners_channel_latency_buckets' is empty, falling back to default values
  | 2022-06-22T10:56:52.103+01:00 | 2022-06-22 09:56:52 UTC \| DD_EXTENSION \| DEBUG \| dogstatsd-udp: 127.0.0.1:8125 successfully initialized
  | 2022-06-22T10:56:52.103+01:00 | 2022-06-22 09:56:52 UTC \| DD_EXTENSION \| DEBUG \| DogStatsD will run 2 workers
  | 2022-06-22T10:56:52.103+01:00 | 2022-06-22 09:56:52 UTC \| DD_EXTENSION \| DEBUG \| Enabling logs collection HTTP route
  | 2022-06-22T10:56:52.103+01:00 | 2022-06-22 09:56:52 UTC \| DD_EXTENSION \| DEBUG \| Subscribing to Logs for types: [platform function extension]
  | 2022-06-22T10:56:52.103+01:00 | 2022-06-22 09:56:52 UTC \| DD_EXTENSION \| INFO \| dogstatsd-udp: starting to listen on 127.0.0.1:8125
  | 2022-06-22T10:56:52.103+01:00 | 2022-06-22 09:56:52 UTC \| DD_EXTENSION \| DEBUG \| Forwarder started
  | 2022-06-22T10:56:52.103+01:00 | 2022-06-22 09:56:52 UTC \| DD_EXTENSION \| DEBUG \| Demultiplexer started
  | 2022-06-22T10:56:52.104+01:00 | LOGS Name: datadog-agent State: Already subscribed Types: [platform,function,extension]
  | 2022-06-22T10:56:52.120+01:00 | 2022-06-22 09:56:52 UTC \| DD_EXTENSION \| INFO \| Starting a serverless logs-agent...
  | 2022-06-22T10:56:52.120+01:00 | 2022-06-22 09:56:52 UTC \| DD_EXTENSION \| INFO \| logs-agent started
  | 2022-06-22T10:56:52.120+01:00 | 2022-06-22 09:56:52 UTC \| DD_EXTENSION \| DEBUG \| Adding AWS Logs Log Source
  | 2022-06-22T10:56:52.123+01:00 | EXTENSION Name: datadog-agent State: Registered Events: [SHUTDOWN,INVOKE]
  | 2022-06-22T10:56:52.126+01:00 | END RequestId: 68533268-b573-4133-8059-f0a7eeb4387b
  | 2022-06-22T10:56:52.126+01:00 | REPORT RequestId: 68533268-b573-4133-8059-f0a7eeb4387b Duration: 576.57 ms Billed Duration: 577 ms Memory Size: 1024 MB Max Memory Used: 60 MB XRAY TraceId: 1-62b2e762-4bb25df156dc28c17f209566 SegmentId: 626d0c6f2e21b005 Sampled: true
  | 2022-06-22T10:56:52.126+01:00 | RequestId: 68533268-b573-4133-8059-f0a7eeb4387b Error: exit status 255 Extension.Crash
  | 2022-06-22T10:57:55.305+01:00 | START RequestId: 68533268-b573-4133-8059-f0a7eeb4387b Version: $LATEST
  | 2022-06-22T10:57:55.700+01:00 | 2022-06-22 09:57:55 UTC \| DD_EXTENSION \| DEBUG \| Datadog extension version : 22\|Datadog environment variables: DD_API_KEY_SECRET_ARN=***\|DD_LOG_LEVEL=debug\|
  | 2022-06-22T10:57:55.700+01:00 | 2022-06-22 09:57:55 UTC \| DD_EXTENSION \| DEBUG \| Starting daemon to receive messages from runtime...
  | 2022-06-22T10:57:55.700+01:00 | 2022-06-22 09:57:55 UTC \| DD_EXTENSION \| DEBUG \| Unable to restore the state from file
  | 2022-06-22T10:57:55.701+01:00 | 2022-06-22 09:57:55 UTC \| DD_EXTENSION \| DEBUG \| Found DD_API_KEY_SECRET_ARN value, trying to use it.
  | 2022-06-22T10:57:55.869+01:00 | 2022-06-22 09:57:55 UTC \| DD_EXTENSION \| ERROR \| Error while trying to read an API Key from Secrets Manager: Secrets Manager read error: ResourceNotFoundException: Secrets Manager can't find the specified secret.
  | 2022-06-22T10:57:55.869+01:00 | 2022-06-22 09:57:55 UTC \| DD_EXTENSION \| ERROR \| No API key configured, exiting
  | 2022-06-22T10:57:55.869+01:00 | 2022-06-22 09:57:55 UTC \| DD_EXTENSION \| DEBUG \| Using a SyncForwarder with a 5s timeout
  | 2022-06-22T10:57:55.869+01:00 | 2022-06-22 09:57:55 UTC \| DD_EXTENSION \| INFO \| No API key provided for domain "https://app.datadoghq.com", removing domain from endpoints
  | 2022-06-22T10:57:55.881+01:00 | 2022-06-22 09:57:55 UTC \| DD_EXTENSION \| INFO \| Retry queue storage on disk is disabled
  | 2022-06-22T10:57:55.881+01:00 | 2022-06-22 09:57:55 UTC \| DD_EXTENSION \| INFO \| Creating TimeSampler #0
  | 2022-06-22T10:57:55.881+01:00 | 2022-06-22 09:57:55 UTC \| DD_EXTENSION \| DEBUG \| 'telemetry.dogstatsd.aggregator_channel_latency_buckets' is empty, falling back to default values
  | 2022-06-22T10:57:55.881+01:00 | 2022-06-22 09:57:55 UTC \| DD_EXTENSION \| DEBUG \| 'telemetry.dogstatsd.listeners_latency_buckets' is empty, falling back to default values
  | 2022-06-22T10:57:55.881+01:00 | 2022-06-22 09:57:55 UTC \| DD_EXTENSION \| DEBUG \| 'telemetry.dogstatsd.listeners_channel_latency_buckets' is empty, falling back to default values
  | 2022-06-22T10:57:55.882+01:00 | 2022-06-22 09:57:55 UTC \| DD_EXTENSION \| DEBUG \| Forwarder started
  | 2022-06-22T10:57:55.882+01:00 | 2022-06-22 09:57:55 UTC \| DD_EXTENSION \| DEBUG \| Demultiplexer started
  | 2022-06-22T10:57:55.882+01:00 | 2022-06-22 09:57:55 UTC \| DD_EXTENSION \| DEBUG \| dogstatsd-udp: 127.0.0.1:8125 successfully initialized
  | 2022-06-22T10:57:55.883+01:00 | 2022-06-22 09:57:55 UTC \| DD_EXTENSION \| DEBUG \| DogStatsD will run 2 workers
  | 2022-06-22T10:57:55.883+01:00 | 2022-06-22 09:57:55 UTC \| DD_EXTENSION \| INFO \| dogstatsd-udp: starting to listen on 127.0.0.1:8125
  | 2022-06-22T10:57:55.883+01:00 | 2022-06-22 09:57:55 UTC \| DD_EXTENSION \| DEBUG \| Enabling logs collection HTTP route
  | 2022-06-22T10:57:55.883+01:00 | 2022-06-22 09:57:55 UTC \| DD_EXTENSION \| DEBUG \| Subscribing to Logs for types: [platform function extension]
  | 2022-06-22T10:57:55.885+01:00 | 2022-06-22 09:57:55 UTC \| DD_EXTENSION \| WARN \| Error loading config: open /var/task/datadog.yaml: no such file or directory
  | 2022-06-22T10:57:55.885+01:00 | 2022-06-22 09:57:55 UTC \| DD_EXTENSION \| INFO \| Features detected from environment:
  | 2022-06-22T10:57:55.885+01:00 | 2022-06-22 09:57:55 UTC \| DD_EXTENSION \| ERROR \| Unable to load trace agent config: you must specify an API Key, either via a configuration file or the DD_API_KEY env var
  | 2022-06-22T10:57:55.885+01:00 | LOGS Name: datadog-agent State: Already subscribed Types: [platform,function,extension]
  | 2022-06-22T10:57:55.899+01:00 | 2022-06-22 09:57:55 UTC \| DD_EXTENSION \| DEBUG \| Enhanced metrics: {durationMs:576.57 billedDurationMs:577 memorySizeMB:1024 maxMemoryUsedMB:60 initDurationMs:0}
  | 2022-06-22T10:57:55.899+01:00 | 2022-06-22 09:57:55 UTC \| DD_EXTENSION \| INFO \| Starting a serverless logs-agent...
  | 2022-06-22T10:57:55.899+01:00 | 2022-06-22 09:57:55 UTC \| DD_EXTENSION \| INFO \| logs-agent started
  | 2022-06-22T10:57:55.899+01:00 | 2022-06-22 09:57:55 UTC \| DD_EXTENSION \| DEBUG \| Adding AWS Logs Log Source
  | 2022-06-22T10:57:55.901+01:00 | EXTENSION Name: datadog-agent State: Registered Events: [INVOKE,SHUTDOWN]
  | 2022-06-22T10:57:55.903+01:00 | END RequestId: 68533268-b573-4133-8059-f0a7eeb4387b
  | 2022-06-22T10:57:55.903+01:00 | REPORT RequestId: 68533268-b573-4133-8059-f0a7eeb4387b Duration: 596.60 ms Billed Duration: 597 ms Memory Size: 1024 MB Max Memory Used: 63 MB XRAY TraceId: 1-62b2e762-4bb25df156dc28c17f209566 SegmentId: 593e15c50f91292a Sampled: true
  | 2022-06-22T10:57:55.903+01:00 | RequestId: 68533268-b573-4133-8059-f0a7eeb4387b Error: exit status 255 Extension.Crash
  | 2022-06-22T11:00:03.071+01:00 | START RequestId: 68533268-b573-4133-8059-f0a7eeb4387b Version: $LATEST
  | 2022-06-22T11:00:03.461+01:00 | 2022-06-22 10:00:03 UTC \| DD_EXTENSION \| DEBUG \| Datadog extension version : 22\|Datadog environment variables: DD_API_KEY_SECRET_ARN=***\|DD_LOG_LEVEL=debug\|
  | 2022-06-22T11:00:03.461+01:00 | 2022-06-22 10:00:03 UTC \| DD_EXTENSION \| DEBUG \| Starting daemon to receive messages from runtime...
  | 2022-06-22T11:00:03.461+01:00 | 2022-06-22 10:00:03 UTC \| DD_EXTENSION \| DEBUG \| Unable to restore the state from file
  | 2022-06-22T11:00:03.462+01:00 | 2022-06-22 10:00:03 UTC \| DD_EXTENSION \| DEBUG \| Found DD_API_KEY_SECRET_ARN value, trying to use it.
  | 2022-06-22T11:00:03.621+01:00 | 2022-06-22 10:00:03 UTC \| DD_EXTENSION \| ERROR \| Error while trying to read an API Key from Secrets Manager: Secrets Manager read error: ResourceNotFoundException: Secrets Manager can't find the specified secret.
  | 2022-06-22T11:00:03.621+01:00 | 2022-06-22 10:00:03 UTC \| DD_EXTENSION \| ERROR \| No API key configured, exiting
  | 2022-06-22T11:00:03.622+01:00 | 2022-06-22 10:00:03 UTC \| DD_EXTENSION \| DEBUG \| Using a SyncForwarder with a 5s timeout
  | 2022-06-22T11:00:03.622+01:00 | 2022-06-22 10:00:03 UTC \| DD_EXTENSION \| INFO \| No API key provided for domain "https://app.datadoghq.com", removing domain from endpoints
  | 2022-06-22T11:00:03.622+01:00 | 2022-06-22 10:00:03 UTC \| DD_EXTENSION \| INFO \| Retry queue storage on disk is disabled
  | 2022-06-22T11:00:03.622+01:00 | 2022-06-22 10:00:03 UTC \| DD_EXTENSION \| INFO \| Creating TimeSampler #0
  | 2022-06-22T11:00:03.622+01:00 | 2022-06-22 10:00:03 UTC \| DD_EXTENSION \| DEBUG \| 'telemetry.dogstatsd.aggregator_channel_latency_buckets' is empty, falling back to default values
  | 2022-06-22T11:00:03.622+01:00 | 2022-06-22 10:00:03 UTC \| DD_EXTENSION \| DEBUG \| Forwarder started
  | 2022-06-22T11:00:03.622+01:00 | 2022-06-22 10:00:03 UTC \| DD_EXTENSION \| DEBUG \| Demultiplexer started
  | 2022-06-22T11:00:03.623+01:00 | 2022-06-22 10:00:03 UTC \| DD_EXTENSION \| DEBUG \| 'telemetry.dogstatsd.listeners_latency_buckets' is empty, falling back to default values
  | 2022-06-22T11:00:03.623+01:00 | 2022-06-22 10:00:03 UTC \| DD_EXTENSION \| DEBUG \| 'telemetry.dogstatsd.listeners_channel_latency_buckets' is empty, falling back to default values
  | 2022-06-22T11:00:03.640+01:00 | 2022-06-22 10:00:03 UTC \| DD_EXTENSION \| DEBUG \| dogstatsd-udp: 127.0.0.1:8125 successfully initialized
  | 2022-06-22T11:00:03.640+01:00 | 2022-06-22 10:00:03 UTC \| DD_EXTENSION \| DEBUG \| DogStatsD will run 2 workers
  | 2022-06-22T11:00:03.640+01:00 | 2022-06-22 10:00:03 UTC \| DD_EXTENSION \| INFO \| dogstatsd-udp: starting to listen on 127.0.0.1:8125
  | 2022-06-22T11:00:03.641+01:00 | 2022-06-22 10:00:03 UTC \| DD_EXTENSION \| WARN \| Error loading config: open /var/task/datadog.yaml: no such file or directory
  | 2022-06-22T11:00:03.641+01:00 | 2022-06-22 10:00:03 UTC \| DD_EXTENSION \| INFO \| Features detected from environment:
  | 2022-06-22T11:00:03.641+01:00 | 2022-06-22 10:00:03 UTC \| DD_EXTENSION \| ERROR \| Unable to load trace agent config: you must specify an API Key, either via a configuration file or the DD_API_KEY env var
  | 2022-06-22T11:00:03.642+01:00 | 2022-06-22 10:00:03 UTC \| DD_EXTENSION \| DEBUG \| Enabling logs collection HTTP route
  | 2022-06-22T11:00:03.642+01:00 | 2022-06-22 10:00:03 UTC \| DD_EXTENSION \| DEBUG \| Subscribing to Logs for types: [platform function extension]
  | 2022-06-22T11:00:03.643+01:00 | LOGS Name: datadog-agent State: Already subscribed Types: [platform,function,extension]
  | 2022-06-22T11:00:03.660+01:00 | 2022-06-22 10:00:03 UTC \| DD_EXTENSION \| DEBUG \| Enhanced metrics: {durationMs:576.57 billedDurationMs:577 memorySizeMB:1024 maxMemoryUsedMB:60 initDurationMs:0}
  | 2022-06-22T11:00:03.660+01:00 | 2022-06-22 10:00:03 UTC \| DD_EXTENSION \| INFO \| Starting a serverless logs-agent...
  | 2022-06-22T11:00:03.660+01:00 | 2022-06-22 10:00:03 UTC \| DD_EXTENSION \| INFO \| logs-agent started
  | 2022-06-22T11:00:03.660+01:00 | 2022-06-22 10:00:03 UTC \| DD_EXTENSION \| DEBUG \| Adding AWS Logs Log Source
  | 2022-06-22T11:00:03.662+01:00 | EXTENSION Name: datadog-agent State: Registered Events: [INVOKE,SHUTDOWN]
  | 2022-06-22T11:00:03.664+01:00 | END RequestId: 68533268-b573-4133-8059-f0a7eeb4387b
  | 2022-06-22T11:00:03.664+01:00 | REPORT RequestId: 68533268-b573-4133-8059-f0a7eeb4387b Duration: 588.80 ms Billed Duration: 589 ms Memory Size: 1024 MB Max Memory Used: 62 MB XRAY TraceId: 1-62b2e762-4bb25df156dc28c17f209566 SegmentId: 1bdbdb51464d72f9 Sampled: true
  | 2022-06-22T11:00:03.664+01:00 | RequestId: 68533268-b573-4133-8059-f0a7eeb4387b Error: exit status 255 Extension.Crash

[astuyve]

Hi @git-rzanelli, thanks for reaching out!

In order to help you, I'll need more information, which includes filling out the rest of the issue template.

Could you share the version of this layer, the extension version, full stack trace, logs, and reproduction steps etc?

Thanks!

[astuyve]

Great, thanks @git-rzanelli! I'll report back when I've got more information to share.

[git-rzanelli]

Sorry, I pressed save before I had entered all of the information. I have updated the ticket.

This issue occurred when the Test instance of Datadog changed. As part of the change, the Datadog Secret Key in AWS changed, including the ARN. So the provided ARN was no longer valid.

[astuyve]

I see, and when the Datadog Secret Key ARN was updated and the function deployed, did this error resolve itself?

[git-rzanelli]

Once the Datadog Secret Key ARN was updated, everything worked as expected and there were no errors. However our team doesn't control the DD instance or the key ARN. So when the instance changed again and the old key was left in place, it failed gracefully with a 403 error in the logs and the lambda process completed as expected. BTW, this happened in a Test environment and we don't expect it to happen in Prod, but it does mean that the lambda's success is fully dependent on the DD layer success.

[tianchu]

Just giving an update, we are still working on this, already have a prototype, but it's non-trivial, so probably will take some time to actually be included in an official release.

[IvanTopolcic]

This should be fixed now, please update to the latest version of the lambda extension.