Disable logging to DataDog for local invocation

[dtothefp]

Expected Behavior

I'd like to be able to locally invoke my function using the Serverless framework and just see DEBUG logs without sending to data dog. Maybe this is possible with existing env variables for configuration but I didn't see these options.

I'm using this library in combination with serverless-plugin-datadog to supply the Forwarder ARN.

Actual Behavior

serverless invoke local -f <my_function> -p ./event.json

See error below

Steps to Reproduce the Problem

1. run a local invocation

Specifications

• Datadog Lambda Layer version: Not sure..where do I find this?
• Node version: 12

Stacktrace

{"status":"error","message":"datadog:api key not configured, see https://dtdg.co/sls-node-metrics"}
{"level":50,"time":1597887273985,"pid":17773,"hostname":"David-Fox-Powell-Pro","msg":"datadog:couldn't read xray trace header from env"}
{"level":50,"time":1597887273996,"pid":17773,"hostname":"David-Fox-Powell-Pro","innerError":{"name":"TypeError","message":"Cannot read property 'split' of undefined","stack":"TypeError: Cannot read property 'split' of undefined\n    at parseLambdaARN (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/datadog-lambda-js/src/utils/arn.ts:22:24)\n    at Object.parseTagsFromARN (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/datadog-lambda-js/src/utils/arn.ts:54:35)\n    at getEnhancedMetricTags (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/datadog-lambda-js/src/metrics/enhanced-metrics.ts:49:15)\n    at incrementEnhancedMetric (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/datadog-lambda-js/src/metrics/enhanced-metrics.ts:74:5)\n    at Object.incrementInvocationsMetric (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/datadog-lambda-js/src/metrics/enhanced-metrics.ts:79:3)\n    at wrappedFunc.<computed> (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/datadog-lambda-js/src/index.ts:109:9)\n    at Object.<anonymous> (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/datadog-lambda-js/src/utils/handler.ts:21:13)\n    at step (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/datadog-lambda-js/dist/utils/handler.js:33:23)\n    at Object.next (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/datadog-lambda-js/dist/utils/handler.js:14:53)\n    at /Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/datadog-lambda-js/dist/utils/handler.js:8:71\n    at new Promise (<anonymous>)\n    at __awaiter (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/datadog-lambda-js/dist/utils/handler.js:4:12)\n    at /Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/datadog-lambda-js/src/utils/handler.ts:19:48\n    at exports.handler (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/functions/edge/entry.js:19:26)\n    at /Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/serverless/lib/plugins/aws/invokeLocal/index.js:841:30\n    at Promise._execute (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/bluebird/js/release/debuggability.js:384:9)\n    at Promise._resolveFromExecutor (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/bluebird/js/release/promise.js:518:18)\n    at new Promise (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/bluebird/js/release/promise.js:103:10)\n    at AwsInvokeLocal.invokeLocalNodeJs (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/serverless/lib/plugins/aws/invokeLocal/index.js:794:12)\n    at AwsInvokeLocal.invokeLocal (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/serverless/lib/plugins/aws/invokeLocal/index.js:202:19)\n    at AwsInvokeLocal.tryCatcher (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/bluebird/js/release/util.js:16:23)\n    at Promise._settlePromiseFromHandler (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/bluebird/js/release/promise.js:547:31)\n    at Promise._settlePromise (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/bluebird/js/release/promise.js:604:18)\n    at Promise._settlePromiseCtx (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/bluebird/js/release/promise.js:641:10)\n    at _drainQueueStep (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/bluebird/js/release/async.js:97:12)\n    at _drainQueue (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/bluebird/js/release/async.js:86:9)\n    at Async._drainQueues (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/bluebird/js/release/async.js:102:5)\n    at Immediate.Async.drainQueues [as _onImmediate] (/Users/davidfox-powell/dev/good-rx/cross-origin-ab-test/node_modules/bluebird/js/release/async.js:15:14)\n    at processImmediate (internal/timers.js:456:21)\n    at process.topLevelDomainCallback (domain.js:137:15)"},"msg":"datadog:Pre-lambda hook threw error"}
{"level":50,"time":1597887275874,"pid":17773,"hostname":"David-Fox-Powell-Pro","innerError":{},"msg":"datadog:failed to flush metrics"

[DarcyRaynerDD]

Hey @dtothefp . It looks like the error is because of some environment variables we usually read from the AWS sandbox that the serverless plugin's invoke environment doesn't provide. I think we might be able to address this at the serverless plugin level by disabling instrumentation when the invoke occurs. DataDog/serverless-plugin-datadog#63 We might be able to address both issues at once.

[dtothefp]

@DarcyRaynerDD your fix for serverless plugin fixed the problem references in that issue but I'm still seeing this on local invoke

{"status":"error","message":"datadog:api key not configured, see https://dtdg.co/sls-node-metrics"}

Do I need to update this lib as well?

[sfirrin]

Hi @dtothefp, that error message comes from this function, and is status: error if the logForwarding config flag is false, and status: debug if the flag is true. The reason for this is that the Datadog API key is not necessary if metrics are written to logs rather than sent directly to Datadog.

You should be able to enable writing the metrics to logs by setting the flushMetricsToLogs serverless plugin config variable.

[dtothefp]

@sfirrin sorry for the late reply, for some reason I'm not receiving notifications for this issue 🤷‍♂️

This solution seems difficult to do because in my mind I would have to toggle the custom flushMetricsToLogs based upon the condition of if the Lambda invocation is local invoke or not which I don't think is possible in serverless.yml. So it would be something like

custom:
  datadog:
    flushMetricsToLogs: # ternary for IS_LOCAL