Lambda Improvements

[AlexanderWert]

post experimental improvements on AWS lambda support for node.js

[trentm]

This was an earlier dump of thoughts on improvements we could make. We can check them off when one of: we decide to not pursue the idea, the current state supports it well enough, there is a separate issue specifically covering the idea.

• Testing on real Lambda, deploy with aws-sdk or with terraform. Have this run daily or weekly, and able to run adhoc manually on PRs.
  • handler: callback, promise, async -> good
  • handler crashes: 1. throw at top-level in a non-async function, 2. uncaughtException (e.g. setImmediate(() => { throw new Error('boom') })), 3. unhandleRejection. -> there are limitations with some of these. Details at lambda wrapper doesn't handle uncaughtException or unhandleRejection #2379 (comment)
  • manually using apm.captureError() -> good, gets the current transaction id.
  • spans right at end captured? -> good
  • function timesout -> lambda instrumentation: handler timeout #2597
  • arch: amd64 and arm
  • doesn't blow up if extension is down (e.g. say ext not installed or ext is accidentally the wrong arch) -> lambda: if extension is down, agent eventually causes null function response #2598
• transaction.outcome is not set to 'failure' when there is an err to callback, or call to context.fail(). Mostly fixed in fix: lambda transaction data fixes #2423 However a call to context.fail() without passing an err argument will result in outcome and result being set to success. I've opened lambda: context.fail() without passing an error will result in trans.{result,outcome} being set to success #2581 for that.
• What should transaction.result be set to? Python is setting "HTTP Nxx"-type result. What about for function errors? This was clarified in Some Lambda spec clarifications apm#542
• Maybe disable gzip compression for serverUrl=localhost. See discussion at Disable payload compression for localhost apm#518 I've opened Disable payload compression for localhost #2582 for this.
• query param to signal "done now" to extension: [META 569] Add query param flushed=true to final intake request at the end of lambda function #2485
• unhandleRejection and uncaughtException trapping (lambda wrapper doesn't handle uncaughtException or unhandleRejection #2379)
• Decide if should automatically disable central config in the agent. Yes, per Update docs describing the simplified onboarding for the extension. apm-aws-lambda#122
• Grok performance impact of agent and extension: Test performance apm-aws-lambda#49
• DX: See if can automatically instrument _HANDLER. Then could avoid all code changes for nicer UX! Then consider having a layer that provides the agent and adds -r ... to NODE_OPTIONS. For frameworks that create a number of separate functions, having the same large agent in each function is a pain (large === slow). lambda: automatically wrap lambda handler #1336 and somewhat Lambda: Layer ARN for Installing Agent and Automatic Handler Wrapping #2543 as well.
• Review decision to have separate envvar for extension server URL. Here I was referring to the use of ELASTIC_APM_LAMBDA_APM_SERVER. I think it would be neat (by which I mean it might be less likely for users to get things wrong) if we just used the existing ELASTIC_APM_SERVER_URL envvar: the ext would use that, and in a lambda env the agents would ignore that and use localhost:8200. I.e. the agents would only work with the extension. I'll ask in the weekly meeting.
• If lambda handler doesn't pass a callback, then attach to the returned promise to detect the end. In this case avoid the wrapping of context.{succeed,fail,done} and consider dropping support for those completely. What we have is "fine", but I think relies on the internal AWS runtime impl calling context.{succeed,fail,done} in a way that gets our wrapped versions. Calling this "fine" for now, unless we demonstrate a bug with it.
• DX: Deployment support for SAM, CloudFormation, serverless, terraform, et al.
• DX: apiKey (revokable) vs secretToken. Already supported.
• grok using KMS or other secrets solutions for env: Support for Secrets Managers apm-aws-lambda#105
• What about Lambda metrics or re-enabling general metrics in Lambda env? There is Support for Metrics apm-aws-lambda#108 for this.
• Lambda tracestate support? lambda: trace-context trace continuation for SNS and SQS single event triggers #1831 for this though it isn't on the 8.2 milestone (yet).
• serverless offline bug: Serverless offline: context.done called twice within handler #1338

[trentm]

All the points above are resolved or moved to separate issues.