From fc29d7e5880a42dc1f0005cde8846d61fda7adf4 Mon Sep 17 00:00:00 2001 From: Colton Myers Date: Mon, 8 Aug 2022 12:34:38 -0600 Subject: [PATCH 1/7] Add ELB section for lambda instrumentation --- .../tracing-instrumentation-aws-lambda.md | 76 +++++++++++++++++-- 1 file changed, 68 insertions(+), 8 deletions(-) diff --git a/specs/agents/tracing-instrumentation-aws-lambda.md b/specs/agents/tracing-instrumentation-aws-lambda.md index 6aab18aa..9165b0b7 100644 --- a/specs/agents/tracing-instrumentation-aws-lambda.md +++ b/specs/agents/tracing-instrumentation-aws-lambda.md @@ -103,9 +103,9 @@ Field | Value | Description | Source `context.service.origin.name` | e.g. `gdnrpwmtsb...amazonaws.com` | The full domain name of the API Gateway. | `event.requestContext.domainName` `context.service.origin.id` | e.g. `gy415nu...` | `event.requestContext.apiId` | `context.service.origin.version` | e.g. `1.0` | `1.0` for API Gateway V1, `2.0` for API Gateway V2. | `event.version` (or `1.0` if that field is not present) -`context.cloud.origin.service.name` | `api gateway` | Fix value for API gateway. | - +`context.cloud.origin.service.name` | `api gateway` | Constant value for API gateway. | - `context.cloud.origin.account.id` | e.g. `12345678912` | Account ID of the API gateway. | `event.requestContext.accountId` -`context.cloud.origin.provider` | `aws` | Use `aws` as fix value. | - +`context.cloud.origin.provider` | `aws` | Use `aws` as constant value. | - **Set `transaction.name` for the API Gateway trigger** @@ -132,6 +132,66 @@ In version 2.0, the `${event.requestContext.routeKey}` can have the format `GET If `use_path_as_transaction_name` is applicable and set to `true`, use `${event.requestContext.http.method} ${event.requestContext.http.path}` as the transaction name. +### Elastic Load Balancer (ELB) + +Elastic Load Balancer (ELB) can be attached directly to lambda, without the use +of API Gateway. In this case the `event` object will be structured differently. + +The agent should use the information in the request and response objects to +fill the HTTP context (`context.request` and `context.response`) fields in the +same way it is done for HTTP transactions. + +In particular, agents must use the `event.headers` to retrieve the +`traceparent` and the `tracestate` and use them to start the transaction for +the lambda function execution. + +In addition the following fields should be set for ELB-based Lambda functions: + +Field | Value | Description | Source +--- | --- | --- | --- +`type` | `request`| Transaction type: constant value for ELB. | - +`name` | e.g. `GET /prod/proxy/{proxy+}` | Transaction name: Http method followed by a whitespace and the (resource) path. See section below. | - +`transaction.result` | `HTTP Xxx` / `success` | `HTTP 5xx` if there was a function error (see [Lambda error handling doc](https://docs.aws.amazon.com/lambda/latest/dg/services-apigateway.html#services-apigateway-errors). If the [invocation response has a "statusCode" field](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-develop-integrations-lambda.html#http-api-develop-integrations-lambda.response), then set to `HTTP Xxx` based on the status code, otherwise `success`. | Error or `response.statusCode`. +`faas.trigger.type` | `http` | Constant value for ELB. | - +`faas.trigger.request_id` | e.g. `Root=1-xxxxxxxxxxxxxx` | AWS Trace ID of the ELB request. | `event.headers.x-amzn-trace-id` +`context.service.origin.name` | e.g. `targetgroup/lambda...5c45c6791a` | ELB target group | Derived from the 6th segment of `event.requestContext.elb.targetGroupArn` +`context.service.origin.id` | e.g. `arn:aws:elasticlo...65c45c6791a` | ELB target group ARN | `event.requestContext.elb.targetGroupArn` | +`context.cloud.origin.service.name` | `elb` | Constant value for ELB. | - +`context.cloud.origin.account.id` | e.g. `123456789012` | Account ID of the ELB. | Derived from the 5th segment of `event.requestContext.elb.targetGroupArn` +`context.cloud.origin.provider` | `aws` | Use `aws` as constant value. | - + +Note that the `context.service.origin.version` is omitted for ELB requests. + +An example ELB event: + +``` +{ + "requestContext": { + "elb": { + "targetGroupArn": "arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/lambda-279XGJDqGZ5rsrHC2Fjr/49e9d65c45c6791a" + } + }, + "httpMethod": "POST", + "path": "/toolz/api/v2.0/downloadPDF/PDF_2020-09-11_11-06-01.pdf", + "queryStringParameters": { + "test%40key": "test%40value", + "language": "en-DE" + }, + "headers": { + "accept-encoding": "gzip,deflate", + "connection": "Keep-Alive", + "host": "blabla.com", + "user-agent": "Apache-HttpClient/4.5.13 (Java/11.0.15)", + "x-amzn-trace-id": "Root=1-xxxxxxxxxxxxxx", + "x-forwarded-for": "199.99.99.999", + "x-forwarded-port": "443", + "x-forwarded-proto": "https" + }, + "body": "blablablabody", + "isBase64Encoded": false +} +``` + ### SQS / SNS Lambda functions that are triggered by SQS (or SNS) accept an `event` input that may contain one or more SQS / SNS messages in the `event.records` array. All message-related context information (including the `traceparent`) is encoded in the individual message attributes (if at all). @@ -154,10 +214,10 @@ Field | Value | Description | Source `faas.trigger.type` | `pubsub` | Constant value for message based triggers | - `context.service.origin.name` | e.g. `my-queue` | SQS queue name | Simple queue name can be derived from the 6th segment of `records[0].eventSourceArn`. `context.service.origin.id` | e.g. `arn:aws:sqs:us-east-2:123456789012:my-queue` | SQS queue ARN. | `records[0].eventSourceArn` -`context.cloud.origin.service.name` | `sqs` | Fix value for SQS. | - +`context.cloud.origin.service.name` | `sqs` | Constant value for SQS. | - `context.cloud.origin.region` | e.g. `us-east-1` | SQS queue region. | `records[0].awsRegion` `context.cloud.origin.account.id` | e.g. `12345678912` | Account ID of the SQS queue. | Parse account segment (5th) from `records[0].eventSourceArn`. -`context.cloud.origin.provider` | `aws` | Use `aws` as fix value. | - +`context.cloud.origin.provider` | `aws` | Use `aws` as constant value. | - An example SQS event: @@ -224,10 +284,10 @@ Field | Value | Description | Source `faas.trigger.type` | `pubsub` | Constant value for message based triggers | - `context.service.origin.name` | e.g. `my-topic` | SNS topic name | Simple topic name can be derived from the 6th segment of `records[0].sns.topicArn`. `context.service.origin.id` | e.g. `arn:aws:sns:us-east-2:123456789012:my-topic` | SNS topic ARN. | `records[0].sns.topicArn` -`context.cloud.origin.service.name` | `sns` | Fix value for SNS. | - +`context.cloud.origin.service.name` | `sns` | Constant value for SNS. | - `context.cloud.origin.region` | e.g. `us-east-1` | SNS topic region. | Parse region segment (4th) from `records[0].sns.topicArn`. `context.cloud.origin.account.id` | e.g. `12345678912` | Account ID of the SNS topic. | Parse account segment (5th) from `records[0].sns.topicArn`. -`context.cloud.origin.provider` | `aws` | Use `aws` as fix value. | - +`context.cloud.origin.provider` | `aws` | Use `aws` as constant value. | - An example SNS event: @@ -285,9 +345,9 @@ Field | Value | Description | Source `context.service.origin.name` | e.g. `mybucket` | S3 bucket name. | `record.s3.bucket.name` `context.service.origin.id` | e.g. `arn:aws:s3:::mybucket` | S3 bucket ARN. | `record.s3.bucket.arn` `context.service.origin.version` | e.g. `2.1` | S3 event version. | `record.eventVersion` -`context.cloud.origin.service.name` | `s3` | Fix value for S3. | - +`context.cloud.origin.service.name` | `s3` | Constant value for S3. | - `context.cloud.origin.region` | e.g. `us-east-1` | S3 bucket region. | `record.awsRegion` -`context.cloud.origin.provider` | `aws` | Use `aws` as fix value. | - +`context.cloud.origin.provider` | `aws` | Use `aws` as constant value. | - ## Data Flushing Lambda functions are immediately frozen as soon as the handler method ends. In case APM data is sent in an asyncronous way (as most of the agents do by default) data can get lost if not sent before the lambda function ends. From 152939ecb867ffdc203a9c6895e1d6abc6bf63ef Mon Sep 17 00:00:00 2001 From: Colton Myers Date: Mon, 8 Aug 2022 12:44:14 -0600 Subject: [PATCH 2/7] Small fix --- specs/agents/tracing-instrumentation-aws-lambda.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/agents/tracing-instrumentation-aws-lambda.md b/specs/agents/tracing-instrumentation-aws-lambda.md index 9165b0b7..6e6c964e 100644 --- a/specs/agents/tracing-instrumentation-aws-lambda.md +++ b/specs/agents/tracing-instrumentation-aws-lambda.md @@ -157,7 +157,7 @@ Field | Value | Description | Source `context.service.origin.name` | e.g. `targetgroup/lambda...5c45c6791a` | ELB target group | Derived from the 6th segment of `event.requestContext.elb.targetGroupArn` `context.service.origin.id` | e.g. `arn:aws:elasticlo...65c45c6791a` | ELB target group ARN | `event.requestContext.elb.targetGroupArn` | `context.cloud.origin.service.name` | `elb` | Constant value for ELB. | - -`context.cloud.origin.account.id` | e.g. `123456789012` | Account ID of the ELB. | Derived from the 5th segment of `event.requestContext.elb.targetGroupArn` +`context.cloud.origin.account.id` | e.g. `123456789012` | Account ID for the ELB. | Derived from the 5th segment of `event.requestContext.elb.targetGroupArn` `context.cloud.origin.provider` | `aws` | Use `aws` as constant value. | - Note that the `context.service.origin.version` is omitted for ELB requests. From a9b17d61fb4ed21f680d50d3dd4b73a8ec26ba22 Mon Sep 17 00:00:00 2001 From: Colton Myers Date: Wed, 10 Aug 2022 16:18:26 -0600 Subject: [PATCH 3/7] Update specs/agents/tracing-instrumentation-aws-lambda.md Co-authored-by: Trent Mick --- specs/agents/tracing-instrumentation-aws-lambda.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/agents/tracing-instrumentation-aws-lambda.md b/specs/agents/tracing-instrumentation-aws-lambda.md index 6e6c964e..9e33ff13 100644 --- a/specs/agents/tracing-instrumentation-aws-lambda.md +++ b/specs/agents/tracing-instrumentation-aws-lambda.md @@ -151,7 +151,7 @@ Field | Value | Description | Source --- | --- | --- | --- `type` | `request`| Transaction type: constant value for ELB. | - `name` | e.g. `GET /prod/proxy/{proxy+}` | Transaction name: Http method followed by a whitespace and the (resource) path. See section below. | - -`transaction.result` | `HTTP Xxx` / `success` | `HTTP 5xx` if there was a function error (see [Lambda error handling doc](https://docs.aws.amazon.com/lambda/latest/dg/services-apigateway.html#services-apigateway-errors). If the [invocation response has a "statusCode" field](https://docs.aws.amazon.com/apigateway/latest/developerguide/http-api-develop-integrations-lambda.html#http-api-develop-integrations-lambda.response), then set to `HTTP Xxx` based on the status code, otherwise `success`. | Error or `response.statusCode`. +`transaction.result` | `HTTP Xxx` / `success` | `HTTP 5xx` if there was a function error. If the [invocation response has a "statusCode" field](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/lambda-functions.html#respond-to-load-balancer), then set to `HTTP Xxx` based on the status code, otherwise `success`. | Error or `response.statusCode`. `faas.trigger.type` | `http` | Constant value for ELB. | - `faas.trigger.request_id` | e.g. `Root=1-xxxxxxxxxxxxxx` | AWS Trace ID of the ELB request. | `event.headers.x-amzn-trace-id` `context.service.origin.name` | e.g. `targetgroup/lambda...5c45c6791a` | ELB target group | Derived from the 6th segment of `event.requestContext.elb.targetGroupArn` From 40e2dfe0e6e40be7093d1f687a2731b27e5d9315 Mon Sep 17 00:00:00 2001 From: Colton Myers Date: Wed, 10 Aug 2022 16:21:37 -0600 Subject: [PATCH 4/7] Review suggestions --- specs/agents/tracing-instrumentation-aws-lambda.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/specs/agents/tracing-instrumentation-aws-lambda.md b/specs/agents/tracing-instrumentation-aws-lambda.md index 9e33ff13..5c1e00eb 100644 --- a/specs/agents/tracing-instrumentation-aws-lambda.md +++ b/specs/agents/tracing-instrumentation-aws-lambda.md @@ -140,6 +140,7 @@ of API Gateway. In this case the `event` object will be structured differently. The agent should use the information in the request and response objects to fill the HTTP context (`context.request` and `context.response`) fields in the same way it is done for HTTP transactions. +[Request/Response Docs](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/lambda-functions.html#receive-event-from-load-balancer) In particular, agents must use the `event.headers` to retrieve the `traceparent` and the `tracestate` and use them to start the transaction for @@ -153,11 +154,12 @@ Field | Value | Description | Source `name` | e.g. `GET /prod/proxy/{proxy+}` | Transaction name: Http method followed by a whitespace and the (resource) path. See section below. | - `transaction.result` | `HTTP Xxx` / `success` | `HTTP 5xx` if there was a function error. If the [invocation response has a "statusCode" field](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/lambda-functions.html#respond-to-load-balancer), then set to `HTTP Xxx` based on the status code, otherwise `success`. | Error or `response.statusCode`. `faas.trigger.type` | `http` | Constant value for ELB. | - -`faas.trigger.request_id` | e.g. `Root=1-xxxxxxxxxxxxxx` | AWS Trace ID of the ELB request. | `event.headers.x-amzn-trace-id` +`faas.trigger.request_id` | e.g. `Root=1-5bdb40ca-556d8b0c50dc66f0511bf520` | AWS Trace ID of the ELB request. | `event.headers.x-amzn-trace-id` `context.service.origin.name` | e.g. `targetgroup/lambda...5c45c6791a` | ELB target group | Derived from the 6th segment of `event.requestContext.elb.targetGroupArn` `context.service.origin.id` | e.g. `arn:aws:elasticlo...65c45c6791a` | ELB target group ARN | `event.requestContext.elb.targetGroupArn` | `context.cloud.origin.service.name` | `elb` | Constant value for ELB. | - `context.cloud.origin.account.id` | e.g. `123456789012` | Account ID for the ELB. | Derived from the 5th segment of `event.requestContext.elb.targetGroupArn` +`context.cloud.origin.region` | e.g. `us-east-2` | Cloud region. | Derived from the 4th segment of `event.requestContext.elb.targetGroupArn` `context.cloud.origin.provider` | `aws` | Use `aws` as constant value. | - Note that the `context.service.origin.version` is omitted for ELB requests. @@ -182,7 +184,7 @@ An example ELB event: "connection": "Keep-Alive", "host": "blabla.com", "user-agent": "Apache-HttpClient/4.5.13 (Java/11.0.15)", - "x-amzn-trace-id": "Root=1-xxxxxxxxxxxxxx", + "x-amzn-trace-id": "Root=1-5bdb40ca-556d8b0c50dc66f0511bf520", "x-forwarded-for": "199.99.99.999", "x-forwarded-port": "443", "x-forwarded-proto": "https" From 4631e6e991c97df12f0b6330e92e2eea55210296 Mon Sep 17 00:00:00 2001 From: Colton Myers Date: Thu, 11 Aug 2022 11:21:59 -0600 Subject: [PATCH 5/7] Add `lambda url` cloud.origin.service.name --- specs/agents/tracing-instrumentation-aws-lambda.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specs/agents/tracing-instrumentation-aws-lambda.md b/specs/agents/tracing-instrumentation-aws-lambda.md index 5c1e00eb..73dca0ac 100644 --- a/specs/agents/tracing-instrumentation-aws-lambda.md +++ b/specs/agents/tracing-instrumentation-aws-lambda.md @@ -82,7 +82,7 @@ Lambda functions can be triggered in many different ways. A generic transaction If none of the above apply, the fallback should be a generic instrumentation (as described above) that can deal with any type of trigger (thus capturing only the minimal available information). -### API Gateway (V1 & V2) +### API Gateway / Lambda URLS There are two different API Gateway versions (V1 & V2) that differ slightly in the information (`event` object) that is passed to the Lambda handler function. With both versions, the `event` object contains information about the http request. @@ -103,7 +103,7 @@ Field | Value | Description | Source `context.service.origin.name` | e.g. `gdnrpwmtsb...amazonaws.com` | The full domain name of the API Gateway. | `event.requestContext.domainName` `context.service.origin.id` | e.g. `gy415nu...` | `event.requestContext.apiId` | `context.service.origin.version` | e.g. `1.0` | `1.0` for API Gateway V1, `2.0` for API Gateway V2. | `event.version` (or `1.0` if that field is not present) -`context.cloud.origin.service.name` | `api gateway` | Constant value for API gateway. | - +`context.cloud.origin.service.name` | `api gateway` or `lambda url` | Constant value. | Detect lambda URLs by searching for `.lambda-url.` in the URL. Otherwise assume API Gateway. `context.cloud.origin.account.id` | e.g. `12345678912` | Account ID of the API gateway. | `event.requestContext.accountId` `context.cloud.origin.provider` | `aws` | Use `aws` as constant value. | - From 7920a6fff7742418057eed15386eca6d65679335 Mon Sep 17 00:00:00 2001 From: Colton Myers Date: Thu, 11 Aug 2022 11:23:40 -0600 Subject: [PATCH 6/7] Be more precise about location of URL --- specs/agents/tracing-instrumentation-aws-lambda.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specs/agents/tracing-instrumentation-aws-lambda.md b/specs/agents/tracing-instrumentation-aws-lambda.md index 73dca0ac..c187fd80 100644 --- a/specs/agents/tracing-instrumentation-aws-lambda.md +++ b/specs/agents/tracing-instrumentation-aws-lambda.md @@ -103,7 +103,7 @@ Field | Value | Description | Source `context.service.origin.name` | e.g. `gdnrpwmtsb...amazonaws.com` | The full domain name of the API Gateway. | `event.requestContext.domainName` `context.service.origin.id` | e.g. `gy415nu...` | `event.requestContext.apiId` | `context.service.origin.version` | e.g. `1.0` | `1.0` for API Gateway V1, `2.0` for API Gateway V2. | `event.version` (or `1.0` if that field is not present) -`context.cloud.origin.service.name` | `api gateway` or `lambda url` | Constant value. | Detect lambda URLs by searching for `.lambda-url.` in the URL. Otherwise assume API Gateway. +`context.cloud.origin.service.name` | `api gateway` or `lambda url` | Constant value. | Detect lambda URLs by searching for `.lambda-url.` in the `event.requestContext.domainName`. Otherwise assume API Gateway. `context.cloud.origin.account.id` | e.g. `12345678912` | Account ID of the API gateway. | `event.requestContext.accountId` `context.cloud.origin.provider` | `aws` | Use `aws` as constant value. | - From 07c491e6e0ca2b2a0442808fea267a9f91984ef3 Mon Sep 17 00:00:00 2001 From: Colton Myers Date: Mon, 22 Aug 2022 12:36:49 -0600 Subject: [PATCH 7/7] Remove request_id and narrow service.origin.name --- specs/agents/tracing-instrumentation-aws-lambda.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/specs/agents/tracing-instrumentation-aws-lambda.md b/specs/agents/tracing-instrumentation-aws-lambda.md index c187fd80..81fd2ed7 100644 --- a/specs/agents/tracing-instrumentation-aws-lambda.md +++ b/specs/agents/tracing-instrumentation-aws-lambda.md @@ -154,8 +154,7 @@ Field | Value | Description | Source `name` | e.g. `GET /prod/proxy/{proxy+}` | Transaction name: Http method followed by a whitespace and the (resource) path. See section below. | - `transaction.result` | `HTTP Xxx` / `success` | `HTTP 5xx` if there was a function error. If the [invocation response has a "statusCode" field](https://docs.aws.amazon.com/elasticloadbalancing/latest/application/lambda-functions.html#respond-to-load-balancer), then set to `HTTP Xxx` based on the status code, otherwise `success`. | Error or `response.statusCode`. `faas.trigger.type` | `http` | Constant value for ELB. | - -`faas.trigger.request_id` | e.g. `Root=1-5bdb40ca-556d8b0c50dc66f0511bf520` | AWS Trace ID of the ELB request. | `event.headers.x-amzn-trace-id` -`context.service.origin.name` | e.g. `targetgroup/lambda...5c45c6791a` | ELB target group | Derived from the 6th segment of `event.requestContext.elb.targetGroupArn` +`context.service.origin.name` | e.g. `targetgroup` | ELB target group | `event.requestContext.elb.targetGroupArn` is formed as `arn:aws:elasticloadbalancing:region-code:account-id:targetgroup/target-group-name/target-group-id`, so use `targetGroupArn.split(':')[5].split('/')[1]` to get the `target-group-name`. `context.service.origin.id` | e.g. `arn:aws:elasticlo...65c45c6791a` | ELB target group ARN | `event.requestContext.elb.targetGroupArn` | `context.cloud.origin.service.name` | `elb` | Constant value for ELB. | - `context.cloud.origin.account.id` | e.g. `123456789012` | Account ID for the ELB. | Derived from the 5th segment of `event.requestContext.elb.targetGroupArn`