feat: automate AppSec enablement setup (e.g: `AWS_LAMBDA_RUNTIME_API`) #143

RomainMuller · 2023-10-05T12:14:26Z

What does this PR do?

In order to simplify onboarding & make it more uniform across languages, inspect the value of the DD_SERVERLESS_APPSEC_ENABLED environment variable, and overrides AWS_LAMBDA_RUNTIME_API to redirect the Lambda runtime API to the Datadog extension so that the AppSec proxy is used, and turns on DD_UNIVERSAL_INSTRUMENTATION (unless it was already manually configured by the user).

Motivation

This is necessary/useful because the AWS_LAMBDA_EXEC_WRAPPER environment variable is not honored by custom runtimes (provided, provided.al2) as well as the go1.x runtime (which is a glorified provided runtime). The datadog Lambda wrapper starts a proxy to inject ASM functionality directly on the Lambda runtime API instead of having to manually instrument each and every lambda handler/application, and modifies AWS_LAMBDA_RUNTIME_API to instruct Lambda language runtime client libraries to go through it instead of directly interacting with the Lambda control plane.

Testing Guidelines

Deployed it on AWS Lambda with a demo app of mine and observed the expected behavior.

Types of changes

Bug fix
New feature
Breaking change
Misc (docs, refactoring, dependency upgrade, etc.)

Checklist

This PR's description is comprehensive
This PR contains breaking changes that are documented in the description
This PR introduces new APIs or parameters that are documented and unlikely to change in the foreseeable future
This PR impacts documentation, and it has been updated (or a ticket has been logged)
This PR's changes are covered by the automated tests
This PR collects user input/sensitive content into Datadog

In order to simplify onboarding & make it more uniform across languages, inspect the value of the `AWS_LAMBDA_EXEC_WRAPPER` environment variable and apply select environment variable changes it perofrms upon decorating a handler. This is necessary/useful because that environment variable is not honored by custom runtimes (`provided`, `provided.al2`) as well as the `go1.x` runtime (which is a glorified provided runtime). The datadog Lambda wrapper starts a proxy to inject ASM functionality directly on the Lambda runtime API instead of having to manually instrument each and every lambda handler/application, and modifies `AWS_LAMBDA_RUNTIME_API` to instruct Lambda language runtime client libraries to go through it instead of directly interacting with the Lambda control plane. APPSEC-11534

codecov-commenter · 2023-10-05T12:17:45Z

Codecov Report

Merging #143 (f146c38) into main (1ca51d5) will decrease coverage by 1.20%.
The diff coverage is 23.80%.

❗ Current head f146c38 differs from pull request most recent head 95f979a. Consider uploading reports for the commit 95f979a to get more accurate results

@@            Coverage Diff             @@
##             main     #143      +/-   ##
==========================================
- Coverage   83.60%   82.40%   -1.20%     
==========================================
  Files          13       13              
  Lines         860      881      +21     
==========================================
+ Hits          719      726       +7     
- Misses        113      126      +13     
- Partials       28       29       +1

Files	Coverage Δ
ddlambda.go	`70.16% <23.80%> (-9.46%)`	⬇️

... and 1 file with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

purple4reina

So, unfortunately, I don't think this does what you think it's doing. I could be wrong, in which case, we should explore this further.

During the initialization phase of a lambda invocation, function initialization happens after extensions and the runtime are initialized. This means that the AWS_LAMBDA_RUNTIME_API env var will get set after the runtime is configured, which is too late.

I tested this out to confirm and indeed did not see any calls to the appsec proxy despite seeing that it was running.

You can read more about the lambda runtime here: https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtime-environment.html

purple4reina · 2023-10-19T15:47:14Z

ddlambda.go

+func setupAppSec() {
+	const ServerlessAppSecEnabledEnvVar = "DD_SERVERLESS_APPSEC_ENABLED"
+	const AwsLambdaRuntimeApiEnvVar = "AWS_LAMBDA_RUNTIME_API"
+	const DatadogAgentUrl = "127.0.0.1:9000"


These should probably be globals.

I don't suppose those would be tremendously useful outside of this particular scope and since they're constants I don't see much of a point hoisting them up into the module namespace. I can do that if that is a strong style preference...

purple4reina · 2023-10-19T15:51:22Z

ddlambda.go

+		return
+	}
+
+	_ = os.Setenv(AwsLambdaRuntimeApiEnvVar, DatadogAgentUrl)


We should consider checking that the extension is indeed present before doing this redirect.

How do you suggest doing this? Just a file presence check? I could do that...

ddlambda.go

RomainMuller · 2023-10-20T09:09:28Z

I tested this out to confirm and indeed did not see any calls to the appsec proxy despite seeing that it was running.

I actually did test this out on my end and it produced the expected behavior. As far as I could tell from the aws-lambda-go source, the environment variable is not read before its own wrapper function is called, so you "simply" have to have the environment variable configured before this happens and you're golden. This particular injection point appears to be "fine" for this particular purpose.

This now makes me feel I need to test that again to make sure I did not get a false positive because of some other configuration...

RomainMuller · 2023-10-20T09:41:21Z

Did some more research into my previous findings w/r/t aws-lambda-go, and found the place where the environment variable is read is not in an init function but in the start function that essentially is called when StartWithOptions is called, which I believe is guaranteed to happen AFTER our changes to env are made...

https://github.com/aws/aws-lambda-go/blob/771b391678d3f54bfa38531774d656f5e0f2ab58/lambda/entry.go#L102

https://github.com/aws/aws-lambda-go/blob/771b391678d3f54bfa38531774d656f5e0f2ab58/lambda/entry.go#L69

purple4reina · 2023-10-24T16:26:12Z

ddlambda.go

+
+	if awsLambdaRpcSupport {
+		if port := os.Getenv(awsLambdaServerPortEnvVar); port != "" {
+			logger.Warn(fmt.Sprintf("%s activation with the go1.x AWS Lambda runtime requires setting the `lambda.norpc` go build tag", serverlessAppSecEnabledEnvVar))


So, I'd hate to say that setting this build tag is required to use appsec, when in fact it is not. Alternatively, they can always set these environment variables manually.

DataDog#143) * feat: honor AWS_LAMBDA_EXEC_WRAPPER when AWS Lambda does not In order to simplify onboarding & make it more uniform across languages, inspect the value of the `AWS_LAMBDA_EXEC_WRAPPER` environment variable and apply select environment variable changes it perofrms upon decorating a handler. This is necessary/useful because that environment variable is not honored by custom runtimes (`provided`, `provided.al2`) as well as the `go1.x` runtime (which is a glorified provided runtime). The datadog Lambda wrapper starts a proxy to inject ASM functionality directly on the Lambda runtime API instead of having to manually instrument each and every lambda handler/application, and modifies `AWS_LAMBDA_RUNTIME_API` to instruct Lambda language runtime client libraries to go through it instead of directly interacting with the Lambda control plane. APPSEC-11534 * pivot to a different, cheaper strategy * typo fix * PR feedback * minor fixups * add warning in go1.x runtime if lambda.norpc build tag was not enabled

* Create codeql-analysis.yml (DataDog#100) * Create codeql-analysis.yml * Update codeql-analysis.yml * Update run_integration_tests.sh * Do not show error messages even if neither DD_API_KEY nor DD_KMS_API_KEY is set when Lambda Extension is running (DataDog#102) * Bump version to 1.4.0 * Bump go + fasthttp + lint (DataDog#104) * Consolidate serverless configurations into one place (DataDog#105) * Update README.md * Update README.md * Bump dd-trace-go to latest version to address some vulnerabilities (DataDog#109) * Bump dd-trace-go to latest version to address some vulnaribilities * update go.sum with `go mod tidy` * Bump version to 1.6.0 * bump codeql (DataDog#112) * Bump dd-trace-go to v1.41 (DataDog#115) * Bump version to 1.7.0 * [SLS-2330] Add support for universal instrumentation with the extension (DataDog#116) add option to use universal instrumentation * [EEP-444] include error in failed metric send log (DataDog#118) Co-authored-by: Corey Griffin <CoreyGriffin@users.noreply.github.com> * [SLS-2492] Upgrade aws sdk v2 (DataDog#113) upgrade sdk * Bump version to 1.8.0 * Use new account in integration tests (DataDog#119) * set the architecture explicitely (DataDog#122) * mask init runtime logs (DataDog#123) * Update libs (DataDog#121) * bump go 1.18 (DataDog#125) * Retry sending trace payloads on failure. (DataDog#128) * Bump version to 1.9.0 * Update DD Trace to v1.51.0(DataDog#133) * Bump go version to 1.20 (DataDog#140) Bump go version to 1.20 * Upgrade version of dd-trace-go to v1.54.1 (DataDog#141) * Bump version to 1.10.0 * Propagate trace context from SQS events (DataDog#142) * Default parent id to be trace id if not found elsewhere. * Look for trace context in context object as well as headers. * Apply trace context before starting the function execution span. * Update signature in tests. * Add spanid of execution span to context. * Do not ignore priority "-128". * Test that default parent id set to trace id. * Test span id added to context. * Test uses trace context from context object. * Bump version to 1.11.0 * feat: automate AppSec enablement setup (e.g: `AWS_LAMBDA_RUNTIME_API`) (DataDog#143) * feat: honor AWS_LAMBDA_EXEC_WRAPPER when AWS Lambda does not In order to simplify onboarding & make it more uniform across languages, inspect the value of the `AWS_LAMBDA_EXEC_WRAPPER` environment variable and apply select environment variable changes it perofrms upon decorating a handler. This is necessary/useful because that environment variable is not honored by custom runtimes (`provided`, `provided.al2`) as well as the `go1.x` runtime (which is a glorified provided runtime). The datadog Lambda wrapper starts a proxy to inject ASM functionality directly on the Lambda runtime API instead of having to manually instrument each and every lambda handler/application, and modifies `AWS_LAMBDA_RUNTIME_API` to instruct Lambda language runtime client libraries to go through it instead of directly interacting with the Lambda control plane. APPSEC-11534 * pivot to a different, cheaper strategy * typo fix * PR feedback * minor fixups * add warning in go1.x runtime if lambda.norpc build tag was not enabled * Bump version to 1.12.0 * Re-add configs after upstream rebase * Bump packages * Remove deprecated `io/ioutil` calls --------- Co-authored-by: Tian Chu <tian.chu@datadoghq.com> Co-authored-by: Soshi Katsuta <skatsuta@users.noreply.github.com> Co-authored-by: Maxime David <maxime.david@datadoghq.com> Co-authored-by: kimi <47579703+kimi-p@users.noreply.github.com> Co-authored-by: Kimi Wu <kimi.wu@datadoghq.com> Co-authored-by: Dylan Yang <dylan.yang@datadoghq.com> Co-authored-by: Corey Griffin <15809365+CoreyGriffin@users.noreply.github.com> Co-authored-by: Corey Griffin <CoreyGriffin@users.noreply.github.com> Co-authored-by: Marcin Rabenda <xrn.design@gmail.com> Co-authored-by: Rey Abolofia <purple4reina@gmail.com> Co-authored-by: Rey Abolofia <rey.abolofia@datadoghq.com> Co-authored-by: Andrew Rodriguez <49878080+zARODz11z@users.noreply.github.com> Co-authored-by: Ivan Topolcic <IvanTopolcic@users.noreply.github.com> Co-authored-by: Romain Marcadier <romain.muller@telecomnancy.net>

RomainMuller force-pushed the main branch from ddd6c96 to c95006f Compare October 5, 2023 12:16

RomainMuller mentioned this pull request Oct 5, 2023

Support for AWS_LAMBDA_EXEC_WRAPPER aws/aws-lambda-go#523

Open

pivot to a different, cheaper strategy

5df319d

RomainMuller force-pushed the main branch from e166c25 to 5df319d Compare October 19, 2023 12:54

RomainMuller marked this pull request as ready for review October 19, 2023 12:58

RomainMuller requested a review from a team as a code owner October 19, 2023 12:58

RomainMuller changed the title ~~feat: honor AWS_LAMBDA_EXEC_WRAPPER when AWS Lambda does not~~ feat: automate AppSec enablement setup (e.g: AWS_LAMBDA_RUNTIME_API) Oct 19, 2023

typo fix

50c6900

purple4reina requested changes Oct 19, 2023

View reviewed changes

PR feedback

df67ed7

RomainMuller requested a review from purple4reina October 20, 2023 09:43

minor fixups

95f979a

RomainMuller mentioned this pull request Oct 23, 2023

chore: add integration tests around Serverless ASM DataDog/datadog-agent#20320

Merged

10 tasks

add warning in go1.x runtime if lambda.norpc build tag was not enabled

5147ad1

purple4reina reviewed Oct 24, 2023

View reviewed changes

purple4reina approved these changes Oct 26, 2023

View reviewed changes

purple4reina merged commit 7ec9261 into DataDog:main Oct 30, 2023
5 of 6 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: automate AppSec enablement setup (e.g: `AWS_LAMBDA_RUNTIME_API`) #143

feat: automate AppSec enablement setup (e.g: `AWS_LAMBDA_RUNTIME_API`) #143

RomainMuller commented Oct 5, 2023 •

edited

codecov-commenter commented Oct 5, 2023 •

edited

purple4reina left a comment

purple4reina Oct 19, 2023

RomainMuller Oct 20, 2023

purple4reina Oct 19, 2023

RomainMuller Oct 20, 2023

RomainMuller commented Oct 20, 2023 •

edited

RomainMuller commented Oct 20, 2023

purple4reina Oct 24, 2023

feat: automate AppSec enablement setup (e.g: AWS_LAMBDA_RUNTIME_API) #143

feat: automate AppSec enablement setup (e.g: AWS_LAMBDA_RUNTIME_API) #143

Conversation

RomainMuller commented Oct 5, 2023 • edited

What does this PR do?

Motivation

Testing Guidelines

Types of changes

Checklist

codecov-commenter commented Oct 5, 2023 • edited

Codecov Report

purple4reina left a comment

Choose a reason for hiding this comment

purple4reina Oct 19, 2023

Choose a reason for hiding this comment

RomainMuller Oct 20, 2023

Choose a reason for hiding this comment

purple4reina Oct 19, 2023

Choose a reason for hiding this comment

RomainMuller Oct 20, 2023

Choose a reason for hiding this comment

RomainMuller commented Oct 20, 2023 • edited

RomainMuller commented Oct 20, 2023

purple4reina Oct 24, 2023

Choose a reason for hiding this comment

feat: automate AppSec enablement setup (e.g: `AWS_LAMBDA_RUNTIME_API`) #143

feat: automate AppSec enablement setup (e.g: `AWS_LAMBDA_RUNTIME_API`) #143

RomainMuller commented Oct 5, 2023 •

edited

codecov-commenter commented Oct 5, 2023 •

edited

RomainMuller commented Oct 20, 2023 •

edited