sam local start-api broken due to missing lambda-runtime-invoked-function-arn header since v1.13.0

[ramosbugs]

Description:

Up to version v1.12.0, I was able to test my API Gateway REST API using sam local start-api with a Lambda built on top of https://github.com/awslabs/aws-lambda-rust-runtime. However, this functionality stopped working in v1.13.0 due to the missing lambda-runtime-invoked-function-arn header that the runtime expects:

2020-12-04 17:54:54,881 | Found one Lambda function with name 'ApiFunction'
2020-12-04 17:54:54,881 | Invoking ignored (provided)
2020-12-04 17:54:54,881 | Environment variables overrides data is standard format
2020-12-04 17:54:54,881 | Loading AWS credentials from session with profile 'None'
2020-12-04 17:54:54,896 | Resolving code path. Cwd=/Users/daramos/...[redacted], CodeUri=.aws-sam/build-backend/ApiFunction
2020-12-04 17:54:54,897 | Resolved absolute path to code is /Users/daramos/...[redacted].../.aws-sam/build-backend/ApiFunction
2020-12-04 17:54:54,897 | Code /Users/daramos/...[redacted].../.aws-sam/build-backend/ApiFunction is not a zip/jar file
2020-12-04 17:54:54,940 | Skip pulling image and use local one: amazon/aws-sam-cli-emulation-image-provided:rapid-1.13.0.

2020-12-04 17:54:54,940 | Mounting /Users/daramos/Documents/Projects/...[redacted].../.aws-sam/build-backend/ApiFunction as /var/task:ro,delegated inside runtime container
2020-12-04 17:54:55,581 | Starting a timer for 3 seconds for function 'ApiFunction'
START RequestId: 5d85e30d-cda5-47dc-a1a6-4ed2ed87268c Version: $LATEST
thread 'main' panicked at 'no entry found for key "lambda-runtime-invoked-function-arn"', /Users/daramos/.cargo/git/checkouts/aws-lambda-rust-runtime-7c865cce90132439/13aa8f0/lambda/src/types.rs:131:35

The code runs fine on production AWS Lambda, so I believe this is a bug in the SAM CLI's local emulation, not in the Rust runtime. For reference, the runtime expects several headers to be provided by Lambda:
https://github.com/awslabs/aws-lambda-rust-runtime/blob/13aa8f01813e6949ca91aedafb286591fdf32217/lambda/src/types.rs#L123-L139

It looks like aws/aws-sam-cli#2425 may have introduced this regression (cc @sriram-mv). The code that previously set these headers was removed in that PR:
https://github.com/aws/aws-sam-cli/blob/559e4a564ba37906f2081c0de415b39bc9b2c3c1/samcli/local/rapid/init.go#L482-L486

Steps to reproduce:

Use sam local start-api to test a REST API locally. In my case, using AWS Labs's Rust runtime.

Observed result:

Lambda function panics due to missing headers (the same Lambda works fine on AWS).

Expected result:

Lambda should execute without error.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: Mac OS 10.15.7
2. sam --version: 1.13.0 (also tested on 1.13.2 and latest develop HEAD).

Add --debug flag to command you are running

[jfuss]

@ramosbugs SAM CLI is no integrated with aws-lambda-rie rather than the community built one we previously had. It seems like that is not passing all the required fields. Locally, there isn't an arn to provide since there is no AWS Resource. If we just construct a fake ARN for this, do you think that is sufficient?

Moving over to the aws-lambda-rie github repo since this relates to that software directly.

[ramosbugs]

fake ARN sounds great, thanks!

[Alappin]

@jfuss I have been getting a similar error using the public.ecr.aws/lambda/provided:al2 docker image. It seems the aws-lambda-rie is not passinglambda-runtime-invoked-function-arn header. This makes it impossible to test custom container images locally, is there a fix for this?

It would be best to just return default mock values from aws-lambda-rie because other Lambda runtimes like like https://github.com/swift-server/swift-aws-lambda-runtime expect the following to return or they throw errors:

"Lambda-Runtime-Aws-Request-Id"
"Lambda-Runtime-Trace-Id"
"Lambda-Runtime-Deadline-Ms"
"Lambda-Runtime-Invoked-Function-Arn"

reference code https://github.com/swift-server/swift-aws-lambda-runtime/blob/a5ef4e9e04952bb457cf96dadd4787dcde92985a/Sources/AWSLambdaRuntimeCore/LambdaRuntimeClient.swift#L168

[bvicinus]

Having the same problem as @Alappin

We are trying to get away from third-party libraries and would love if this worked out of the box so we can use the aws supported and maintained version :)

[sj4nes]

Having the same issue. Is there a work-around?

Also what's weird is the Python handlers don't even blink at this missing header but the example Rust handler bails.

[dbanty]

Having the same issue. Is there a work-around?

the example Rust handler bails.

@sj4nes if you’re doing a Rust handler the newer lamedh crates have worked fine for me (https://github.com/lamedh-dev/aws-lambda-rust-runtime). I also had luck pinning sam-cli to 1.12.0 with the official runtime crates.

[dspatoulas]

This was fixed as part of #23, but it appears that a new version of the RIE was never published to https://github.com/aws/aws-lambda-runtime-interface-emulator/releases. The available releases still show v1.0, which was released on Nov 30, 2020 and predates the actual fix for the missing header in the emulator.

As a temporary work around, I had to compile the RIE binary from the default branch and manually insert the RIE into our lambda container images following the documentation from https://docs.aws.amazon.com/lambda/latest/dg/images-test.html#images-test-alternative. My projects are using the public.ecr.aws/lambda/provided:al2 image, which appears to have an outdated version of the RIE baked into the image.

It would be awesome if a new version could be published and made available as part of the base image from AWS. Currently, the official rust runtime from AWS panics when running locally due to the missing header (awslabs/aws-lambda-rust-runtime#313).

[ghost]

I'm experiencing a similar problem when the X-Amzn-Trace-Id header is not present.

Panics with: thread 'main' panicked at 'no entry found for key "lambda-runtime-trace-id"', /Users/felix.marezki/.cargo/registry/src/github.com-1ecc6299db9ec823/lambda_runtime-0.3.0/src/types.rs:135:28
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace

But maybe there that is more of an issue for the lambda_runtime!?

[nakedible-p]

I second all the other comments - please release new versions of public.ecr.aws/lambda/provided:al2 image!

[kenshih]

👋 just noting a suggested fix to the rust runtime: awslabs/aws-lambda-rust-runtime#332
Hope you are all well.