Invalid API Gateway Response Keys: {'base64Encoded'} in {'statusCode': 200, 'headers': {'Content-Type': 'text/plain'}, 'body': 'Hello World', 'base64Encoded': False}

[ghost]

Description

We are using aws-serverless-java-container-core to set up our API Gateway proxy lambdas. Each lambda returns an AwsProxyResponse. When run in AWS these lambdas work fine through the configured API Gateways; however, when testing locally with the aws-sam-cli (sam local start-api) we are only able to receive 502 Bad Gateway responses from the local gateway.

It appears that release 0.16.0 of aws-sam-cli added validation to the parameters returned from lambdas (#1154). When we check the logs for our local lambdas we find that the property for base64encoding is not respecting the declared JsonProperty annotation and is being serialized with a value different that the one that is expected. The serialized value is consistent between the local and AWS environments.

This is a similar issue to the one described in issue 830, but it is unrelated. We are not using the multiValueHeaders field in any of our lambda responses, and the attached error log does not contain this field.

The issue we are facing is that the "isBase64Encoded" field of the AwsProxyResponse object is being serialized with a name of "base64Encoded" rather than the expected name of "isBase64Encoded". This is resulting in all requests to the local gateway returning 502 regardless of the actual status of the lambda. We are using the AwsProxyResponse object provided by aws-serverless-java-container-core defined here: https://github.com/awslabs/aws-serverless-java-container/blob/master/aws-serverless-java-container-core/src/main/java/com/amazonaws/serverless/proxy/model/AwsProxyResponse.java

Downgrading the aws-sam-cli to version 0.15.0 does prevent the issue from occurring as there is no validation logic present in that version

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

aws-sam-cli version: 0.16.1
aws-serverless-java-container-core version: 1.3

Add --debug flag to command you are running

Error Log:
2019-05-22 14:56:37 Invalid API Gateway Response Keys: {'base64Encoded'} in {'statusCode': 200, 'headers': {'Content-Type': 'text/plain'}, 'body': 'Hello World', 'base64Encoded': False}
2019-05-22 14:56:37 Function returned an invalid response (must include one of: body, headers or statusCode in the response object). Response received: {"statusCode":200,"headers":{"Content-Type":"text/plain"},"body":"Hello World","base64Encoded":false}
2019-05-22 14:56:37 127.0.0.1 - - [22/May/2019 14:56:37] "GET /echoStatusCode/200 HTTP/1.1" 502 -

[jfuss]

@psaws This is a duplicate of #830. See my comment here: #1190 (comment)

Closing as duplicate.

[jfuss]

I miss read the issue, my fault.

[jfuss]

@psaws SAM CLI does not serialize the data. The expected key is "isBase64Encoded" from https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html#api-gateway-simple-proxy-for-lambda-output-format. You should cut an issue against the aws-serverless-java-container-core project.

[vladamx]

It seems like this issue was fixed on aws-serverless-java-container-core side in 1.3.2, but I am still getting the same error.
Please, take a look at this
related issue

[testphreak]

Would be great to see this fixed for those working with custom event types.

[gakinson]

I am still experiencing this issue locally using the 1.3.2 version.

[gakinson]

Hello, here are my notes on how I could get it working by duplicating your class and playing around with annotations.

Failed attempts

1. Move the @JsonProperty annotation to the field and removing it from the "is" getter
2. Tried add additional letters tot he prefix

Successful attempts

1. Deleting the "isBase64Encoded()" and putting JsonProperty("isBase64Encoded") to the field isBase64Encoded

• Issue with this is that there is no getter now.

2. Rename the "isBase64Encoded()" function to "getIsBase64Encoded()"

Still trying some others that are backwards compatible, but please address this in some way.

[afayes]

I am still getting the issue.
SAM version is 0.39.0

I am trying to follow this tutorial: https://aws.amazon.com/blogs/opensource/java-apis-aws-lambda/
and get the following error:

Invalid API Gateway Response Keys: {'base64Encoded'} in {'statusCode': 200, 'headers': {'Content-Type': 'application/json'}, 'body': '{"pong":"Hello, World!"}', 'base64Encoded': False} Function returned an invalid response (must include one of: body, headers, multiValueHeaders or statusCode in the response object). Response received: {"statusCode":200,"headers":{"Content-Type":"application/json"},"body":"{\"pong\":\"Hello, World!\"}","base64Encoded":false}

It is Jan 2020, should this not be fixed by now?

[jfuss]

This is not related to SAM CLI and is a combination of AWS Lambda and they dependency you use: aws/serverless-java-container#262 (comment)

SAM CLI only serialize the output of the container into json, therefore if you are running into this issue, something is wrong in your serialization within your code.

Closing as it is not something SAM CLI controls.

@afayes You are getting that error because the keys are wrong in your output (base64Encoded should be isBase64Encoded). See my earlier comment above: #1193 (comment)

[ranjitparva]

For Kotlin: Use @JvmName
@get:JvmName("getIsBase64Encoded") val base64Encoded: Boolean = false

[kastork]

Here's an example trace running local (1.3.2)

› sam local start-api -t template.yaml
Mounting mnE2AB52C4 at http://127.0.0.1:3000/{proxy+} [DELETE, GET, HEAD, OPTIONS, PATCH, POST, PUT]
Mounting None at http://127.0.0.1:3000/ [DELETE, GET, HEAD, OPTIONS, PATCH, POST, PUT]
You can now browse to the above endpoints to invoke your functions. You do not need to restart/reload SAM CLI while working on your functions, changes will be reflected instantly/automatically. You only need to restart SAM CLI if you update your AWS SAM template
2020-09-27 14:26:26  * Running on http://127.0.0.1:3000/ (Press CTRL+C to quit)
Invoking com.example.ProxyHandler (java11)
Decompressing /Users/kirk/s/local/cdk-mn/backend/build/libs/demo-all.jar
Skip pulling image and use local one: amazon/aws-sam-cli-emulation-image-java11:rapid-1.3.2.

Mounting /private/var/folders/66/_95_86950lb9drnn0s2dp5h40000gn/T/tmpyedmand9 as /var/task:ro,delegated inside runtime container
21:26:55.376 [main] INFO  i.m.f.a.p.AbstractLambdaContainerHandler - Starting Lambda Container Handler
21:26:55.571 [main] INFO  i.m.context.env.DefaultEnvironment - Established active environments: [ec2, cloud, function, lambda]
21:26:59.045 [main] DEBUG com.example.ProxyHandler - My handler bean is here.
START RequestId: de41ebe6-e81a-1f1d-d7c8-e9bedd8090ac Version: $LATEST
21:26:59.138 [main] DEBUG com.example.ProxyHandler - POST
21:26:59.138 [main] DEBUG com.example.ProxyHandler - /book
21:26:59.138 [main] DEBUG com.example.ProxyHandler - {"name": "foo"}
21:26:59.138 [main] DEBUG com.example.ProxyHandler - lambdainternal.api.LambdaContext@73a1e9a9
21:27:01.485 [main] DEBUG com.example.BookController - Request body is Book(name=foo)
21:27:01.596 [main] DEBUG com.example.ProxyHandler - Actual AwsProxyResponse is of type com.amazonaws.serverless.proxy.model.AwsProxyResponse
21:27:01.630 [main] DEBUG com.example.ProxyHandler - {"statusCode":200,"multiValueHeaders":{"Content-Type":["application/json"]},"body":"{\"name\":\"foo\",\"isbn\":\"f3634054-18be-4493-874a-ce0693724031\"}","isBase64Encoded":false}
END RequestId: de41ebe6-e81a-1f1d-d7c8-e9bedd8090ac
REPORT RequestId: de41ebe6-e81a-1f1d-d7c8-e9bedd8090ac  Init Duration: 4678.88 ms       Duration: 2510.43 ms    Billed Duration: 2600 ms        Memory Size: 512 MB        Max Memory Used: 110 MB
Invalid lambda response received: Invalid API Gateway Response Keys: {'base64Encoded'} in {'statusCode': 200, 'multiValueHeaders': {'Content-Type': ['application/json']}, 'body': '{"name":"foo","isbn":"f3634054-18be-4493-874a-ce0693724031"}', 'base64Encoded': False}
2020-09-27 14:27:02 127.0.0.1 - - [27/Sep/2020 14:27:02] "POST /book HTTP/1.1" 502 -

Please notice the application is returning a correct proxy object (the json you see is from me serializing the AwsProxyResponse to JSON using jackson.

{"statusCode":200,"multiValueHeaders":{"Content-Type":["application/json"]},"body":"{\"name\":\"foo\",\"isbn\":\"f3634054-18be-4493-874a-ce0693724031\"}","isBase64Encoded":false}

But gateway is seeing something else

{'statusCode': 200, 'multiValueHeaders': {'Content-Type': ['application/json']}, 'body': '{"name":"foo","isbn":"f3634054-18be-4493-874a-ce0693724031"}', 'base64Encoded': False}

This leads to an HTTP response of BAD GATEWAY.

HTTP/1.0 502 BAD GATEWAY
Content-Length: 36
Content-Type: application/json
Date: Sun, 27 Sep 2020 21:27:02 GMT
Server: Werkzeug/1.0.1 Python/3.7.9

{
    "message": "Internal server error"
}

Now this exact same code is deployed to api-gateway, and the response is correct

› http POST https://uu5dplmknj.execute-api.us-west-2.amazonaws.com/prod/book name=foo
HTTP/1.1 200 OK
Connection: keep-alive
Content-Length: 60
Content-Type: application/json
Date: Sun, 27 Sep 2020 21:34:41 GMT
Via: 1.1 a22f4ead7e43f2bbb5f34d2cc611ecb4.cloudfront.net (CloudFront)
X-Amz-Cf-Id: XkUxvYLw9Lfez13Bjh3x_R9wOw3mqJbs_v-z_y2SWFZMzk8Aww44cw==
X-Amz-Cf-Pop: SFO5-C3
X-Amzn-Trace-Id: Root=1-5f710569-c68d1b5822258f0873d65848;Sampled=0
X-Cache: Miss from cloudfront
x-amz-apigw-id: Ti3IbF29vHcFg0Q=
x-amzn-RequestId: 7510b491-4dcb-4699-8af9-694745b49129

{
    "isbn": "94f82841-f63e-4e18-9f29-2b459e0369f1",
    "name": "foo"
}

Cloud watch logs don't show what lambda sends to gateway, but here's the comparable log output that goes with the above sam local log.

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
|   timestamp   |                                                                                                                                          message                                                                                                                                           |
|---------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| 1601240544169 | START RequestId: d6139ad4-8c11-43ed-b677-b7265ef847ed Version: $LATEST                                                                                                                                                                                                                     |
| 1601240544307 | �INFO �i.m.f.a.p.AbstractLambdaContainerHandler�- Starting Lambda Container Handler                                                                                                                                   |
| 1601240546414 | �INFO �i.m.context.env.DefaultEnvironment�- Established active environments: [ec2, cloud, function, lambda]                                                                                                           |
| 1601240546414 | �DEBUG�com.example.ProxyHandler�- My handler bean is here.                                                                                                                                                            |
| 1601240546503 | �DEBUG�com.example.ProxyHandler�- POST                                                                                                                                                                                |
| 1601240546503 | �DEBUG�com.example.ProxyHandler�- /book                                                                                                                                                                               |
| 1601240546503 | �DEBUG�com.example.ProxyHandler�- {"name": "foo"}                                                                                                                                                                     |
| 1601240546503 | �DEBUG�com.example.ProxyHandler - lambdainternal.api.LambdaContext@4c402120                                                                                                                                           |
| 1601240551265 | �DEBUG�com.example.BookController�- Request body is Book(name=foo)                                                                                                                                                    |
| 1601240551524 | �DEBUG�com.example.ProxyHandler�- Actual AwsProxyResponse is of type com.amazonaws.serverless.proxy.model.AwsProxyResponse                                                                                            |
| 1601240551623 | �DEBUG�com.example.ProxyHandler�- {"statusCode":200,"multiValueHeaders":{"Content-Type":["application/json"]},"body":"{\"name\":\"foo\",\"isbn\":\"43094531-6441-4f0a-811c-76a4e3dcfd3a\"}","isBase64Encoded":false}  |
| 1601240551643 | END RequestId: d6139ad4-8c11-43ed-b677-b7265ef847ed                                                                                                                                                                                                                                        |
| 1601240551643 | REPORT RequestId: d6139ad4-8c11-43ed-b677-b7265ef847ed Duration: 5148.61 ms Billed Duration: 5200 ms Memory Size: 512 MB Max Memory Used: 179 MB Init Duration: 3092.41 ms                                                                                                                 |
| 1601240727148 | START RequestId: 4bb2d2d9-2fce-4d06-a5a6-5075c6c7b51b Version: $LATEST                                                                                                                                                                                                                     |
| 1601240727153 | DEBUG�com.example.ProxyHandler�- GET                                                                                                                                                                                 |
| 1601240727153 | DEBUG�com.example.ProxyHandler�- /book                                                                                                                                                                               |
| 1601240727153 | DEBUG�com.example.ProxyHandler�- null                                                                                                                                                                                |
| 1601240727153 | DEBUG�com.example.ProxyHandler�- lambdainternal.api.LambdaContext@7aaca91a                                                                                                                                           |
| 1601240727158 | DEBUG�com.example.ProxyHandler�- Actual AwsProxyResponse is of type com.amazonaws.serverless.proxy.model.AwsProxyResponse                                                                                            |
| 1601240727158 | DEBUG�com.example.ProxyHandler�- {"statusCode":405,"multiValueHeaders":{},"isBase64Encoded":false}                                                                                                                   |
| 1601240727164 | END RequestId: 4bb2d2d9-2fce-4d06-a5a6-5075c6c7b51b                                                                                                                                                                                                                                        |
| 1601240727164 | REPORT RequestId: 4bb2d2d9-2fce-4d06-a5a6-5075c6c7b51b Duration: 12.65 ms Billed Duration: 100 ms Memory Size: 512 MB Max Memory Used: 179 MB                                                                                                                                              |
| 1601240994775 | START RequestId: cf2ed0f9-8290-46c2-980d-35ba96638140 Version: $LATEST                                                                                                                                                                                                                     |
| 1601240994780 | DEBUG�com.example.ProxyHandler�- HEAD                                                                                                                                                                                |
| 1601240994780 | DEBUG�com.example.ProxyHandler�- /book                                                                                                                                                                               |
| 1601240994780 | DEBUG�com.example.ProxyHandler�- null                                                                                                                                                                                |
| 1601240994780 | DEBUG�com.example.ProxyHandler�- lambdainternal.api.LambdaContext@20435c40                                                                                                                                           |
| 1601240994781 | DEBUG�com.example.ProxyHandler�- Actual AwsProxyResponse is of type com.amazonaws.serverless.proxy.model.AwsProxyResponse                                                                                            |
| 1601240994781 | DEBUG�com.example.ProxyHandler�- {"statusCode":405,"multiValueHeaders":{},"isBase64Encoded":false}                                                                                                                   |
| 1601240994781 | END RequestId: cf2ed0f9-8290-46c2-980d-35ba96638140                                                                                                                                                                                                                                        |
| 1601240994782 | REPORT RequestId: cf2ed0f9-8290-46c2-980d-35ba96638140 Duration: 3.43 ms Billed Duration: 100 ms Memory Size: 512 MB Max Memory Used: 179 MB                                                                                                                                               |
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

[kastork]

@jfuss

Looking at the object returned by the serverless java container (AwsProxyResponse), the properties are correct.

    @JsonProperty("isBase64Encoded")
    public boolean isBase64Encoded() {
        return isBase64Encoded;
    }

    public void setBase64Encoded(boolean base64Encoded) {
        isBase64Encoded = base64Encoded;
    }

So I think upstream is not responsible for this.

In my comments above I serialized this object with Jackson, and as you can see, the serialization also results in isBase64Encoded.

In your comment above where you say "SAM CLI only serialize the output of the container into json", I believe you are saying that CLI serializes the AwsProxyResponse, if this is true, then it would appear that CLI's serialization of it is the cause of this.

There is a ticket on this problem over at serverless java container... aws/serverless-java-container#262