Best way to pass binary content through ApiGateway unmodified

[0xdevalias]

I searched the existing issues for ContentHandling and most that contain it (as part of their example code) seem to be setting at as part of a fully custom OpenAPI/Swagger spec.

I was wondering if there is a canonical way for setting this on the generated 'implicit ApiGateway', or if I should be resorting to manually creating my entire spec for this kind of behaviour?

[0xdevalias]

Reading the documentation closer it looks like I can set it with BinaryMediaTypes, so I assume that to do this for the implicit API I would set it in the global section

Globals:
  Api:
    BinaryMediaTypes:
      - application~1octet-stream

[0xdevalias]

Ok, so I tried this as above, but it seems that it defaults to base64 encoding the binary, and not sure where/how I would be able to set ContentHandling (Ref 2)

Basically, I want to reverse-proxy a request through apigateway/lambda to another endpoint, and the binary payloads should be passed through unmodified.

I assume this will require some combination of ContentHandling and/or PassthroughBehavior (Ref 2), but I am having trouble figuring out how exactly to map that into SAM.

Ref:

• https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-payload-encodings-workflow.html
• https://docs.aws.amazon.com/apigateway/latest/developerguide/integration-passthrough-behaviors.html

For proxy integrations, API Gateway passes entire the request through to your backend, and you do not have the option to modify the passthrough behaviors.

The actual passthrough behavior of an incoming request is determined by the option you choose for a specified mapping template, during integration request set-up, and the Content Type header that a client set in the incoming request.

[0xdevalias]

Reading a little deeper.. the base64 encoding may actually be occurring at the awslabs/aws-lambda-go-api proxy level:

• https://github.com/awslabs/aws-lambda-go-api-proxy/blob/master/core/response.go#L81-L86
• https://github.com/awslabs/aws-lambda-go-api-proxy/blob/master/gorillamux/adapter.go#L31-L36

The following issue (on a related library) makes me think that setting the IsBase64Encoded (as this code does) would allow the API Gateway to then determine if/how it base64 decodes the response:

• Add support for binary response content aws-lambda-dotnet#74

So this makes me think that there is still something within SAM/CloudFormation that should allow us to tell it to decode the base64 and return binary again.. based on the previous docs, my guess would be setting a ContentHandling of CONVERT_TO_BINARY would resolve it.

Looking at these OpenAPI extensions:

• https://mermade.github.io/shins/semoasa.html#openapi-extensions-documentation
• https://docs.aws.amazon.com/apigateway/latest/developerguide/api-gateway-swagger-extensions-integration.html

It sounds like adding the following to the OpenAPI body would enable this:

x-amazon-apigateway-integration:
  contentHandling: CONVERT_TO_BINARY

Looking at the generated/processed template json (within CloudFormation):

"x-amazon-apigateway-integration" : {
  "uri" : {
    "Fn::Sub" : "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${Proxy.Arn}/invocations"
  },
  "httpMethod" : "POST",
  "type" : "aws_proxy"
}

It looks as though we aren't setting any of those keys at the moment.. but they do match the rest of the extensions detailed above.

So I guess ultimately, my question/request ends up as:

• Is there currently support for defining any of these extra keys as part of the SAM spec?
• If not, can we add functionality that would allow us to define them?

[0xdevalias]

Potentially related issues:

• "sam package" command does not honor Globals aws-sam-cli#653
• Unable to set "contentHandling" #553
• Support API Gateway Logging Metrics, Usage Plans, EndpointConfiguration, and CORS #248

[0xdevalias]

Decided to try implementing this myself using the more manual version of AWS::Serverless::Api. By inspecting the end CloudFormation json, and converting/working backwards into SAM yaml, I came up with this:

  FooTestApi:
    Type: AWS::Serverless::Api
    Properties:
      Name: FooTestApi
      StageName: Prod
      DefinitionBody:
        swagger: '2.0'
        info:
          version: '1.0'
          title: !Ref AWS::StackName
        paths:
          "/{proxy+}":
            x-amazon-apigateway-any-method:
              x-amazon-apigateway-integration:
                type: aws_proxy
                uri: !Sub arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${AgentProxy.Arn}/invocations
                httpMethod: POST
                contentHandling: CONVERT_TO_BINARY
              responses: {}

I also created this permission manually (though I expect if I set the RestApiId property on my function it may have been generated for me):

  AgentProxyCatchAllPermissionProdFooTestApi:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName: !Ref AgentProxy
      SourceArn: !Sub
        - arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${__ApiId__}/${__Stage__}/*/*
        - __Stage__: Prod
          __ApiId__: !Ref FooTestApi
      Principal: apigateway.amazonaws.com
      Action: lambda:invokeFunction

This seemed to result in the expected field being available in the produced CloudFormation json, confirming my theory:

"FooTestApi" : {
"Properties" : {
   "Name" : "FooTestApi",
   "Parameters" : {
      "endpointConfigurationTypes" : "REGIONAL"
   },
   "Body" : {
      "info" : {
         "title" : {
            "Ref" : "AWS::StackName"
         },
         "version" : "1.0"
      },
      "swagger" : "2.0",
      "paths" : {
         "/{proxy+}" : {
            "x-amazon-apigateway-any-method" : {
               "x-amazon-apigateway-integration" : {
                  "type" : "aws_proxy",
                  "httpMethod" : "POST",
                  "uri" : {
                     "Fn::Sub" : "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${AgentProxy.Arn}/invocations"
                  },
                  "contentHandling" : "CONVERT_TO_BINARY"
               },
               "responses" : {}
            }
         }
      }
   },
...snip...

Interestingly, it didn't seem to add my binary media type setting (set in the Globals), even though it showed up as a key in the produced CloudFormation json (aws/aws-sam-cli#653 ?):

"BinaryMediaTypes" : [
   "application~1octet-stream"
]

Next, I tried entirely removing FooTestApi and associated elements from my stack, and readding it (so that it was created 'new', rather than updated). This ended up in the appropriate application/octet-stream being set when I look in the console, and (finally) when I run my endpoint, it actually passes through the binary as expected.

[0xdevalias]

So, to resolve this properly, it seems that:

• Need to solve why updates aren't properly setting the binary types ("sam package" command does not honor Globals aws-sam-cli#653 / CodeUri defined in Globals section is ignored when packaging aws-cli#3376, potentially related to Support API Gateway Logging Metrics, Usage Plans, EndpointConfiguration, and CORS #248)
• Need to implement a 'SAM-native' way of configuring "contentHandling" : "CONVERT_TO_BINARY" as described above for the implicit api (Unable to set "contentHandling" #553)

[brettstack]

Sorry about this missing feature. Closing in favor of #553.

[brettstack]

PR for fixing BinaryMediaTypes getting lost in update #954

[keetonian]

To get around needing contentHandling: "CONVERT_TO_BINARY", you can set

BinaryMediaTypes:
  - "*~1*"

on your API definition. However, I think this will have the side effect of converting all responses to binary (unless CONVERT_TO_TEXT is specified in contentHandling), which won't work for most cases. I posted in #553 a proposal to support contentHandling in Event definitions, will go forward with that proposal.

[thanhlq]

Hi everybody,

So does AWS Api gateway support to receive a binary payload? then in api gateway can help to convert that binary to base64 for me?

Thanks very much for any tips...

[praneetap]

@thanhlq In lambda proxy integration data is always converted to base64 if the content-type is marked as binary on the incoming request. On the response, the JSON proxy response can contain a field that indicates whether or not the payload isBase64Encoded: true. The only thing that affects the handling of data to a lambda proxy are the BinaryMediaTypes and the content of the JSON response.

[brahma712]

I have a lambda function that is integrated to API Gateway with a Lambda Proxy integration. Basically the lambda function is grabbing a zip file from S3 and returning it as payload by auto downloading the file when the endpoint is hit in a browser. The contentType I sent is application/octet-stream. However, when the zip file is downloaded, it turns into a cpgz. I am not sure how to resolve this issue. My API gateway was created via the console for proof of concept and not via the cfn template (though eventually we will deploy it through our cfn template) hence I dont know how to pass in the contentHandling parameter. Do you have any thoughts on how I can solve this issue?

[bohana]

Hi all

The only thing that affects the handling of data to a lambda proxy are the BinaryMediaTypes and the content of the JSON response.

I added the binary media types in my template.yaml:

        Properties:
            StageName: dev
            OpenApiVersion: '3.0.0'
            BinaryMediaTypes:
                    - application~1octet-stream
                    - application~1zip
                    - application~1gzip

And after deployment I do see them in the console. (I used sam 1.20.0)

My application also sets isBase64Encoded to True at the Lambda layer, but the gateway still does not convert the payload from base64 to binary.

This should be happening right? What am I missing?