Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Best way to pass binary content through ApiGateway unmodified #566

Closed
0xdevalias opened this issue Sep 6, 2018 · 13 comments
Closed

Best way to pass binary content through ApiGateway unmodified #566

0xdevalias opened this issue Sep 6, 2018 · 13 comments

Comments

@0xdevalias
Copy link

@0xdevalias 0xdevalias commented Sep 6, 2018

I searched the existing issues for ContentHandling and most that contain it (as part of their example code) seem to be setting at as part of a fully custom OpenAPI/Swagger spec.

I was wondering if there is a canonical way for setting this on the generated 'implicit ApiGateway', or if I should be resorting to manually creating my entire spec for this kind of behaviour?

@0xdevalias
Copy link
Author

@0xdevalias 0xdevalias commented Sep 6, 2018

Reading the documentation closer it looks like I can set it with BinaryMediaTypes, so I assume that to do this for the implicit API I would set it in the global section

Globals:
  Api:
    BinaryMediaTypes:
      - application~1octet-stream
@0xdevalias 0xdevalias closed this Sep 6, 2018
@0xdevalias 0xdevalias changed the title Best way to set PassthroughBehavior for implicit ApiGateway Best way to set ContentHandling for implicit ApiGateway Sep 6, 2018
@0xdevalias 0xdevalias changed the title Best way to set ContentHandling for implicit ApiGateway Best way to pass binary content through ApiGateway unmodified Sep 6, 2018
@0xdevalias
Copy link
Author

@0xdevalias 0xdevalias commented Sep 6, 2018

Ok, so I tried this as above, but it seems that it defaults to base64 encoding the binary, and not sure where/how I would be able to set ContentHandling (Ref 2)

Basically, I want to reverse-proxy a request through apigateway/lambda to another endpoint, and the binary payloads should be passed through unmodified.

I assume this will require some combination of ContentHandling and/or PassthroughBehavior (Ref 2), but I am having trouble figuring out how exactly to map that into SAM.

Ref:

For proxy integrations, API Gateway passes entire the request through to your backend, and you do not have the option to modify the passthrough behaviors.

The actual passthrough behavior of an incoming request is determined by the option you choose for a specified mapping template, during integration request set-up, and the Content Type header that a client set in the incoming request.

@0xdevalias 0xdevalias reopened this Sep 6, 2018
@0xdevalias
Copy link
Author

@0xdevalias 0xdevalias commented Sep 6, 2018

Reading a little deeper.. the base64 encoding may actually be occurring at the awslabs/aws-lambda-go-api proxy level:

The following issue (on a related library) makes me think that setting the IsBase64Encoded (as this code does) would allow the API Gateway to then determine if/how it base64 decodes the response:

So this makes me think that there is still something within SAM/CloudFormation that should allow us to tell it to decode the base64 and return binary again.. based on the previous docs, my guess would be setting a ContentHandling of CONVERT_TO_BINARY would resolve it.

Looking at these OpenAPI extensions:

It sounds like adding the following to the OpenAPI body would enable this:

x-amazon-apigateway-integration:
  contentHandling: CONVERT_TO_BINARY

Looking at the generated/processed template json (within CloudFormation):

"x-amazon-apigateway-integration" : {
  "uri" : {
    "Fn::Sub" : "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${Proxy.Arn}/invocations"
  },
  "httpMethod" : "POST",
  "type" : "aws_proxy"
}

It looks as though we aren't setting any of those keys at the moment.. but they do match the rest of the extensions detailed above.

So I guess ultimately, my question/request ends up as:

  • Is there currently support for defining any of these extra keys as part of the SAM spec?
  • If not, can we add functionality that would allow us to define them?
@0xdevalias
Copy link
Author

@0xdevalias 0xdevalias commented Sep 6, 2018

Potentially related issues:

@0xdevalias
Copy link
Author

@0xdevalias 0xdevalias commented Sep 7, 2018

Decided to try implementing this myself using the more manual version of AWS::Serverless::Api. By inspecting the end CloudFormation json, and converting/working backwards into SAM yaml, I came up with this:

  FooTestApi:
    Type: AWS::Serverless::Api
    Properties:
      Name: FooTestApi
      StageName: Prod
      DefinitionBody:
        swagger: '2.0'
        info:
          version: '1.0'
          title: !Ref AWS::StackName
        paths:
          "/{proxy+}":
            x-amazon-apigateway-any-method:
              x-amazon-apigateway-integration:
                type: aws_proxy
                uri: !Sub arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${AgentProxy.Arn}/invocations
                httpMethod: POST
                contentHandling: CONVERT_TO_BINARY
              responses: {}

I also created this permission manually (though I expect if I set the RestApiId property on my function it may have been generated for me):

  AgentProxyCatchAllPermissionProdFooTestApi:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName: !Ref AgentProxy
      SourceArn: !Sub
        - arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${__ApiId__}/${__Stage__}/*/*
        - __Stage__: Prod
          __ApiId__: !Ref FooTestApi
      Principal: apigateway.amazonaws.com
      Action: lambda:invokeFunction

This seemed to result in the expected field being available in the produced CloudFormation json, confirming my theory:

"FooTestApi" : {
"Properties" : {
   "Name" : "FooTestApi",
   "Parameters" : {
      "endpointConfigurationTypes" : "REGIONAL"
   },
   "Body" : {
      "info" : {
         "title" : {
            "Ref" : "AWS::StackName"
         },
         "version" : "1.0"
      },
      "swagger" : "2.0",
      "paths" : {
         "/{proxy+}" : {
            "x-amazon-apigateway-any-method" : {
               "x-amazon-apigateway-integration" : {
                  "type" : "aws_proxy",
                  "httpMethod" : "POST",
                  "uri" : {
                     "Fn::Sub" : "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${AgentProxy.Arn}/invocations"
                  },
                  "contentHandling" : "CONVERT_TO_BINARY"
               },
               "responses" : {}
            }
         }
      }
   },
...snip...

Interestingly, it didn't seem to add my binary media type setting (set in the Globals), even though it showed up as a key in the produced CloudFormation json (aws/aws-sam-cli#653 ?):

"BinaryMediaTypes" : [
   "application~1octet-stream"
]

image

Next, I tried entirely removing FooTestApi and associated elements from my stack, and readding it (so that it was created 'new', rather than updated). This ended up in the appropriate application/octet-stream being set when I look in the console, and (finally) when I run my endpoint, it actually passes through the binary as expected.

@0xdevalias
Copy link
Author

@0xdevalias 0xdevalias commented Sep 7, 2018

So, to resolve this properly, it seems that:

  • Need to solve why updates aren't properly setting the binary types (aws/aws-sam-cli#653 / aws/aws-cli#3376, potentially related to #248)
  • Need to implement a 'SAM-native' way of configuring "contentHandling" : "CONVERT_TO_BINARY" as described above for the implicit api (#553)
@brettstack
Copy link
Contributor

@brettstack brettstack commented Sep 8, 2018

Sorry about this missing feature. Closing in favor of #553.

@brettstack
Copy link
Contributor

@brettstack brettstack commented Jun 7, 2019

PR for fixing BinaryMediaTypes getting lost in update #954

@keetonian
Copy link
Contributor

@keetonian keetonian commented Jul 23, 2019

To get around needing contentHandling: "CONVERT_TO_BINARY", you can set

BinaryMediaTypes:
  - "*~1*"

on your API definition. However, I think this will have the side effect of converting all responses to binary (unless CONVERT_TO_TEXT is specified in contentHandling), which won't work for most cases. I posted in #553 a proposal to support contentHandling in Event definitions, will go forward with that proposal.

@thanhlq
Copy link

@thanhlq thanhlq commented Oct 1, 2019

Hi everybody,

So does AWS Api gateway support to receive a binary payload? then in api gateway can help to convert that binary to base64 for me?

Thanks very much for any tips...

@praneetap
Copy link
Contributor

@praneetap praneetap commented Oct 1, 2019

@thanhlq In lambda proxy integration data is always converted to base64 if the content-type is marked as binary on the incoming request. On the response, the JSON proxy response can contain a field that indicates whether or not the payload isBase64Encoded: true. The only thing that affects the handling of data to a lambda proxy are the BinaryMediaTypes and the content of the JSON response.

@brahma712
Copy link

@brahma712 brahma712 commented Feb 4, 2021

I have a lambda function that is integrated to API Gateway with a Lambda Proxy integration. Basically the lambda function is grabbing a zip file from S3 and returning it as payload by auto downloading the file when the endpoint is hit in a browser. The contentType I sent is application/octet-stream. However, when the zip file is downloaded, it turns into a cpgz. I am not sure how to resolve this issue. My API gateway was created via the console for proof of concept and not via the cfn template (though eventually we will deploy it through our cfn template) hence I dont know how to pass in the contentHandling parameter. Do you have any thoughts on how I can solve this issue?

@bohana
Copy link

@bohana bohana commented Mar 19, 2021

Hi all

The only thing that affects the handling of data to a lambda proxy are the BinaryMediaTypes and the content of the JSON response.

I added the binary media types in my template.yaml:

        Properties:
            StageName: dev
            OpenApiVersion: '3.0.0'
            BinaryMediaTypes:
                    - application~1octet-stream
                    - application~1zip
                    - application~1gzip

And after deployment I do see them in the console. (I used sam 1.20.0)

My application also sets isBase64Encoded to True at the Lambda layer, but the gateway still does not convert the payload from base64 to binary.

This should be happening right? What am I missing?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
7 participants