New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Best way to pass binary content through ApiGateway unmodified #566

Closed
0xdevalias opened this Issue Sep 6, 2018 · 7 comments

Comments

Projects
None yet
2 participants
@0xdevalias

0xdevalias commented Sep 6, 2018

I searched the existing issues for ContentHandling and most that contain it (as part of their example code) seem to be setting at as part of a fully custom OpenAPI/Swagger spec.

I was wondering if there is a canonical way for setting this on the generated 'implicit ApiGateway', or if I should be resorting to manually creating my entire spec for this kind of behaviour?

@0xdevalias

This comment has been minimized.

0xdevalias commented Sep 6, 2018

Reading the documentation closer it looks like I can set it with BinaryMediaTypes, so I assume that to do this for the implicit API I would set it in the global section

Globals:
  Api:
    BinaryMediaTypes:
      - application~1octet-stream

@0xdevalias 0xdevalias closed this Sep 6, 2018

@0xdevalias 0xdevalias changed the title Best way to set PassthroughBehavior for implicit ApiGateway Best way to set ContentHandling for implicit ApiGateway Sep 6, 2018

@0xdevalias 0xdevalias changed the title Best way to set ContentHandling for implicit ApiGateway Best way to pass binary content through ApiGateway unmodified Sep 6, 2018

@0xdevalias

This comment has been minimized.

0xdevalias commented Sep 6, 2018

Ok, so I tried this as above, but it seems that it defaults to base64 encoding the binary, and not sure where/how I would be able to set ContentHandling (Ref 2)

Basically, I want to reverse-proxy a request through apigateway/lambda to another endpoint, and the binary payloads should be passed through unmodified.

I assume this will require some combination of ContentHandling and/or PassthroughBehavior (Ref 2), but I am having trouble figuring out how exactly to map that into SAM.

Ref:

For proxy integrations, API Gateway passes entire the request through to your backend, and you do not have the option to modify the passthrough behaviors.

The actual passthrough behavior of an incoming request is determined by the option you choose for a specified mapping template, during integration request set-up, and the Content Type header that a client set in the incoming request.

@0xdevalias 0xdevalias reopened this Sep 6, 2018

@0xdevalias

This comment has been minimized.

0xdevalias commented Sep 6, 2018

Reading a little deeper.. the base64 encoding may actually be occurring at the awslabs/aws-lambda-go-api proxy level:

The following issue (on a related library) makes me think that setting the IsBase64Encoded (as this code does) would allow the API Gateway to then determine if/how it base64 decodes the response:

So this makes me think that there is still something within SAM/CloudFormation that should allow us to tell it to decode the base64 and return binary again.. based on the previous docs, my guess would be setting a ContentHandling of CONVERT_TO_BINARY would resolve it.

Looking at these OpenAPI extensions:

It sounds like adding the following to the OpenAPI body would enable this:

x-amazon-apigateway-integration:
  contentHandling: CONVERT_TO_BINARY

Looking at the generated/processed template json (within CloudFormation):

"x-amazon-apigateway-integration" : {
  "uri" : {
    "Fn::Sub" : "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${Proxy.Arn}/invocations"
  },
  "httpMethod" : "POST",
  "type" : "aws_proxy"
}

It looks as though we aren't setting any of those keys at the moment.. but they do match the rest of the extensions detailed above.

So I guess ultimately, my question/request ends up as:

  • Is there currently support for defining any of these extra keys as part of the SAM spec?
  • If not, can we add functionality that would allow us to define them?
@0xdevalias

This comment has been minimized.

0xdevalias commented Sep 6, 2018

Potentially related issues:

@0xdevalias

This comment has been minimized.

0xdevalias commented Sep 7, 2018

Decided to try implementing this myself using the more manual version of AWS::Serverless::Api. By inspecting the end CloudFormation json, and converting/working backwards into SAM yaml, I came up with this:

  FooTestApi:
    Type: AWS::Serverless::Api
    Properties:
      Name: FooTestApi
      StageName: Prod
      DefinitionBody:
        swagger: '2.0'
        info:
          version: '1.0'
          title: !Ref AWS::StackName
        paths:
          "/{proxy+}":
            x-amazon-apigateway-any-method:
              x-amazon-apigateway-integration:
                type: aws_proxy
                uri: !Sub arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${AgentProxy.Arn}/invocations
                httpMethod: POST
                contentHandling: CONVERT_TO_BINARY
              responses: {}

I also created this permission manually (though I expect if I set the RestApiId property on my function it may have been generated for me):

  AgentProxyCatchAllPermissionProdFooTestApi:
    Type: AWS::Lambda::Permission
    Properties:
      FunctionName: !Ref AgentProxy
      SourceArn: !Sub
        - arn:aws:execute-api:${AWS::Region}:${AWS::AccountId}:${__ApiId__}/${__Stage__}/*/*
        - __Stage__: Prod
          __ApiId__: !Ref FooTestApi
      Principal: apigateway.amazonaws.com
      Action: lambda:invokeFunction

This seemed to result in the expected field being available in the produced CloudFormation json, confirming my theory:

"FooTestApi" : {
"Properties" : {
   "Name" : "FooTestApi",
   "Parameters" : {
      "endpointConfigurationTypes" : "REGIONAL"
   },
   "Body" : {
      "info" : {
         "title" : {
            "Ref" : "AWS::StackName"
         },
         "version" : "1.0"
      },
      "swagger" : "2.0",
      "paths" : {
         "/{proxy+}" : {
            "x-amazon-apigateway-any-method" : {
               "x-amazon-apigateway-integration" : {
                  "type" : "aws_proxy",
                  "httpMethod" : "POST",
                  "uri" : {
                     "Fn::Sub" : "arn:aws:apigateway:${AWS::Region}:lambda:path/2015-03-31/functions/${AgentProxy.Arn}/invocations"
                  },
                  "contentHandling" : "CONVERT_TO_BINARY"
               },
               "responses" : {}
            }
         }
      }
   },
...snip...

Interestingly, it didn't seem to add my binary media type setting (set in the Globals), even though it showed up as a key in the produced CloudFormation json (awslabs/aws-sam-cli#653 ?):

"BinaryMediaTypes" : [
   "application~1octet-stream"
]

image

Next, I tried entirely removing FooTestApi and associated elements from my stack, and readding it (so that it was created 'new', rather than updated). This ended up in the appropriate application/octet-stream being set when I look in the console, and (finally) when I run my endpoint, it actually passes through the binary as expected.

@0xdevalias

This comment has been minimized.

0xdevalias commented Sep 7, 2018

So, to resolve this properly, it seems that:

  • Need to solve why updates aren't properly setting the binary types (awslabs/aws-sam-cli#653 / aws/aws-cli#3376, potentially related to #248)
  • Need to implement a 'SAM-native' way of configuring "contentHandling" : "CONVERT_TO_BINARY" as described above for the implicit api (#553)
@brettstack

This comment has been minimized.

Contributor

brettstack commented Sep 8, 2018

Sorry about this missing feature. Closing in favor of #553.

@brettstack brettstack closed this Sep 8, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment