New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support New AWS APIGW Binary Responses #2797

Open
jthomerson opened this Issue Nov 25, 2016 · 47 comments

Comments

Projects
None yet
@jthomerson
Contributor

jthomerson commented Nov 25, 2016

This is a Feature Proposal

Description

Previously, AWS API Gateway did not support binary responses, making it impossible to return images from your serverless API. Now they do (see https://aws.amazon.com/blogs/compute/binary-support-for-api-integrations-with-amazon-api-gateway/). We need to be able to configure HTTP endpoints/events in serverless to use this new functionality.

@pmuens pmuens added the kind/feature label Nov 26, 2016

@vladgolubev

This comment has been minimized.

Show comment
Hide comment
@vladgolubev

vladgolubev Dec 11, 2016

Contributor

Does it mean we will be able to gzip response?

Contributor

vladgolubev commented Dec 11, 2016

Does it mean we will be able to gzip response?

@adambiggs

This comment has been minimized.

Show comment
Hide comment
@adambiggs

adambiggs Dec 17, 2016

Contributor

My team is eagerly anticipating this feature in Serverless. We have an image resizing service that currently needs to proxy responses through a traditional server to return images... 😫

Does anybody know of a workaround we could use until this is added to Serverless?

Contributor

adambiggs commented Dec 17, 2016

My team is eagerly anticipating this feature in Serverless. We have an image resizing service that currently needs to proxy responses through a traditional server to return images... 😫

Does anybody know of a workaround we could use until this is added to Serverless?

@vladgolubev

This comment has been minimized.

Show comment
Hide comment
@vladgolubev

vladgolubev Dec 17, 2016

Contributor
Contributor

vladgolubev commented Dec 17, 2016

@adambiggs

This comment has been minimized.

Show comment
Hide comment
@adambiggs

adambiggs Dec 17, 2016

Contributor

@vladgolubev our image resizing service resizes the images on-demand based on query string params. The lambda first checks if the requested image size already exists on S3. If it does, it returns the existing image, and if not it generates it, stores it on S3 and then returns it. So direct S3 links won't work for us.

Our current workaround is to run a simple Node.js reverse proxy that calls the Lambda image resize service, gets the direct S3 link, and then returns the binary response to the client...

But this is a temporary solution. When Serverless adds binary response support (or if we can find a workaround, like making changes directly in AWS after deploying with Serverless) we can get rid of the reverse proxy without changing any previously generated image URLs.

Contributor

adambiggs commented Dec 17, 2016

@vladgolubev our image resizing service resizes the images on-demand based on query string params. The lambda first checks if the requested image size already exists on S3. If it does, it returns the existing image, and if not it generates it, stores it on S3 and then returns it. So direct S3 links won't work for us.

Our current workaround is to run a simple Node.js reverse proxy that calls the Lambda image resize service, gets the direct S3 link, and then returns the binary response to the client...

But this is a temporary solution. When Serverless adds binary response support (or if we can find a workaround, like making changes directly in AWS after deploying with Serverless) we can get rid of the reverse proxy without changing any previously generated image URLs.

@bbilger

This comment has been minimized.

Show comment
Hide comment
@bbilger

bbilger Dec 21, 2016

Contributor

Note: there's some thread in the forum about this: http://forum.serverless.com/t/returning-binary-data-jpg-from-lambda-via-api-gateway/796
The current issue - as far as I know - is that the binaryMediaTypes cannot be set in CloudFormation.
So your only option right now is to configure them manually (if I remember correctly it survives a re-deployment).
See here: https://github.com/bbilger/jrestless-examples/tree/master/aws/gateway/aws-gateway-binary
or here: https://github.com/krisgholson/serverless-thumbnail
or in the blog post: https://aws.amazon.com/blogs/compute/binary-support-for-api-integrations-with-amazon-api-gateway/

and sure you can also gzip: https://github.com/bustlelabs/gziptest/ (not using serverless)

I would, however, not expect too much from it since
a) API Gateway has a size limit of 10MB (+ I guess you have to substract the base64 overhead from it)
b) only if a proper Accept header (for the response) or Content-Type header (for the request) i.e. one registered as binary media type is set, you'll get binary data, else you'll end up with base64 encoded content.
c) If I remember correctly there are issues with multiple accept headers like "Accept: image/png,image/gif"

Contributor

bbilger commented Dec 21, 2016

Note: there's some thread in the forum about this: http://forum.serverless.com/t/returning-binary-data-jpg-from-lambda-via-api-gateway/796
The current issue - as far as I know - is that the binaryMediaTypes cannot be set in CloudFormation.
So your only option right now is to configure them manually (if I remember correctly it survives a re-deployment).
See here: https://github.com/bbilger/jrestless-examples/tree/master/aws/gateway/aws-gateway-binary
or here: https://github.com/krisgholson/serverless-thumbnail
or in the blog post: https://aws.amazon.com/blogs/compute/binary-support-for-api-integrations-with-amazon-api-gateway/

and sure you can also gzip: https://github.com/bustlelabs/gziptest/ (not using serverless)

I would, however, not expect too much from it since
a) API Gateway has a size limit of 10MB (+ I guess you have to substract the base64 overhead from it)
b) only if a proper Accept header (for the response) or Content-Type header (for the request) i.e. one registered as binary media type is set, you'll get binary data, else you'll end up with base64 encoded content.
c) If I remember correctly there are issues with multiple accept headers like "Accept: image/png,image/gif"

@nikgraf

This comment has been minimized.

Show comment
Hide comment
@pmuens

This comment has been minimized.

Show comment
Hide comment
@pmuens

pmuens Apr 4, 2017

Member

@pmuens seems like somebody got it to work

Wow that's pretty cool! Thanks for sharing! 👍

/cc @brianneisler @eahefnawy

Member

pmuens commented Apr 4, 2017

@pmuens seems like somebody got it to work

Wow that's pretty cool! Thanks for sharing! 👍

/cc @brianneisler @eahefnawy

@brianneisler brianneisler added this to the 1.14 milestone May 8, 2017

@ryanmurakami ryanmurakami referenced this issue May 11, 2017

Closed

Adding Content Handling for API Gateway #3595

2 of 7 tasks complete

@brianneisler brianneisler removed this from the 1.14 milestone May 15, 2017

@brianneisler

This comment has been minimized.

Show comment
Hide comment
@brianneisler

brianneisler May 15, 2017

Member

Removed from the 1.14 milestone since we're still waiting on CloudFormation support.

Member

brianneisler commented May 15, 2017

Removed from the 1.14 milestone since we're still waiting on CloudFormation support.

@schickling

This comment has been minimized.

Show comment
Hide comment
@schickling

schickling May 28, 2017

Contributor

Is there a timeline from AWS for CF support?

Contributor

schickling commented May 28, 2017

Is there a timeline from AWS for CF support?

@vangorra

This comment has been minimized.

Show comment
Hide comment
@vangorra

vangorra May 28, 2017

We ended up defining our function through serverless but defined the API gateway in the resources section with swagger yml. Works great.

We ended up defining our function through serverless but defined the API gateway in the resources section with swagger yml. Works great.

@pmuens

This comment has been minimized.

Show comment
Hide comment
@pmuens

pmuens May 29, 2017

Member

Is there a timeline from AWS for CF support?

@schickling unfortunately not yet.

We ended up defining our function through serverless but defined the API gateway in the resources section with swagger yml. Works great.

That sounds like a good workaround. Thanks for sharing @vangorra 👍


For everyone else who wants to use this now. A new Serverless plugin for this was published recently: https://github.com/ryanmurakami/serverless-apigwy-binary

Member

pmuens commented May 29, 2017

Is there a timeline from AWS for CF support?

@schickling unfortunately not yet.

We ended up defining our function through serverless but defined the API gateway in the resources section with swagger yml. Works great.

That sounds like a good workaround. Thanks for sharing @vangorra 👍


For everyone else who wants to use this now. A new Serverless plugin for this was published recently: https://github.com/ryanmurakami/serverless-apigwy-binary

@rupakg rupakg modified the milestone: 1.15 Jun 1, 2017

@amv

This comment has been minimized.

Show comment
Hide comment
@amv

amv Jul 2, 2017

Any word on the CF support?

I am having some troubles with the mentioned plugin, mainly because it forces me to change the integration from lambda-proxy to lambda. I can't seem to get my Content-Types remaining as they are supposed to, as the plugin seems to have a side effect of clearing the Content-Type header mapping for the default 200 pattern.

I was however able to make my binary endpoint work by using the lambda-proxy, manually adding */* to "Binary Support" through the AWS Api Gateway GUI, and sending my data as base64 like this:

callback(null, {
  statusCode: 200,
  body: filebuffer.toString('base64'),
  isBase64Encoded: true,
  headers: { "Content-Type" : "image/png" }
} );`

Does anyone know if this be the approach for the official serverless support, or will we have to work with lambda integration? Could both options be allowed through configuration?

amv commented Jul 2, 2017

Any word on the CF support?

I am having some troubles with the mentioned plugin, mainly because it forces me to change the integration from lambda-proxy to lambda. I can't seem to get my Content-Types remaining as they are supposed to, as the plugin seems to have a side effect of clearing the Content-Type header mapping for the default 200 pattern.

I was however able to make my binary endpoint work by using the lambda-proxy, manually adding */* to "Binary Support" through the AWS Api Gateway GUI, and sending my data as base64 like this:

callback(null, {
  statusCode: 200,
  body: filebuffer.toString('base64'),
  isBase64Encoded: true,
  headers: { "Content-Type" : "image/png" }
} );`

Does anyone know if this be the approach for the official serverless support, or will we have to work with lambda integration? Could both options be allowed through configuration?

@ajkerr

This comment has been minimized.

Show comment
Hide comment
@amv

This comment has been minimized.

Show comment
Hide comment
@amv

amv Jul 6, 2017

This is great news!

I have been using manual instructions on how to set the api binary types on my serverless phantomJS screenshot project README file, but I would be more than happy to remove the guides once we get a version of serverless out which properly supports this :)

Ping @brianneisler for the good news and expedited future roadmap inclusion 👍

amv commented Jul 6, 2017

This is great news!

I have been using manual instructions on how to set the api binary types on my serverless phantomJS screenshot project README file, but I would be more than happy to remove the guides once we get a version of serverless out which properly supports this :)

Ping @brianneisler for the good news and expedited future roadmap inclusion 👍

@pmuens

This comment has been minimized.

Show comment
Hide comment
@pmuens

pmuens Jul 6, 2017

Member

Nice! Thanks for posting the update @ajkerr 👍

@amv thanks for your comment! This has been on the roadmap for a long time and has a pretty high priority. Unfortunately it was blocked by the lack of CloudFormation support (until now 🙏).

Anyone here who would like to jump into an implementation / WIP PR? We're more than happy to help out when problems come up!

This way we can get it ready for v1.18 or v1.19.

Member

pmuens commented Jul 6, 2017

Nice! Thanks for posting the update @ajkerr 👍

@amv thanks for your comment! This has been on the roadmap for a long time and has a pretty high priority. Unfortunately it was blocked by the lack of CloudFormation support (until now 🙏).

Anyone here who would like to jump into an implementation / WIP PR? We're more than happy to help out when problems come up!

This way we can get it ready for v1.18 or v1.19.

@rcoh

This comment has been minimized.

Show comment
Hide comment
@rcoh

rcoh Jul 7, 2017

@pmuens I'd like to give this a go tomorrow! First contribution to severless so I'd appreciate being pointed in the right direction.

rcoh commented Jul 7, 2017

@pmuens I'd like to give this a go tomorrow! First contribution to severless so I'd appreciate being pointed in the right direction.

@pmuens

This comment has been minimized.

Show comment
Hide comment
@pmuens

pmuens Jul 7, 2017

Member

@pmuens I'd like to give this a go tomorrow! First contribution to severless so I'd appreciate being pointed in the right direction.

Awesome @rcoh! That's super nice 🎉 🙌 👍

Let me see...

So it looks like support for the BinaryMediaTypes config parameter needs to be added to the AWS::ApiGateway::RestApi resource (according to this documentation).

You can find the code for the compilation of the RestApi here. The rest of the plugin which compiles all of the API Gateway resources can be found here.

Other than that it looks like this documentation describes how everything should work together.

The only thing I haven't found yet is the config for ContentHandling in the CloudFormation resource definition for an IntegrationResponse.

🤔 not sure if it's not added yet or if it's undocumented.

Other than that there's also this forum link which might shed some lights into the way this works in general.

Thanks again for looking into this @rcoh 👍 Let us know if you need anything else! Happy to help you get this into master!

Member

pmuens commented Jul 7, 2017

@pmuens I'd like to give this a go tomorrow! First contribution to severless so I'd appreciate being pointed in the right direction.

Awesome @rcoh! That's super nice 🎉 🙌 👍

Let me see...

So it looks like support for the BinaryMediaTypes config parameter needs to be added to the AWS::ApiGateway::RestApi resource (according to this documentation).

You can find the code for the compilation of the RestApi here. The rest of the plugin which compiles all of the API Gateway resources can be found here.

Other than that it looks like this documentation describes how everything should work together.

The only thing I haven't found yet is the config for ContentHandling in the CloudFormation resource definition for an IntegrationResponse.

🤔 not sure if it's not added yet or if it's undocumented.

Other than that there's also this forum link which might shed some lights into the way this works in general.

Thanks again for looking into this @rcoh 👍 Let us know if you need anything else! Happy to help you get this into master!

@rcoh

This comment has been minimized.

Show comment
Hide comment
@rcoh

rcoh Jul 17, 2017

Started to work on this. Was hoping to be able to test my changes to serverless by using npm link but when I do that, serverless stops working:

The AWS security token included in the request is invalid / expired.

Unrelated to that, what were you think of for the API from the serverless/end user side of things @pmuens ?

rcoh commented Jul 17, 2017

Started to work on this. Was hoping to be able to test my changes to serverless by using npm link but when I do that, serverless stops working:

The AWS security token included in the request is invalid / expired.

Unrelated to that, what were you think of for the API from the serverless/end user side of things @pmuens ?

@pmuens

This comment has been minimized.

Show comment
Hide comment
@pmuens

pmuens Jul 18, 2017

Member

Started to work on this.

Great @rcoh 🎉 👍 Really looking forward to this feature!

Was hoping to be able to test my changes to serverless by using npm link but when I do that, serverless stops working

😬 That was a bug we recently introduced in master, but it was reverted a few days ago.

Which version of Serverless are you using @rcoh? Can you pull the most recent master? This should fix the issue.

Let us know if you need help with this or anything else!

Member

pmuens commented Jul 18, 2017

Started to work on this.

Great @rcoh 🎉 👍 Really looking forward to this feature!

Was hoping to be able to test my changes to serverless by using npm link but when I do that, serverless stops working

😬 That was a bug we recently introduced in master, but it was reverted a few days ago.

Which version of Serverless are you using @rcoh? Can you pull the most recent master? This should fix the issue.

Let us know if you need help with this or anything else!

@rcoh

This comment has been minimized.

Show comment
Hide comment
@rcoh

rcoh Jul 18, 2017

rcoh commented Jul 18, 2017

@brianneisler

This comment has been minimized.

Show comment
Hide comment
@brianneisler

brianneisler Jul 18, 2017

Member

@rcoh could you elaborate on the idea of total magic? I'm not sure i follow how we could get this to work without requiring it be declared as a property in the serverless.yml

Member

brianneisler commented Jul 18, 2017

@rcoh could you elaborate on the idea of total magic? I'm not sure i follow how we could get this to work without requiring it be declared as a property in the serverless.yml

@amv

This comment has been minimized.

Show comment
Hide comment
@amv

amv Jul 18, 2017

I think what @rcoh is suggesting as the "magic" version would be that the basic deployment process would simply add a Binary Support for */* on all Api Gateways.

I have no idea what the thinking behind allowing binary support for only some content types is, as I believe the magic keyword for converting Base64 to binary is the isBase64Encoded: true attribute on the Lambda-Proxy content payload.

Maybe the situation is different without Lambda-Proxy, and there one would want to not add some content types as "Binary Supported"?

There probably is some reason why the feature exists, and some old code might start to behave differently if */* Binary Support just appeared to all Api Gateways deployed in the future, so I think there is at least mandatory case for allowing the '/' Binary Support to be not set.

The safest way (from backwards compatibility point of view) would be to make this an optional addition from serverless.yml, but a more user friendly option might be to add the */* Binary support by default, and allow changing or omitting it using a directive in serverless.yml.

amv commented Jul 18, 2017

I think what @rcoh is suggesting as the "magic" version would be that the basic deployment process would simply add a Binary Support for */* on all Api Gateways.

I have no idea what the thinking behind allowing binary support for only some content types is, as I believe the magic keyword for converting Base64 to binary is the isBase64Encoded: true attribute on the Lambda-Proxy content payload.

Maybe the situation is different without Lambda-Proxy, and there one would want to not add some content types as "Binary Supported"?

There probably is some reason why the feature exists, and some old code might start to behave differently if */* Binary Support just appeared to all Api Gateways deployed in the future, so I think there is at least mandatory case for allowing the '/' Binary Support to be not set.

The safest way (from backwards compatibility point of view) would be to make this an optional addition from serverless.yml, but a more user friendly option might be to add the */* Binary support by default, and allow changing or omitting it using a directive in serverless.yml.

@bbilger

This comment has been minimized.

Show comment
Hide comment
@bbilger

bbilger Jul 18, 2017

Contributor

@amv
the strongest argument against this - regardless if it's default or not and unless something has changed here recently - is that if you set */* as binary media type, you'll need to base64 encode all response-bodies (if not you'll see a 500) and - if I remember correctly - base64-decode all request-bodies. So unless you use some framework or common code that handles this, it'll get really annoying.
(Amazon implemented this in a really horrible way)

Having said this: it might make sense to have some opt-in shortcut for well-known binary media/mime types: image/png, ...

Contributor

bbilger commented Jul 18, 2017

@amv
the strongest argument against this - regardless if it's default or not and unless something has changed here recently - is that if you set */* as binary media type, you'll need to base64 encode all response-bodies (if not you'll see a 500) and - if I remember correctly - base64-decode all request-bodies. So unless you use some framework or common code that handles this, it'll get really annoying.
(Amazon implemented this in a really horrible way)

Having said this: it might make sense to have some opt-in shortcut for well-known binary media/mime types: image/png, ...

@amv

This comment has been minimized.

Show comment
Hide comment
@amv

amv Jul 18, 2017

@bbilger with https://github.com/amv/serverless-screenshot-get/blob/master/handler.js I have tested that even if I have "Binary Support" enabled for */*, I can return both text as it is like this:

callback(null, { statusCode: 500, body: 'Plain Text Error', headers: { "Content-Type" : "text/plain" } } );

.. and binary like this:

callback(null, { statusCode: 200, body: buffer.toString('base64'), isBase64Encoded: true, headers: { "Content-Type" : "image/png" } } );

.. but this is while using the Lambda-Proxy integration. I don't know how this works with the plain Lambda integration.

amv commented Jul 18, 2017

@bbilger with https://github.com/amv/serverless-screenshot-get/blob/master/handler.js I have tested that even if I have "Binary Support" enabled for */*, I can return both text as it is like this:

callback(null, { statusCode: 500, body: 'Plain Text Error', headers: { "Content-Type" : "text/plain" } } );

.. and binary like this:

callback(null, { statusCode: 200, body: buffer.toString('base64'), isBase64Encoded: true, headers: { "Content-Type" : "image/png" } } );

.. but this is while using the Lambda-Proxy integration. I don't know how this works with the plain Lambda integration.

@bbilger

This comment has been minimized.

Show comment
Hide comment
@bbilger

bbilger Jul 18, 2017

Contributor

@amv
(only talking about lambda-proxy integration)

you are totally right on GET or rather response bodies - isBase64Encoded for sure has some purpose - sorry, totally forgot about that!!!
(you'll only see the 500 when isBase64Encoded=true and the body is not base64-encoded)

What is still true, however, is that one needs to decode the request body (POST, PUT, PATCH) in that case

exports.handler = (event, context, callback) => {
    callback(null, { statusCode: 200, body: event.body, headers: { 'Content-Type' : 'text/plain' } } );
};
curl -H 'Content-Type: text/plain' -H 'Accept: text/plain' --data 'test'  https://APIGWID.REGION.amazonaws.com/STAGE/PATH
# will return 'dGVzdA==' instead of 'test' if you register '*/*' as binary media type
Contributor

bbilger commented Jul 18, 2017

@amv
(only talking about lambda-proxy integration)

you are totally right on GET or rather response bodies - isBase64Encoded for sure has some purpose - sorry, totally forgot about that!!!
(you'll only see the 500 when isBase64Encoded=true and the body is not base64-encoded)

What is still true, however, is that one needs to decode the request body (POST, PUT, PATCH) in that case

exports.handler = (event, context, callback) => {
    callback(null, { statusCode: 200, body: event.body, headers: { 'Content-Type' : 'text/plain' } } );
};
curl -H 'Content-Type: text/plain' -H 'Accept: text/plain' --data 'test'  https://APIGWID.REGION.amazonaws.com/STAGE/PATH
# will return 'dGVzdA==' instead of 'test' if you register '*/*' as binary media type
@rcoh

This comment has been minimized.

Show comment
Hide comment
@rcoh

rcoh Jul 19, 2017

My inclination would be to opt for "least magic" and mimic the (horrible AWS API at first). Moving forward maybe we can add some helpful shortcuts to make things work more smoothly.

Seems like it's easier to go from no magic to magic then the other way around.

rcoh commented Jul 19, 2017

My inclination would be to opt for "least magic" and mimic the (horrible AWS API at first). Moving forward maybe we can add some helpful shortcuts to make things work more smoothly.

Seems like it's easier to go from no magic to magic then the other way around.

@amv

This comment has been minimized.

Show comment
Hide comment
@amv

amv Jul 19, 2017

Oh.. I did not know setting Binary Support affected the incoming payload too! Thanks @bbilger!

Given this (and other possible stuff Binary Support does that I don't know of :P ), just mimicing the AWS API sounds like the best option for me too, with no Binary Support enabled at all by default.

So just provide a list of content types in the yaml, which will be added to the list of registered BinarySupport content types in the Api Gateway?

amv commented Jul 19, 2017

Oh.. I did not know setting Binary Support affected the incoming payload too! Thanks @bbilger!

Given this (and other possible stuff Binary Support does that I don't know of :P ), just mimicing the AWS API sounds like the best option for me too, with no Binary Support enabled at all by default.

So just provide a list of content types in the yaml, which will be added to the list of registered BinarySupport content types in the Api Gateway?

@pmuens

This comment has been minimized.

Show comment
Hide comment
@pmuens

pmuens Jul 19, 2017

Member

Thanks for the nice discussion about this @rcoh @amv @bbilger 👍

I agree that we should maybe start with an opt-in functionality where the user specifies a list of content types. This way we can still add the magic later on if users complain that it's cumbersome.

I personally like this approach more since it's explicit and everyone who looks at the serverless.yml file knows what's going on.

Open to other solutions though!

Member

pmuens commented Jul 19, 2017

Thanks for the nice discussion about this @rcoh @amv @bbilger 👍

I agree that we should maybe start with an opt-in functionality where the user specifies a list of content types. This way we can still add the magic later on if users complain that it's cumbersome.

I personally like this approach more since it's explicit and everyone who looks at the serverless.yml file knows what's going on.

Open to other solutions though!

@talawahtech

This comment has been minimized.

Show comment
Hide comment
@talawahtech

talawahtech Aug 2, 2017

I tried to work around this manually by overriding ApiGatewayRestApi in the resources section of severless.yml and specifying the BinaryMediaTypes there as per the documentation. See below:

    ApiGatewayRestApi: 
      Type: "AWS::ApiGateway::RestApi"
      Properties:
        Name: dev-my-api
        BinaryMediaTypes:
          - "*/*"

Unfortunately I am seeing some unusual behavior. If I use the code above for a brand new deployment, everything works fine, however if I subsequently try to update an existing deployment with another entry in the BinaryMediaTypes list I get the following error:

An error occurred while provisioning your stack: ApiGatewayRestApi - Invalid patch path /binaryMediaTypes/image/gif

Doing some googling led to figure out that it works if I encode the forward slash as "~1" like this image~1gif. The problem is, that solution only works updates, not when you are doing a brand new deployment!

This seems like it is most likely a Cloudformation bug, but I just wanted to check in here to see if anyone else was seeing the same issue or if maybe serverless is doing it's own encoding/decoding of the slash character before uploading the template.

I tried to work around this manually by overriding ApiGatewayRestApi in the resources section of severless.yml and specifying the BinaryMediaTypes there as per the documentation. See below:

    ApiGatewayRestApi: 
      Type: "AWS::ApiGateway::RestApi"
      Properties:
        Name: dev-my-api
        BinaryMediaTypes:
          - "*/*"

Unfortunately I am seeing some unusual behavior. If I use the code above for a brand new deployment, everything works fine, however if I subsequently try to update an existing deployment with another entry in the BinaryMediaTypes list I get the following error:

An error occurred while provisioning your stack: ApiGatewayRestApi - Invalid patch path /binaryMediaTypes/image/gif

Doing some googling led to figure out that it works if I encode the forward slash as "~1" like this image~1gif. The problem is, that solution only works updates, not when you are doing a brand new deployment!

This seems like it is most likely a Cloudformation bug, but I just wanted to check in here to see if anyone else was seeing the same issue or if maybe serverless is doing it's own encoding/decoding of the slash character before uploading the template.

@marcusmolchany marcusmolchany referenced this issue Aug 8, 2017

Open

Version 1.0.0 Todos #2

16 of 22 tasks complete
@stormit-vn

This comment has been minimized.

Show comment
Hide comment
@stormit-vn

stormit-vn Sep 23, 2017

Is there anyone has a complete solution this make this work? I think this is not a part of serverless framework as I have manually created an APIG using AWS Console, just created a resource and enabled CORS, then deploy. If we don't add any binary meta types to APIG, then OPTIONS method will work fine, but if we added some values i.e. / or application/json, the 500 error occurs. I tried to find out what is the root cause leads to 500 errors, but I could not find any log on CloudWatch. Anyone found something helpful?

The reason why we need to to enable binary is we want to compress the response body (i.e. gzip or deflate), if we don't enable binary, then APIG cannot respond data which is binary to client

Is there anyone has a complete solution this make this work? I think this is not a part of serverless framework as I have manually created an APIG using AWS Console, just created a resource and enabled CORS, then deploy. If we don't add any binary meta types to APIG, then OPTIONS method will work fine, but if we added some values i.e. / or application/json, the 500 error occurs. I tried to find out what is the root cause leads to 500 errors, but I could not find any log on CloudWatch. Anyone found something helpful?

The reason why we need to to enable binary is we want to compress the response body (i.e. gzip or deflate), if we don't enable binary, then APIG cannot respond data which is binary to client

@vladgolubev

This comment has been minimized.

Show comment
Hide comment
@vladgolubev

vladgolubev Sep 23, 2017

Contributor

I think this was already solved as a plugin by @maciejtreder https://github.com/maciejtreder/serverless-apigw-binary

So it's up to maintainers to decide whether this should be a part of the framework core or not

Contributor

vladgolubev commented Sep 23, 2017

I think this was already solved as a plugin by @maciejtreder https://github.com/maciejtreder/serverless-apigw-binary

So it's up to maintainers to decide whether this should be a part of the framework core or not

@darrenhgc

This comment has been minimized.

Show comment
Hide comment
@darrenhgc

darrenhgc Sep 24, 2017

I can confirm what @talawahtech is seeing. Our updates only work when the forward slash is encoded as ~1.

    ApiGatewayRestApi:
      Type: AWS::ApiGateway::RestApi
      Properties:
        Name: $<self:custom.apiGateway>
        BinaryMediaTypes:
          - "application~1octet-stream"

To add: I'm pretty sure that this is a problem with API Gateway or CloudFormation.

darrenhgc commented Sep 24, 2017

I can confirm what @talawahtech is seeing. Our updates only work when the forward slash is encoded as ~1.

    ApiGatewayRestApi:
      Type: AWS::ApiGateway::RestApi
      Properties:
        Name: $<self:custom.apiGateway>
        BinaryMediaTypes:
          - "application~1octet-stream"

To add: I'm pretty sure that this is a problem with API Gateway or CloudFormation.

@stormit-vn

This comment has been minimized.

Show comment
Hide comment
@stormit-vn

stormit-vn Sep 24, 2017

@vladgolubev This is not issue to enable or add binary support to APIG, it is the issue that when we enabled binary support for APIG, then the OPTIONS which is use MOCK integration is failed and return 500 error.

Here is the log from CloudWatch
(9089cdf6-a108-11e7-a235-79c9ada9b2d3) Method request body before transformations: [Binary Data]
(9089cdf6-a108-11e7-a235-79c9ada9b2d3) Execution failed due to configuration error: Unable to transform request
(9089cdf6-a108-11e7-a235-79c9ada9b2d3) Method completed with status: 500

@vladgolubev This is not issue to enable or add binary support to APIG, it is the issue that when we enabled binary support for APIG, then the OPTIONS which is use MOCK integration is failed and return 500 error.

Here is the log from CloudWatch
(9089cdf6-a108-11e7-a235-79c9ada9b2d3) Method request body before transformations: [Binary Data]
(9089cdf6-a108-11e7-a235-79c9ada9b2d3) Execution failed due to configuration error: Unable to transform request
(9089cdf6-a108-11e7-a235-79c9ada9b2d3) Method completed with status: 500

@CharithW

This comment has been minimized.

Show comment
Hide comment
@CharithW

CharithW Nov 1, 2017

@stormit-vn I am facing the same scenario. Were you able to tackle a fix for this ?

CharithW commented Nov 1, 2017

@stormit-vn I am facing the same scenario. Were you able to tackle a fix for this ?

@stormit-vn

This comment has been minimized.

Show comment
Hide comment
@stormit-vn

stormit-vn Nov 1, 2017

@CharithW Char unfortunately this is an AWS issue, there is no way to fix it except AWS provide a fix. We have reported this issue to our AWS consultant but didn't get any feedback

@CharithW Char unfortunately this is an AWS issue, there is no way to fix it except AWS provide a fix. We have reported this issue to our AWS consultant but didn't get any feedback

ks888 added a commit to ks888/LambStatus that referenced this issue Jan 20, 2018

Use data url to represent image logo
API Gateway support binary data, but it doesn't work well with CloudFormation. See serverless/serverless#2797
@mvayngrib

This comment has been minimized.

Show comment
Hide comment
@mvayngrib

mvayngrib Feb 21, 2018

Contributor

in case people are still having trouble with this, i have binary responses working, without the need for additional plugins. I think AWS fixed some of the issues people mentioned. Here's what you'll need:

  1. add this line to the built-in aws cors plugin to fix pre-flight support for binary responses (ContentHandling: 'CONVERT_TO_TEXT'): mvayngrib@e796fb5

  2. as mentioned above, enable BinaryMediaTypes via CloudFormation

ApiGatewayRestApi:
  Type: AWS::ApiGateway::RestApi
  Properties:
    Name: <YourCustomName>
    BinaryMediaTypes:
      - "*/*" # or whichever ones you need
  1. If you're using something like serverless-http, keep in mind that APIGateway may ungzip the request body, without removing the Content-Encoding header, which can confuse your compression middleware. I'm using serverless-http + koa, and have this block in my code:
  const headers = caseless(request.headers) // 'caseless' npm module
  if (!this.isUsingServerlessOffline && headers.get('content-encoding') === 'gzip') {
    this.logger.info('stripping content-encoding header as APIGateway already gunzipped')
    headers.set('content-encoding', 'identity')
    event.headers = request.headers
  }

Edit: as @talawahtech said above, this will only work for new deployments. The path patch error is still there

Contributor

mvayngrib commented Feb 21, 2018

in case people are still having trouble with this, i have binary responses working, without the need for additional plugins. I think AWS fixed some of the issues people mentioned. Here's what you'll need:

  1. add this line to the built-in aws cors plugin to fix pre-flight support for binary responses (ContentHandling: 'CONVERT_TO_TEXT'): mvayngrib@e796fb5

  2. as mentioned above, enable BinaryMediaTypes via CloudFormation

ApiGatewayRestApi:
  Type: AWS::ApiGateway::RestApi
  Properties:
    Name: <YourCustomName>
    BinaryMediaTypes:
      - "*/*" # or whichever ones you need
  1. If you're using something like serverless-http, keep in mind that APIGateway may ungzip the request body, without removing the Content-Encoding header, which can confuse your compression middleware. I'm using serverless-http + koa, and have this block in my code:
  const headers = caseless(request.headers) // 'caseless' npm module
  if (!this.isUsingServerlessOffline && headers.get('content-encoding') === 'gzip') {
    this.logger.info('stripping content-encoding header as APIGateway already gunzipped')
    headers.set('content-encoding', 'identity')
    event.headers = request.headers
  }

Edit: as @talawahtech said above, this will only work for new deployments. The path patch error is still there

@Schlesiger

This comment has been minimized.

Show comment
Hide comment
@Schlesiger

Schlesiger Mar 5, 2018

@mvayngrib Is there a reason not to use serverless-apigw-binary plugin? I followed this example and finally have woff, jpeg, etc. working on my single-page application. The plugin also works on new instances and redeploys.

Additionally, I was going to use the express compression middleware but learned API Gateway supports compression. I tried the serverless-content-encoding plugin in tandem with serverless-apigw-binary and everything is working well.

I am trying not to be too reliant on AWS features and would prefer express-only solutions, but Google Cloud Functions and others still have a lot of catch up to do. In the meantime I don't feel like this is too much dependency on AWS features and it should be pretty easy to migrate/refactor down the road.

@mvayngrib Is there a reason not to use serverless-apigw-binary plugin? I followed this example and finally have woff, jpeg, etc. working on my single-page application. The plugin also works on new instances and redeploys.

Additionally, I was going to use the express compression middleware but learned API Gateway supports compression. I tried the serverless-content-encoding plugin in tandem with serverless-apigw-binary and everything is working well.

I am trying not to be too reliant on AWS features and would prefer express-only solutions, but Google Cloud Functions and others still have a lot of catch up to do. In the meantime I don't feel like this is too much dependency on AWS features and it should be pretty easy to migrate/refactor down the road.

@mvayngrib

This comment has been minimized.

Show comment
Hide comment
@mvayngrib

mvayngrib Mar 5, 2018

Contributor

@Schlesiger the plugin's great, i used it for a while. However, for my project, I need people to be able to launch from my cloudformation templates as is (without any post-processing by serverless plugins)

Contributor

mvayngrib commented Mar 5, 2018

@Schlesiger the plugin's great, i used it for a while. However, for my project, I need people to be able to launch from my cloudformation templates as is (without any post-processing by serverless plugins)

@danielrbradley

This comment has been minimized.

Show comment
Hide comment
@danielrbradley

danielrbradley Mar 14, 2018

Just wondering if this release of the AWS serverless application model has any impact on how binary media types might be supported?

https://github.com/awslabs/serverless-application-model/releases/tag/1.4.0

Binary Media Types

Send images, pdf, or any binary data through your APIs by adding the following configuration:

BinaryMediaTypes:
  # API Gateway will convert ~1 to / 
  - image~1png
  - image~1gif

Just wondering if this release of the AWS serverless application model has any impact on how binary media types might be supported?

https://github.com/awslabs/serverless-application-model/releases/tag/1.4.0

Binary Media Types

Send images, pdf, or any binary data through your APIs by adding the following configuration:

BinaryMediaTypes:
  # API Gateway will convert ~1 to / 
  - image~1png
  - image~1gif
@eraserfusion

This comment has been minimized.

Show comment
Hide comment
@eraserfusion

eraserfusion Apr 9, 2018

@mvayngrib, Can you make a pull request for your change here: mvayngrib/serverless@e796fb5. This fixes an issue that I have had for several days and I think it would benefit the serverless community.

@mvayngrib, Can you make a pull request for your change here: mvayngrib/serverless@e796fb5. This fixes an issue that I have had for several days and I think it would benefit the serverless community.

@mvayngrib

This comment has been minimized.

Show comment
Hide comment
@mvayngrib

mvayngrib Apr 9, 2018

Contributor

@eraserfusion np, see #4895, though without tests i doubt it'll be merged any time soon :)

Contributor

mvayngrib commented Apr 9, 2018

@eraserfusion np, see #4895, though without tests i doubt it'll be merged any time soon :)

@HyperBrain HyperBrain closed this in #4895 May 12, 2018

@ronkorving

This comment has been minimized.

Show comment
Hide comment
@ronkorving

ronkorving May 18, 2018

Contributor

When #4895 got merged, it closed this issue. But as that PR describes, it only solves this issue partially. Can it be reopened until binary support is a full first-class citizen? Or did I miss something and is it actually supported out of the box now?

cc @HyperBrain

Contributor

ronkorving commented May 18, 2018

When #4895 got merged, it closed this issue. But as that PR describes, it only solves this issue partially. Can it be reopened until binary support is a full first-class citizen? Or did I miss something and is it actually supported out of the box now?

cc @HyperBrain

@HyperBrain

This comment has been minimized.

Show comment
Hide comment
@HyperBrain

HyperBrain May 18, 2018

Member

@ronkorving Thanks for the hint.

@mvayngrib @eraserfusion I'll reopen the issue. Can you elaborate on the merged PR and maybe tell what exactly is needed additionally now to have full support of binary responses?
We might have to adjust the issue subject then.

Member

HyperBrain commented May 18, 2018

@ronkorving Thanks for the hint.

@mvayngrib @eraserfusion I'll reopen the issue. Can you elaborate on the merged PR and maybe tell what exactly is needed additionally now to have full support of binary responses?
We might have to adjust the issue subject then.

@HyperBrain HyperBrain reopened this May 18, 2018

@mvayngrib

This comment has been minimized.

Show comment
Hide comment
@mvayngrib

mvayngrib May 22, 2018

Contributor

@HyperBrain see #2797 (comment) , I don't really have anything to add :)

Contributor

mvayngrib commented May 22, 2018

@HyperBrain see #2797 (comment) , I don't really have anything to add :)

@ronkorving

This comment has been minimized.

Show comment
Hide comment
@ronkorving

ronkorving May 22, 2018

Contributor

@mvayngrib Do you think there's anything against that just being the default setting?

Contributor

ronkorving commented May 22, 2018

@mvayngrib Do you think there's anything against that just being the default setting?

@mvayngrib

This comment has been minimized.

Show comment
Hide comment
@mvayngrib

mvayngrib May 22, 2018

Contributor

@ronkorving not sure i understood, which thing being the default setting?

Contributor

mvayngrib commented May 22, 2018

@ronkorving not sure i understood, which thing being the default setting?

@ronkorving

This comment has been minimized.

Show comment
Hide comment
@ronkorving

ronkorving May 23, 2018

Contributor

@mvayngrib Is there any reason not to just have serverless configure support for binary by default, without users having to be explicit about it? It doesn't hurt non-binary responses in any way, does it? I'm a total serverless noob, so I may be misunderstanding some of the philosophies at play here completely.

Contributor

ronkorving commented May 23, 2018

@mvayngrib Is there any reason not to just have serverless configure support for binary by default, without users having to be explicit about it? It doesn't hurt non-binary responses in any way, does it? I'm a total serverless noob, so I may be misunderstanding some of the philosophies at play here completely.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment