How to include binaries within generated function zip?

[nicka]

I can't figure out how to store binaries within the generated function zip.

For example:

functions/ffmpeg
  - event.json
  - ffmpeg(binary)
  - handler.js
  - s-function.json

Only thing present on AWS is the _serverless_handler.js file.

[eahefnawy]

could you deploy your function again with sls function deploy -t (notice the -t flag) ... after deployment is complete you should see a meta/tmp folder that contains the package that was uploaded to AWS. Could you show us what you got inside that folder?

[nicka]

Wow, thanks for the quick reply. Do I need to add something to s-function.json?

[nicka]

My bad something like this works:

{
  "name": "fetcher",
  ...
  "custom": {
    "excludePatterns": [],
    "optimize": {
      "exclude": [
        "aws-sdk"
      ],
      "includePaths": [
        "binaries"
      ]
    }
  },
  ...
}

Other than that it would be awesome if https://github.com/serverless/serverless-optimizer-plugin could support something like the example below and put those files to the root of the zip:

"includeFiles": [
  "binaries/ffmpeg"
]

[eahefnawy]

oh yeah I was just gonna ask you about the optimize plugin! 😄

Please let me know if there's anything else I can help you with 😊

[nicka]

Haha thanks man!

How would you feel about something like?

"includeFiles": [
  "binaries/ffmpeg"
]

[eahefnawy]

hmmmm I'm not sure I completely understand what you mean by "put those files in the root of the zip", but generally we really hate changing the directory/file structure of our users projects (they may have this structure for a strict reason). So I'm not sure if moving files around to the root of the zip would be a great idea.

[nicka]

That's completely understandable. The binary is now part of the zip, next problem is permissions.

[Error: Command failed: /bin/sh: /var/task/binaries/main/ffmpeg: Permission denied

If I zip the dir myself Lambda triggers the function without any problems. Could it be that the optimize plugin is not respecting original file permissions?

EDIT
Seems like this issue with wrench ryanmcgrath/wrench-js#92

EDIT
Seems like we need to move away from wrench ryanmcgrath/wrench-js@d8e29ae

EDIT
wrench has been removed from serverless-optimizer-plugin in serverless/serverless-optimizer-plugin@0ebc25b 👍

[nicka]

@eahefnawy Fixed the permissions issue in #960

[eahefnawy]

@nicka thanks for keeping us updated! Is everything working for u now? :)

[nicka]

@eahefnawy No problem! Only thing left is for serverless to retain original file permissions while generating the zip. But that should be fixed once #960 is approved 😄

[eahefnawy]

wow thanks a lot for the PR! 😊

[peterschwarz21]

Is there an update for this for serverless.yml? Using serverless webpack I;m not sure how to include binaries.

[pmuens]

@peterschwarz21 that should be possible with include https://serverless.com/framework/docs/providers/aws/guide/packaging#exclude--include

The usage of globs can help here!

How does your serverless.yml look like? Where do you face problems?

[peterschwarz21]

I must be missing something really easy lol. Here is my dir structure:

Myapp
   /node_modules
    /bin/pdftk (this is the binary)
    function1.js
    function2.js

Now when I deploy it to AWS (or run serverless deploy -t), the /bin dir is not included. i just need to include that directory in the zip file before it gets uploaded to AWS. I feel like this should be easy, but it isn't.

The serverless.yml file is like this:

service: myapp-app-api
package:
  include:
    - bin/**
plugins:
  - serverless-webpack

custom:
  webpackIncludeModules: true

.... etc

[peterschwarz21]

Success! Using the copy-webpack-plugin I was able to upload a static file to AWS.

https://github.com/kevlened/copy-webpack-plugin

[pmuens]

@peterschwarz21 awesome! Thanks for keeping us posted on this one! Glad to hear that it has been resolved 👍

[laardee]

@peterschwarz21 you might face permission issues if you use copy-webpack-plugin with binaries that you need to execute in lambda environment. I had to use serverless-plugin-optimize to get ffmpeg working in my project.

[b04zdotcom]

@laardee Can you elaborate? I'm currently facing permission issues with ffmpeg in lambda using the copy-webpack-plugin.

EDIT
I was able to solve this by using the webpack-shell-plugin to create the .webpack folder that serverless-webpack uses and then copy the binary over to that folder.

plugins: [
  new WebpackShellPlugin({ onBuildStart:['mkdir .webpack', 'cp ./lib/ffmpeg .webpack'] })
],

[laardee]

@boazdejong nice, I need to check out that plugin also.

[HyperBrain]

@boazdejong @laardee With the upcoming V3 version of the webpack plugin copying the file right into the .webpack folder might not work anymore, because of the individual packaging support there might be multiple subfolders now where the artifacts/handlers are built and packaged.
I see great value in a raw file import functionality for the webpack plugin that would just copy given files as is to the handler's package folder. Can you please create a feature request within the repo and I'll schedule the feature as soon as possible. (https://github.com/elastic-coders/serverless-webpack/issues)

[b04zdotcom]

@HyperBrain That would be a great feature! I just added the request. Thank you.
serverless-heaven/serverless-webpack#205