Native modulse for Sharp, Installing modules in lambda docker

[danielstaleiny]

Question

Description

Hi I am trying to make sharp work with serverless and webpack plugin. I keep hitting the error module initialization error: Error. I suspect it is because sharp lib uses native and that is why it needs to be installed in lambda docker. Is there way to set up this workflow ?

Reference to sharp note:
http://sharp.dimens.io/en/stable/install/#aws-lambda

How should be a workflow to correctly bundle this ?

I tried to do it manually but webpack plugin re-bundled it on deployment. Any help is appreciated.

I am on 5.1.5 of Serverless-webpack
and Serverless 1.27.2

[HyperBrain]

Hi @danielstaleiny , interesting question 👍 . I will investigate the docs and see how their intentions are about a working deployment/installation and how that would play together with serverless-webpack.

At anyone coming here who frequently uses native modules (or lambda docker) or sharp, feel free to add your thoughts.

[HyperBrain]

Quick update: I fetched some infos from their website:

Pre-compiled binaries for sharp are provided for use with Node versions 4, 6, 8 and 9 on 64-bit Windows, OS X and Linux platforms.
Most recent Linux-based operating systems with glibc running on x64 and ARMv6+ CPUs should "just work"

This could mean that we only have to add the correct binary to the webpack package and use node-loader to make it work. Amazon Linux should be no problem.
I'll investigate further to see if such a standard workflow could be used.

[danielstaleiny]

@HyperBrain Can you please point me to docs where I can dive into setting these options ? :)) some examples anything. I would like to get moving with this otherwise I have to deploy it without serverless framework. :)) I appreciate you to having a look at this.

[HyperBrain]

I think you have to add the node-loader to your package.json and add .node to the resolve array in the webpack config.

Then we need to make it build the linux version instead of the one for the local environment - for the rebuild script you have to set the proper node target version:

# serverless.yml
custom:
  webpack:
    packagerOptions:
      scripts:
        - npm rebuild sharp --target=8.10.0 --target_arch=x64 --target_platform=linux

Finally, include the Webpack copy plugin into your webpack configuration to include the binary folder from sharp to your package.

Sorry that I could not go into details right now, I'm just writing this without having it tested myself (especially the rebuild command line, so you might need to test and fix that) - I'm currently at work and do not have very much time.
The rough configuration given above should be the general approach that works.

Maybe someone who also uses native modules and added related issues here can help out. @janthonyeconomist, I think you also used a native node module and integrated that. Maybe you can help out here 😃 . Thanks.

[janthonyeconomist]

Sorry. I don't think I can be much help. I ended up going with a pure JS version of the library https://www.npmjs.com/package/maxmind to avoid the problem.

[JakubMatejka]

We use sharp in https://github.com/keboola/developer-portal and the deployment is run in Docker. The image inherits from amazonlinux to maintain the closest possible compatibility with lambda environment and it works fine so far. See:

• https://github.com/keboola/developer-portal/blob/master/Dockerfile
• https://github.com/keboola/developer-portal/blob/master/docker-compose.yml

[HyperBrain]

Ok. Then you even might not need the "rebuild" script configuration as the install of the native module already fetches the correct version. Just the node-loader and the copy plugin to include the files should work then.

[danielstaleiny]

Thank you guys, works like charm. @JakubMatejka is Python27 needed in dockerfile ? and what does

nodeExternals({
      modulesFromFile: true
})

do ?

[JakubMatejka]

@danielstaleiny Great! Python is a prerequisite for node-gyp which is required by sharp. And webpack-node-externals module tells Webpack which modules are actively used in code and should be bundled (https://github.com/serverless-heaven/serverless-webpack#node-modules--externals), modulesFromFile: true reads the dependencies list from package.json instead of node_modules directory (https://www.npmjs.com/package/webpack-node-externals#optionsmodulesfromfile-false).

[danielstaleiny]

I thought so, thank you Jakub. Closing this issue :)

[rad182]

hi @danielstaleiny sorry for digging this issue. I'm also having the same problem.
I already added the webpack packagerOptions but still doesn't work.
Can you share how you made this work?

[danielstaleiny]

Here is simplify version of the JakubMatejka solution to this problem. You just need to package your function in docker. https://github.com/danielstaleiny/serverless-example

[rad182]

thanks a lot @danielstaleiny !

[danielstaleiny]

Be careful on node-gyp issues with yarn. Everything worked for me and suddenly one day I woke up to continue working on my project and node-gyp failed to rebuild. After some digging I updated node-gyp version to latest 3.x which supposed to work with node 10.x and resolved my issue. I came back to yarn and it still failed to rebuild. After some digging I found that yarn has some issues with node-gyp and libraries like sharp, node-canvas. To resolve issues with node-canvas using node-gyp I was forced to use npm. (and also make sure that your default python executable is 2.xx not python 3.xx) 1 day wasted. Happy hacking guys.