New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Need to bundle specific gRPC binary with Lambda function using serverless webpack #342

Closed
pflugs30 opened this Issue Mar 9, 2018 · 9 comments

Comments

Projects
None yet
2 participants
@pflugs30
Copy link

pflugs30 commented Mar 9, 2018

  • Serverless-Webpack Version you're using: 5.0.0
  • Webpack version you're using: 3.11.0
  • Serverless Framework Version you're using: 1.26.1
  • Operating System: win-64

I'm trying to use the serverless framework to build a Lambda function that pulls messages from Google PubSub. When using the PubSub NodeJS package, one of the dependencies is gRPC which requires that a binary file be downloaded during the install process. Since I'm on a windows laptop for dev, the gRPC install script downloads node-v48-win32-x64-unknown. However, Lambda runs on Linux, and it expects build node-v48-linux-x64-glibc. Per this comment, it is possible to force npm to rebuild the gRPC package to include the Linxu binary.

My problem is that I can't figure out how to include that in the serverless deployment. When serverless webpack bundles everything up, it includes gRPC with the windows version. When this whole package is successfully deployed to AWS, the Lambda function throws the following error:

Unable to import module 'handler': Error
Expected directory: node-v48-linux-x64-glibc
Found: [node-v48-win32-x64-unknown]
This problem can often be fixed by running "npm rebuild" on the current system
Original error: Cannot find module '/var/task/node_modules/grpc/src/node/extension_binary/node-v48-linux-x64-glibc/grpc_node.node'
at Object. (/var/task/node_modules/grpc/src/grpc_extension.js:53:17)...

So my question is: how can I insert the package rebuild command as part of the serverless-webpack process so the correct version of the gRPC binary gets sent to AWS? It's as if the serverless-webpack is doing a full npm install, and I'd like to insert after that is done to rebuild the gRPC module. I understand that this is a complicated situation and question, and I'm happy to provide any other information.

Serverless.yaml

service: myService

provider:
  name: aws
  runtime: nodejs6.10
  stage: dev
  region: us-east-1

plugins:
  # Use serverless-webpack plugin to transpile ES6/ES7
  - serverless-webpack
  - serverless-offline
  
custom:
  webpack:
    includeModules: true # enable auto-packing of external modules

functions:
  pullData:
    handler: handler.pullData
    events:
      - http:
          path: pullData
          method: post
          cors: true

webpack.config.js

const path = require('path');
const slsw = require("serverless-webpack");
const nodeExternals = require("webpack-node-externals");

module.exports = {
  entry: './handler.js',
  target: 'node',
  externals: [nodeExternals()],
  module: {
    loaders: [{
      test: /\.js$/,
      loaders: ['babel-loader'],
      include: __dirname,
      exclude: /node_modules/
    }]
  },
  output: {
    libraryTarget: 'commonjs',
    path: path.join(__dirname, '.webpack'),
    filename: 'handler.js'
  }
};
@HyperBrain

This comment has been minimized.

Copy link
Member

HyperBrain commented Mar 9, 2018

Hi @pflugs30 , thanks for asking.

Yes, that is indeed a special setup, especially that you need to package something that is foreign to the build architecture.
An idea would be to try to add a "prepare" script to your package.json that executes the build of the linux variant. According to https://docs.npmjs.com/misc/scripts prepare scripts are run when doing a npm install without arguments - i.e. it would run it when sls-webpack does its npm install.

@HyperBrain

This comment has been minimized.

Copy link
Member

HyperBrain commented Mar 9, 2018

After thinking again, I doubt that the script approach will work at all, because currently the webpack plugin does not allow for custom scripts to be executed while packaging the functions/service.
It will create a package.json that only contains the dependencies but does not add scripts there.

However, with the new packager implementation, it would be possible to implement that as a new feature. I'll create a separate task for that and link this issue.

@pflugs30

This comment has been minimized.

Copy link
Author

pflugs30 commented Mar 9, 2018

Thanks for the response, @HyperBrain. If there's anything I can do to help or test, please let me know. It appears that I may be stuck on my project until this is done.

For those who may find there way here via Google, here are things I've tried that have NOT worked:

  1. Manually copying the correct compiled binary into the package created by Serverless. Failed due to incorrect SHA checksum (I think).
  2. Manually editing the package.json file of the gRPC node module to hard code the version I wanted. Didn't actually copy the desired file into the serverless package.
  3. Adding the --target... parameters to the serverless package command call.
  4. Looking for a way to customize the webpack config file to force a rebuild of the gRPC package during build.

If anyone has other ideas, I'm all ears! :-) Thanks.

@HyperBrain

This comment has been minimized.

Copy link
Member

HyperBrain commented Mar 9, 2018

@pflugs30 Working on a fix right now. I think I'll have a PR ready soon, so that you can test if it solves the problem.

@HyperBrain HyperBrain referenced this issue Mar 9, 2018

Merged

Support custom packager scripts #344

7 of 7 tasks complete
@HyperBrain

This comment has been minimized.

Copy link
Member

HyperBrain commented Mar 9, 2018

PR is available. Use "serverless-webpack": "github:serverless-heaven/serverless-webpack#support-scripts" to test if it works.

@pflugs30

This comment has been minimized.

Copy link
Author

pflugs30 commented Mar 9, 2018

Thank you very much, @HyperBrain. I am beginning to test it right now, and I will get back to you ASAP. Thank you for your incredibly prompt response.

@HyperBrain HyperBrain added this to the 5.1.0 milestone Mar 9, 2018

@pflugs30

This comment has been minimized.

Copy link
Author

pflugs30 commented Mar 9, 2018

Closed with the resolution of Issue #344.

@HyperBrain

This comment has been minimized.

Copy link
Member

HyperBrain commented Mar 14, 2018

Released with 5.1.0

@pflugs30

This comment has been minimized.

Copy link
Author

pflugs30 commented Apr 20, 2018

Further comment for future Googlers (including myself)...

I now am using nvm for Windows to dynamically switch between versions of Node and NPM. I am also now designing for NodeJS v8.10 since AWS Lambdas now support it. As such, when I redeployed my Lambda function, I encountered the following error:

Unable to import module 'handler': Error
Expected directory: node-v57-linux-x64-glibc
Found: [node-v48-linux-x64-glibc, node-v57-win32-x64-unknown]
This problem can often be fixed by running "npm rebuild" on the current system
Original error: Cannot find module '/var/task/node_modules/grpc/src/node/extension_binary/node-v57-linux-x64-glibc/grpc_node.node'
at Object.<anonymous> (/var/task/node_modules/grpc/src/grpc_extension.js:53:17)
...

I realized I was still building the gprc for NodeJS v6.10. As such, I had to change the postinstall command within my package.json to one that would build the expected version, which is node-v57-linux-x64-glibc.

The corrected command in the package.json file is below:
"postinstall": "npm rebuild grpc --target=8.1.0 --target_arch=x64 --target_platform=linux --target_libc=glibc"

Here is the resulting section of the serverless.yaml file:

custom:
  webpack:
    packExternalModulesMaxBuffer: 1000000
    includeModules: true # enable auto-packing of external modules
    packagerOptions:
      scripts:
        - npm rebuild grpc --target=8.1.0 --target_arch=x64 --target_platform=linux --target_libc=glibc
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment