Multiple Issues with Layers + Node Modules

[iq-dot]

This is a Bug Report

Description

I am having various issues trying to get a layer with node_modules which is shared by several functions in my serverless file.

I have tried first creating a layer with just node_modules, however due to how the layer is extracted in /opt my functions can not find node_modules.

I tried to set the NODE_PATH but it overwrites the existing NODE_PATH leading to other errors, I could not find any way to append, is there a way?

I then tried to created a 'nodejs' folder to match the directory structures of where layers are searched for, however I can not use package exclude I get the error 'No file matches include / exclude patterns'.

Here is my config with the relevant option:

layers:
  analyticsShared:
    path: nodejs
    name: analytics-shared-${opt:stage, self:provider.stage}
    description: "Shared node modules for analytics"

package:
  exclude:
    - node_modules/**
    - nodejs/**

functions:
  track:
    handler: analytics.queue
    timeout: 30
    layers:
      - {Ref: AnalyticsSharedLambdaLayer}

I tried using node_modules directly without package exclude and symlink but I need a way to append to the NODE_PATH.

There seems to be multiple issues with tryings to get node_modules as a layer, is there a documented way to get this working or are these known issues?

Additional Data

• Serverless Framework Version you're using: 1.4
• Operating System: Mac OSX

[jimmdd]

what's your layers' folder structure? The folder should be [name]/nodejs/node_module, for instance, base on your layers definition, the folder for your layers should be nodejs/nodejs/node_module, check lambda layers file path and folder structure here: https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html, I have a oracledb layers example with serverless, you can ignore the oracle lib but should be able to help you here: https://github.com/jimmdd/node-oracle-layers-serverless

[iq-dot]

If I do that then I need to exclude the folder from the package but package exclude doesn't work with the layers, I tried by creating a folder structure and doing a symlink to the node modules folder at my project root.

…

On Fri, 19 Apr 2019, 19:37 jimmdd, ***@***.***> wrote: what's your layers' folder structure? The folder should be [name]/nodejs/node_module, for instance, base on your layers definition, the folder for your layers should be nodejs/nodejs/node_module — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#6005 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA2HQHFY2U733BJNDNFHAWDPRIGNNANCNFSM4HEHX5GQ> .

[jimmdd]

you don't have to do symlink or reference to node module folder, the layer will be added to your lambda after you deploy and it will be in the node_module in the /opt/nodejs/node_module. If you run sls package and then you will see the deployment package has the layer packed seperately so you don't have to exlcude files. you only need to require the module you use in your code. just add the modules you need as dev dependencies for you to run locally. Alternatively, you can use serverless-webpack to pack your code and then force exclude modules you need. I have been using this for all my projects with layers and works perfectly.

[iq-dot]

I'll give it a go with the dev depencies, which is a viable solution. I do think I need the symlink though as the folder structure doesn't match how node module packages are installed and maybe confusing to other developers working on it. How are you proposing I manage a folder structure for layers and a standard node modules folder at root for working locally and managing depencies with npm?

…

On Fri, 19 Apr 2019, 20:06 jimmdd, ***@***.***> wrote: you don't have to do symlink or reference to node module folder, the layer will be added to your lambda after you deploy. you only need to require the module you use in your code. just add the modules you need as dev dependencies for you to run locally. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#6005 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA2HQHAYAQZ7G7465HXRHBLPRIJ3JANCNFSM4HEHX5GQ> .

[jimmdd]

I agree severless should handle the link between lambda and layers better on local.
Here is how I did it, (oraclelib folder is the layer)

for your layers: root/layers/nodejs/node_modules/[your dependencies on layers]
for your normal lambda dependencies on the same root: root/node_modules/[your lambda dependencies and layers dependencies as dev dependencies], of course, you don't have to put your code directly in the root as same as layers folder, just make sure your normal lambda's package.json and serverless.yml and node_module are in the same root.
Additionally, you can always kept a package.json file in your layers with all the dependencies you need. and add a npm install command in that directory before you build.

[iq-dot]

So do you manually keep the layers folder node modules and the normal node modules in sync manually?

…

On Tue, 23 Apr 2019, 15:48 jimmdd, ***@***.***> wrote: I agree severless should handle the link between lambda and layers better on local. Here is how I did it, (oraclelib folder is the layer) [image: Screen Shot 2019-04-22 at 4 45 08 PM] <https://user-images.githubusercontent.com/11653910/56528950-ff1c1c00-651d-11e9-90b2-22654ee3ea76.png> for your layers: root/layers/nodejs/node_modules/[your dependencies on layers] for your normal lambda dependencies on the same root: root/node_modules/[your lambda dependencies and layers dependencies as dev dependencies], of course, you don't have to put your code directly in the root as same as layers folder, just make sure your normal lambda's package.json and serverless.yml and node_module are in the same root. Additionally, you can always kept a package.json file in your layers with all the dependencies you need. and add a npm install command in that directory before you build. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#6005 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AA2HQHHP33Z7HPFRX47CML3PR4OTBANCNFSM4HEHX5GQ> .

[jimmdd]

yes i would say manually in sync. layers supposed to be used to store some static things not change often. Recently, I just started to separate layers and manage them in different projects and then just ref them using their arn.

[piotr-cz]

My solution is to

• add layer as devDependency in the root project via file:layername/nodejs
  this way it's dependencies are included in root's node_modules for local development but excluded on deploy
• use serverless-plugin-layer-manager to install layer dependencies inside layer directory on sls deploy/ package