package individually doesnt include node_modules and package.json

[yeukfei02]

Hi,

this is a nice lib.
but I try this lib with the serverless framework, I face some problem:

serverless.yml

package:
  individually: true

I didn't set anything in custom..inside the .serverless folder I don't see any node_modules + package.json file.
If I deploy this lambda func to aws, I guess will fail?
How can I make node_modules + package.json file included in each lambda func zip?

Thanks!

[olup]

I'd say you have to declare externals if you want to have your modules added as externals modules to your zip. Otherwise, all the code is bundle as one js file, isn't it ?

It shouldn't fail, as long as you have no native dependencies in your modules - ie only javascript depencies that are bundable by esbuild

[yeukfei02]

@olup

lets say I have 20+ restapi in serverless.yml
I check another issue say need to add include to each http event, so i need to manually add all include to those 20+ API...

because I was using serverless-webpack and this can help to add node_modules + package.json file to the individual package.
I am looking for esbuild to replace webpack because it is faster

[olup]

just use the external key of the config if you need, otherwise a single bundle should do.

"I check another issue say need to add include to each http event, so i need to manually add all include to those 20+ API..." this is a workaround only needed if you want to have specific bundle size on each lambda. But it's a workaround, and may not be advisable in your own case.

The bottom line is this package will work with individual packaging, but will not optimize the bundle size for each lambda, period. (or until the other ticket is addressed by maintainer or contributor)

[yeukfei02]

@olup

thanks for the answer.
I think I will keep serverless-webpack first, but I will use esbuild-loader with it first.
for all the lambda func to do package individually.

[floydspace]

Hey @yeukfei02, we released the individually packaging fix in v1.8.0, could you please verify that it resolves your issue.

[yeukfei02]

@floydspace

hey i just try in one of my project, I use with serverless framework and run yarn run package, this means serverless package..it cannot include serverless_sdk folder

yarn run package 
yarn run v1.22.10
$ serverless package
Serverless: Compiling with esbuild...
Serverless: Compiling completed.
Serverless: Zip function: k11-eshop-facebook-lambda-dev-eshopFacebook - 899.43 KB [75 ms]
 
  Error --------------------------------------------------
 
  Error: ENOENT: no such file or directory, scandir 'serverless_sdk'

thanks.

[olup]

hey @yeukfei02 . could you post here your serverless.yml file ?
serverless_sdk is a special case that we should specifically handle - as it is added to the bundle automatically by serverless.

[yeukfei02]

@olup

serverless.yml

org: keysocapp
app: k11-eshop-facebook-lambda
service: k11-eshop-facebook-lambda

frameworkVersion: '2'
useDotenv: true

provider:
  name: aws
  runtime: nodejs12.x
  stage: dev
  region: ap-southeast-1
  apiName: k11-eshop-facebook-lambda-dev
  tracing:
    apiGateway: true
    lambda: true
  environment:
    NODE_ENV: development
    MY_AWS_ACCESS_KEY: ${param:MY_AWS_ACCESS_KEY}
    MY_AWS_SECRET_ACCESS_KEY: ${param:MY_AWS_SECRET_ACCESS_KEY}
    K11_ESHOP_FACEBOOK_LAMBDA_DATABASE_URL_DEV: ${param:K11_ESHOP_FACEBOOK_LAMBDA_DATABASE_URL_DEV}
    K11_ESHOP_FACEBOOK_LAMBDA_DATABASE_URL_PROD: ${param:K11_ESHOP_FACEBOOK_LAMBDA_DATABASE_URL_PROD}
  iam:
    role:
      statements:
        - Effect: 'Allow'
          Action:
            - 'xray:PutTraceSegments'
            - 'xray:PutTelemetryRecords'
          Resource:
            - '*'
        - Effect: 'Allow'
          Action:
            - 's3:PutObjectACL'
          Resource:
            - '*'
  apiGateway:
    shouldStartNameWithService: true
    description: K11 EShop Facebook Lambda Staging
  lambdaHashingVersion: 20201221
  vpc:
    securityGroupIds:
      - sg-0ba9e979cffe84d1c
    subnetIds:
      - subnet-0274899de33681214
  timeout: 30

package:
  individually: true

functions:
  eshopFacebook:
    handler: src/eshopFacebook/handler.eshopFacebook
    events:
      - http:
          path: /
          method: post
          cors: true
      - schedule: cron(0 15 * * ? *)
    timeout: 300

plugins:
  # - serverless-webpack
  - serverless-esbuild
  - serverless-prune-plugin
  # - serverless-plugin-typescript
  - serverless-offline
  # - serverless-plugin-split-stacks

custom:
  # webpack:
  #   webpackConfig: ./webpack.config.js
  #   includeModules: true
  #   packager: 'yarn'
  #   packagerOptions:
  #     scripts:
  #       - prisma generate
  prune:
    automatic: true
    number: 3

serverless version: 2.25.2

[olup]

Interesting. Would it be a problem for you to attach or link the zip'ed package for this function ? I'd like to check how the serverless_sdk is imported in the bundle. Other question, in your handler's code, do you import or require serverless sdk at any point ? and can I see the package.json ?

[yeukfei02]

@olup

package.json

{
  "name": "k11-eshop-facebook-lambda",
  "version": "0.1.0",
  "main": "app.js",
  "repository": "https://donaldwu05@bitbucket.org/keysocapp/k11-eshop-facebook-lambda.git",
  "author": "Donald Wu <donald.wu@keysocapp.com>",
  "license": "MIT",
  "dependencies": {
    "@fast-csv/format": "^4.3.5",
    "@prisma/client": "2.16.1",
    "aws-sdk": "^2.845.0",
    "aws-xray-sdk": "^3.2.0",
    "dotenv": "^8.2.0"
  },
  "devDependencies": {
    "@types/aws-lambda": "^8.10.72",
    "@types/jest": "^26.0.20",
    "@types/node": "^14.14.28",
    "@types/serverless": "^1.78.20",
    "@typescript-eslint/eslint-plugin": "^4.15.1",
    "@typescript-eslint/parser": "^4.15.1",
    "aws-lambda": "^1.0.6",
    "codecov": "^3.8.1",
    "copy-webpack-plugin": "5.1.1",
    "esbuild-loader": "^2.9.2",
    "eslint": "^7.20.0",
    "eslint-config-prettier": "^7.2.0",
    "eslint-plugin-prettier": "^3.3.1",
    "git-cz": "^4.7.6",
    "husky": "^5.0.9",
    "jest": "^26.6.3",
    "prettier": "^2.2.1",
    "prisma": "2.16.1",
    "serverless": "^2.25.2",
    "serverless-offline": "^6.8.0",
    "serverless-plugin-split-stacks": "^1.11.0",
    "serverless-plugin-typescript": "^1.1.9",
    "serverless-prune-plugin": "^1.4.3",
    "serverless-webpack": "^5.3.5",
    "standard-version": "^9.1.0",
    "ts-jest": "^26.5.1",
    "ts-node": "^9.1.1",
    "typescript": "^4.1.5",
    "webpack": "4.46.0",
    "webpack-node-externals": "^2.5.2"
  },
  "scripts": {
    "dev": "serverless offline start",
    "deploy": "serverless deploy",
    "package": "serverless package",
    "dashboard": "serverless dashboard",
    "remove": "serverless remove",
    "test": "jest --detectOpenHandles",
    "lint": "eslint '*/**/*.{ts,tsx}' --quiet --fix",
    "generate:prisma": "cd ./prisma && npx prisma introspect && npx prisma generate",
    "format:prisma": "cd ./prisma && npx prisma format",
    "prisma-studio": "npx prisma studio",
    "commit": "git-cz",
    "release": "standard-version"
  },
  "husky": {
    "hooks": {
      "pre-commit": "yarn lint && yarn run format:prisma",
      "pre-push": "yarn lint && yarn run format:prisma"
    }
  }
}

my handler.ts didnot import any sls sdk..i believe when i run yarn run package the serverless-esbuild didnot include node_modules + package.json + schema.prisma

its ok. i am still using serverless-webpack

[olup]

serverless esbuild did not include node_modules nor package.json because you did not declare any externals. This is intended behavior. The is no dependencies to add to the bundle if you don't declare them in externals (the same goes for serverless-webpack, but you might be using node_externals plugin that takes ALL your dependencies as externals).

Same goes for your prisma schema. You have to declare it otherwise it won't be picked up as it's not a js/ts file.

the serverless_sdk is added automatically to your code by serverless because you must be using their hosted dashboard and metrics.

[yeukfei02]

@olup

yes I use webpack-node-externals in the webpack.config.js
I think for my case will use serverless-webpack + esbuild-loader first.

thanks.

[dan-newscorp]

+1 I had the same issue as OP.

I was unable to include the package.json at all using the documented method for packaging e.g.

// serverless.yml
package:
  patterns:
    - '!node_modules/@aws-sdk/**'
    - package.json

Specifying externals, did include the node_modules correctly, but still no package.json

In my use case the lack of a package.json meant the lambda was not executing as an esm module.

Fortunately i was able to add outputFileExtension: .mjs to achieve this.

P.S. Love the plugin. So much faster than webpack