Exclude not excluding a node_modules subdirectory #7837

nampas · 2020-06-12T16:45:40Z

I was trying to pull all my node_modules into a lambda layer, but running into an issue because the lambda layer is too big.

ServerlessError: An error occurred: NodemodulesLambdaLayer - Unzipped size must be smaller than 262144000 bytes

Fair enough, it is big. I went ahead and tried to exclude the aws-sdk node_modules subdirectory:

layers:
  nodemodules:
    name: ${self:service.name}-node-modules-${self:provider.stage}
    path: node_modules
    package:
      exclude:
        - '!node_modules/**'
        - 'node_modules/aws-sdk/**'

package:
  exclude:
    - .envrc*
    - node_modules/**
    - kubernetes/**
    - ci/**
    - images/**

That doesn't work though. I'm still getting the aws-sdk included in the zip:

$ sls package
... does its thing ...
$ unzip .serverless/nodemodules && ls -la .serverless/nodemodules | grep aws-sdk
drwxr-xr-x   24 npastor  staff     768 Jun 12 12:30 aws-sdk

serverless.yml

service: resourceSync

plugins:
  - serverless-plugin-vpc-eni-cleanup
  - serverless-s3-remover
  - serverless-step-functions

custom:
  remover:
    buckets:
      - ${self:custom.lastRunTimeBucketName}
  byStage:
    default:
      # properties
    prod:
      # prod properties
    staging:
      # staging proeprties 
  # Consolidated properties for current stage

provider:
  #
  # General settings
  #
  name: aws
  # Deploy a different stage via `serverless deploy -v --stage somestage`
  stage: ${opt:stage, 'dev'}
  deploymentBucket:
    name: ${self:custom.s3DeployBucket}
  #
  # Lambda-specific settings
  #
  runtime: nodejs10.x
  memorySize: 256
  timeout: 30 # seconds
  versionFunctions: false
  logRetentionInDays: 14
  environment:
    # Env vars
  tags:
    owner: core-marketplace
  role: ${self:custom.lambdaRole}
  vpc: ${self:custom.vpc}
  layers: [{ Ref: NodemodulesLambdaLayer }]

layers:
  nodemodules:
    name: ${self:service.name}-node-modules-${self:provider.stage}
    path: node_modules
    package:
      exclude:
        - '!node_modules/**'
        - 'node_modules/aws-sdk/**'

package:
  exclude:
    - .envrc*
    - node_modules/**
    - kubernetes/**
    - ci/**
    - images/**

functions:
  # Token authentication function for api gateway requests
  tokenAuthorizer:
    enabled: ${self:custom.apiGatewayEnabled}
    name: ${self:service.name}-tokenAuthorizer-${self:provider.stage}
    handler: src/handlers/token_authorizer.handler
    environment:
      TOKEN: ${ssm:${self:service.name}-tokenAuthorizerKey-dev}

  # A function that receives messages from the main event queue
  processResourceUpdate:
    name: ${self:service.name}-processResourceUpdate-${self:provider.stage}
    handler: src/handlers/process_resource_update.handler
    timeout: 120 # needs to be less than or equal to the EventQueue VisibilityTimeout
    events:
      - sqs:
          arn:
            Fn::GetAtt:
              - EventQueue
              - Arn
          batchSize: 1
    environment:
      REDIS_ENDPOINT: ${self:custom.redisEndpoint}
      REDIS_KEY_EXPIRATION: ${self:custom.redisKeyExpiration}
      RESOURCE_CATALOG_S3_BUCKET: ${self:custom.resourceCatalogS3Bucket}
    reservedConcurrency: 50

  # There are some other functions but they're not much different

stepFunctions:
  stateMachines:
    # Input:
    #   {
    #     afterId:    10000, // optional. ID to start after. Default is 0.
    #     untilId:    20000, // optional. Maximum ID to fetch. Default is undefined.
    #     queryLimit:  5000, // optional. Limit of how many ids to query per Lambda invocation. Default is 5000.
    #   }
    reimport:
      id: ResourceReimportStateMachine
      name: ${self:service.name}-reimport-${self:provider.stage}
      role: ${self:custom.sfnRole}
      definition:
        StartAt: PublishItems
        States:
          PublishItems:
            Type: Task
            Resource:
              Fn::GetAtt: [PublishResourceReimportLambdaFunction, Arn]
            Retry:
              - ErrorEquals:
                - States.ALL
                IntervalSeconds: 4
                MaxAttempts: 4
                BackoffRate: 2
            Next: ContinueOrStop
          ContinueOrStop:
            Type: Choice
            Choices:
              - Variable: '$.hasMore'
                BooleanEquals: True
                Next: PublishItems
            Default: Finished
          Finished:
            Type: Succeed
    # One other state machine

REPLACE WITH FULL COMMAND NAME output

$ SLS_DEBUG=* sls package
Serverless: Load command interactiveCli
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command config:tabcompletion
Serverless: Load command config:tabcompletion:install
Serverless: Load command config:tabcompletion:uninstall
Serverless: Load command create
Serverless: Load command install
Serverless: Load command package
Serverless: Load command deploy
Serverless: Load command deploy:function
Serverless: Load command deploy:list
Serverless: Load command deploy:list:functions
Serverless: Load command invoke
Serverless: Load command invoke:local
Serverless: Load command info
Serverless: Load command logs
Serverless: Load command metrics
Serverless: Load command print
Serverless: Load command remove
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command slstats
Serverless: Load command plugin
Serverless: Load command plugin
Serverless: Load command plugin:install
Serverless: Load command plugin
Serverless: Load command plugin:uninstall
Serverless: Load command plugin
Serverless: Load command plugin:list
Serverless: Load command plugin
Serverless: Load command plugin:search
Serverless: Load command config
Serverless: Load command config:credentials
Serverless: Load command rollback
Serverless: Load command rollback:function
Serverless: Load command upgrade
Serverless: Load command uninstall
Serverless: Load command s3remove
Serverless: Load command invoke
Serverless: Load command invoke:stepf
Serverless: Load command login
Serverless: Load command logout
Serverless: Load command generate-event
Serverless: Load command test
Serverless: Load command dashboard
Serverless: Load command output
Serverless: Load command output:get
Serverless: Load command output:list
Serverless: Load command param
Serverless: Load command param:get
Serverless: Load command param:list
Serverless: Load command studio
Serverless: Load command dev
Serverless: [AWS ssm 400 0.207s 0 retries] getParameter({ Name: 'resourceSync-graphApiSecret-dev',
WithDecryption: false })
Serverless: [AWS ssm 400 0.222s 0 retries] getParameter({ Name: 'resourceSync-datadogApiKey-dev',
WithDecryption: false })
Serverless: [AWS ssm 400 0.207s 0 retries] getParameter({ Name: 'resourceSync-rollbarToken-dev', WithDecryption: false })
Serverless Warning --------------------------------------

A valid SSM parameter to satisfy the declaration 'ssm:resourceSync-rollbarToken-dev' could not be found.