Outage: Version 4.1.13 Breaking Changes (Deployments + Building)

[NicoHinderling]

Issue description

As of one hour ago, all of our SLS commands have broken. It seems like the inclusion/exclusion logic changes are causing issues.

the errors pop up for non aws-sdk things too (ex. "semver" or "datadog-lambda-js")

All of our CI is essentially broken :(

Two asks:

1. Most important How can we pin to a specific version of serverless? This is the second time now where deployments broke from a bad version release and as far as I can tell, there's no way to turn off the auto updating of serverless4. I already tried pinning to version 4.1.12 in our package.json but it still installs 4.1.13 under the hood which makes it impossible for us to mitigate this issue ourselves right now. Please tell me if I'm missing something!
2. Request to help mitigate the underlying issues in 4.1.13 so we can eventually roll forward

Context

No response

[noahsmartin]

I suspect it's this line: https://github.com/serverless/serverless/pull/12678/files#diff-8e49a112b340fce94cbf8b22d2d457889d33bd028e3e83d02d332e98a5a74a1bR410 which if I'm reading it correctly would break any external/excluded rules that are for dependencies not install in the package.json (for example anything in a lambda layer which we'd want to mark as external)

[NicoHinderling]

cc @austencollins @Mmarzex

[austencollins]

We're looking into this right now..

[austencollins]

Working on the fix. ETA coming shortly.

[Mmarzex]

@NicoHinderling are you able to share what your esbuild configuration is? I believe I know the issue but want to confirm based on what you are doing.

[NicoHinderling]

for this service it is currently:

build:
  esbuild:
    external:
      - '.prisma'
      - 'semver'
      - 'sharp'
      - 'pg-format'
      - 'simple-plist'
      - 'mustache'
      - 'datadog-lambda-js' # required to be external: https://docs.datadoghq.com/serverless/guide/serverless_tracing_and_webpack/#esbuild
    exclude:
      - 'datadog-lambda-js' # required to be also be excluded: https://docs.datadoghq.com/serverless/guide/serverless_tracing_and_webpack/#esbuild
    minify: false

here's another esbuild config from a different service that's also experiencing this issue:

build:
  esbuild:
    external:
      - '.prisma'
      - 'semver'
      - 'sharp'
      - '/opt/lambda-layer-sharp-heic/nodejs/node20/node_modules/sharp'
      - 'simple-plist'
      - 'tree-sitter'
      - 'tree-sitter-swift' # including this depdendency causes 'npx sls package' build time to go from 6s to 80s
      - 'datadog-lambda-js'
    exclude:
      - 'sharp'
      - '/opt/lambda-layer-sharp-heic/nodejs/node20/node_modules/sharp'
      - 'datadog-lambda-js'
    minify: false

[Mmarzex]

Thanks @NicoHinderling sorry this took so long, wanted to verify everything before publishing a release. A new release is coming shortly.

[NicoHinderling]

Hey @Mmarzex just checking in

Is an ETA of EOD pst time looking possible?

[Mmarzex]

Sorry @NicoHinderling there is currently a Github Actions outage that is blocking the release pipeline from finishing. Will update as soon as it is working again.

[NicoHinderling]

Thanks for the timely update. Ah dang, brutal timing. Sounds good 🙏

[Mmarzex]

@NicoHinderling A new version has finally been released. If you want to run a serverless update and try it again?