npm 8 / Node 16 produces deployments zip's too large to deploy (50meg), npm 6 / Node 12 fine (12meg) #10863

tomchiverton · 2022-03-17T11:19:59Z

Are you certain it's a bug?

Yes, it looks like a bug

Is the issue caused by a plugin?

It is not a plugin issue

Are you using the latest version?

Yes, I'm using the latest version

Is there an existing issue for this?

I have searched existing issues, it hasn't been reported yet

Issue description

Node 16 with NPM 8 generates deployment packages far too big - >50meg

Appears to be including things like aws-sdk that are not required.

Exact sane project using Node 12 NPM 6 produces ~12meg

Service configuration (serverless.yml) content

N/A

Command name and used flags

serverless package

Command output

$ node -v; npm version ; sls --version; sls package ; ls -laht .serverless/*zip
v16.14.0
{
  'lone-star': '1.0.0',
  npm: '8.3.1',
  node: '16.14.0',
  v8: '9.4.146.24-node.20',
  uv: '1.43.0',
  zlib: '1.2.11',
  brotli: '1.0.9',
  ares: '1.18.1',
  modules: '93',
  nghttp2: '1.45.1',
  napi: '8',
  llhttp: '6.0.4',
  openssl: '1.1.1m+quic',
  cldr: '40.0',
  icu: '70.1',
  tz: '2021a3',
  unicode: '14.0',
  ngtcp2: '0.1.0-DEV',
  nghttp3: '0.1.0-DEV'
}
Running "serverless" from node_modules
Framework Core: 1.63.0
Plugin: 3.3.0
SDK: 2.3.0
Components Core: 1.1.2
Components CLI: 1.4.0
Running "serverless" from node_modules
 
 Serverless Warning --------------------------------------
 
  A valid SSM parameter to satisfy the declaration 'ssm:/XX/jwt/secret/v1~true' could not be found.
 
 
 Serverless Warning --------------------------------------
 
  A valid SSM parameter to satisfy the declaration 'ssm:/XX/jwt/secret/v2~true' could not be found.
 
 
 Serverless Warning --------------------------------------
 
  A valid SSM parameter to satisfy the declaration 'ssm:/rds/our_db/user/incentives-sls/password~true' could not be found.
 
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Warning: Not deploying alerts on stage v1
-rw-r--r-- 1 tchiverton domain^users 49M Mar 17 11:14 .serverless/lone-star.zip


$ nvm use 12

$ node -v; npm version ; sls --version; sls package ; ls -laht .serverless/*zip
v12.22.9
{
  'lone-star': '1.0.0',
  npm: '6.14.15',
  ares: '1.18.1',
  brotli: '1.0.9',
  cldr: '37.0',
  http_parser: '2.9.4',
  icu: '67.1',
  llhttp: '2.1.4',
  modules: '72',
  napi: '8',
  nghttp2: '1.41.0',
  node: '12.22.9',
  openssl: '1.1.1m',
  tz: '2019c',
  unicode: '13.0',
  uv: '1.40.0',
  v8: '7.8.279.23-node.56',
  zlib: '1.2.11'
}
Running "serverless" from node_modules
Framework Core: 1.63.0
Plugin: 3.3.0
SDK: 2.3.0
Components Core: 1.1.2
Components CLI: 1.4.0
Running "serverless" from node_modules
 
 Serverless Warning --------------------------------------
 
  A valid SSM parameter to satisfy the declaration 'ssm:/XX/jwt/secret/v2~true' could not be found.
 
 
 Serverless Warning --------------------------------------
 
  A valid SSM parameter to satisfy the declaration 'ssm:/XX/jwt/secret/v1~true' could not be found.
 
 
 Serverless Warning --------------------------------------
 
  A valid SSM parameter to satisfy the declaration 'ssm:/rds/our_db/user/incentives-sls/password~true' could not be found.
 
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Warning: Not deploying alerts on stage v1
-rw-r--r-- 1 tchiverton domain^users 15M Mar 17 11:13 .serverless/lone-star.zip

Environment information

See above

pgrzesik · 2022-03-17T11:40:12Z

Hello @tomchiverton - thanks for reporting. It seems like you're using a heavily outdated version of the Framework - do you have the option to check if that problem persists on recent versions as well?

tomchiverton · 2022-03-17T12:12:23Z

I've run sudo npm i - g serverless already, but just to be sure I ran it again and re-ran test under Node 16 / NPM 8:

$ sudo npm i -g serverless 
npm WARN deprecated querystring@0.2.1: The querystring API is considered Legacy. new code should use the URLSearchParams API instead.
npm WARN deprecated formidable@1.2.6: Please upgrade to latest, formidable@v2 or formidable@v3! Check these notes: https://bit.ly/2ZEqIau
npm WARN deprecated querystring@0.2.0: The querystring API is considered Legacy. new code should use the URLSearchParams API instead.
npm WARN deprecated querystring@0.2.0: The querystring API is considered Legacy. new code should use the URLSearchParams API instead.
npm WARN deprecated uuid@3.3.2: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.
npm WARN deprecated superagent@3.8.3: Please upgrade to v7.0.2+ of superagent.  We have fixed numerous issues with streams, form-data, attach(), filesystem errors not bubbling up (ENOENT on attach()), and all tests are now passing.  See the releases tab for more information at <https://github.com/visionmedia/superagent/releases>.

changed 390 packages, and audited 391 packages in 11s

52 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

$ node -v; npm version ; sls --version; sls package ; ls -laht .serverless/*zip
v16.14.0
{
...
Framework Core: 1.63.0
Plugin: 3.3.0
SDK: 2.3.0
Components Core: 1.1.2
Components CLI: 1.4.0
...
Serverless: Packaging service...
Serverless: Excluding development dependencies...
Serverless: Warning: Not deploying alerts on stage v1
-rw-r--r-- 1 tchiverton domain^users 49M Mar 17 12:12 .serverless/texas-lone-stars.zip

pgrzesik · 2022-03-17T12:14:19Z

Hello @tomchiverton - based on the output you're providing you're using 1.63.0 version of the Framework which is very old. Are you sure you don't have it installed either locally or as a standalone binary installation?

tomchiverton · 2022-03-17T12:18:58Z

I'm not really sure how to check that.

$ grep serverless package.json 
    "serverless-aws-static-file-handler": "tomchiverton/serverless-aws-static-file-handler#httpApi",
    "serverless-mysql": "^1.5.4",
    "serverless-plugin-aws-alerts": "1.6.1",
$ grep serverless package-lock.json 
    "serverless-aws-static-file-handler": {
      "version": "git+ssh://git@github.com/tomchiverton/serverless-aws-static-file-handler.git#29e49d3dffdf885b02f18c3e5fa1ed68b8b55bcb",
      "from": "serverless-aws-static-file-handler@tomchiverton/serverless-aws-static-file-handler#httpApi",
    "serverless-mysql": {
      "resolved": "https://registry.npmjs.org/serverless-mysql/-/serverless-mysql-1.5.4.tgz",
    "serverless-plugin-aws-alerts": {
      "resolved": "https://registry.npmjs.org/serverless-plugin-aws-alerts/-/serverless-plugin-aws-alerts-1.6.1.tgz",
$

pgrzesik · 2022-03-17T12:23:13Z

Based on the outputs it looks like it's using some kind of local installation - is there a chance that you have a node_modules somewhere in parent directory that has older serverless installed?

tomchiverton · 2022-03-17T12:35:31Z

I removed node_modules. re-ran "npm i" and again :

$ npm i
npm WARN old lockfile 
npm WARN old lockfile The package-lock.json file was created with an old version of npm,
npm WARN old lockfile so supplemental metadata must be fetched from the registry.
...
$ node -v; npm version ; sls --version; sls package ; ls -laht .serverless/*zip
v16.14.0
{
  'texas-lone-stars': '1.0.0',
  npm: '8.3.1',
  node: '16.14.0',
  v8: '9.4.146.24-node.20',
  uv: '1.43.0',
  zlib: '1.2.11',
  brotli: '1.0.9',
  ares: '1.18.1',
  modules: '93',
  nghttp2: '1.45.1',
  napi: '8',
  llhttp: '6.0.4',
  openssl: '1.1.1m+quic',
  cldr: '40.0',
  icu: '70.1',
  tz: '2021a3',
  unicode: '14.0',
  ngtcp2: '0.1.0-DEV',
  nghttp3: '0.1.0-DEV'
}
Running "serverless" from node_modules
Framework Core: 1.63.0
Plugin: 3.3.0
SDK: 2.3.0
Components Core: 1.1.2
Components CLI: 1.4.0
...
-rw-r--r-- 1 tchiverton domain^users 49M Mar 17 12:29 .serverless/texas-lone-stars.zip

There is a node_modules folder ../../../node_modules relative to my working folder that has a serverless folder in.

Renamed that node_modules, re-ran and now I get messages about needing to fix my SSM variable syntax. Interesting.

  - Cannot resolve variable at "provider.environment.JWT_TOKEN_v2": Parameter name: can't be prefixed with "ssm" (case-insensitive). If formed as a path, it can consist of sub-paths divided by slash symbol; each sub-path can be formed as a mix of letters, numbers and the following 3 symbols .-_,

fixed that up easily and

$ node -v; npm version ; sls --version; sls package --aws-profile serverless-dev-extravision  ; ls -laht .serverless/*zip
v16.14.0
{
  'texas-lone-stars': '1.0.0',
  npm: '8.3.1',
  node: '16.14.0',
  v8: '9.4.146.24-node.20',
  uv: '1.43.0',
  zlib: '1.2.11',
  brotli: '1.0.9',
  ares: '1.18.1',
  modules: '93',
  nghttp2: '1.45.1',
  napi: '8',
  llhttp: '6.0.4',
  openssl: '1.1.1m+quic',
  cldr: '40.0',
  icu: '70.1',
  tz: '2021a3',
  unicode: '14.0',
  ngtcp2: '0.1.0-DEV',
  nghttp3: '0.1.0-DEV'
}
Framework Core: 3.7.5
Plugin: 6.1.5
SDK: 4.3.2
...
-rw-r--r-- 1 tchiverton domain^users 17M Mar 17 12:34 .serverless/texas-lone-stars.zip

So why is Serverless looking up in parent folders ?!?

pgrzesik · 2022-03-17T12:41:08Z

Hey @tomchiverton - so it seems like the issue was solved for you?

The reason it's looking up in parent folders is that people sometimes have different configurations where they nest services inside a bigger projects - in that cases you can e.g. install serverless in root directory of the project and use it as a fallback for all the subprojects.

tomchiverton · 2022-04-08T13:45:39Z

No, it's not solved; As soon as we "npm i" a Serverless plugin - such as serverless-plugin-aws-alerts the whole @serverless / serverless folder is copied to node_modules in the deployment package.

Because the plugins specify only Serverless v2.x or v1.x, so it down grades the whole "sls" experience from the global install of v3.x to v2.x for this particular project. I understand this is by design :)

Bu I think this is still a bug in how dependencies are removed, because the plugin's have no runtime dependency on anything, so there's no need to bundle them and 40meg of their dependencies into the deployment .zip

tomchiverton · 2022-04-08T13:57:58Z

Ex

serverless create -u https://github.com/serverless/examples/tree/master/aws-node -n aws-node
cd aws-node
sls package 
ls -laht .serverless/aws-node.zip

1.5K

npm i serverless-aws-static-file-handler
sls package
ls -laht .serverless/aws-node.zip

47M

pgrzesik · 2022-04-08T14:01:46Z

Hey @tomchiverton - are you adding the plugins as dependencies or as devDependencies in package.json?

tomchiverton · 2022-04-08T14:10:01Z

https://github.com/activescott/serverless-aws-static-file-handler#install says to not have them as devDeps ?

pgrzesik · 2022-04-08T14:12:23Z

Thanks for sharing @tomchiverton - I personally don't see a valid reason for adding plugins as dependencies - that signifies to our logic that it's code that is supposed to be included in final zip, maybe that plugin is specific in some case but I'm pretty sure it's just an honest mistake from the authors - do the problem persist for you if you're installing it as dev deps?

tomchiverton · 2022-04-08T14:13:12Z

From a clean new project (as above)

npm install serverless-aws-static-file-handler --save-dev
sls package
ls -laht .serverless/aws-node.zip

132k

I'll have to test in a non-toy project to see if it still works.

pgrzesik · 2022-04-08T14:16:26Z

Thanks @tomchiverton - please let us know if the problem still persist on the other project, thanks in advance 🙇

tomchiverton · 2022-04-08T14:23:13Z

Doesn't work, because the usage of serverless-aws-static-file-handler is at run time as a require().

It's package.json appears to be correct : https://github.com/activescott/serverless-aws-static-file-handler/blob/main/package.json

Doesn't list anything too odd.

So I still don't understand why sls is packaging all the extra stuff in when it's installed (as non-dev, as intended and expected)

tomchiverton · 2022-04-08T14:26:20Z

npm ls -a snippit

├─┬ serverless-aws-static-file-handler@3.1.6
│ ├── lodash@4.17.21 deduped
│ ├─┬ mime-types@2.1.35
│ │ └── mime-db@1.52.0
│ ├── mustache@4.2.0
│ └─┬ serverless@2.72.3
│   ├─┬ @serverless/cli@1.6.0
│   │ ├─┬ @serverless/core@1.1.2

pgrzesik · 2022-04-08T14:35:20Z

So in this situation that's going to be the case - this plugin has a peer dependency on serverless which installs it on newer npm versions by default. As the plugin is a dependency, then it's peer dependencies are also dependencies, not dev dependencies. You can use --legacy-peer-deps flag of npm to not install serverless automatically due to it being configured as peer dependency.

tomchiverton · 2022-04-08T14:36:53Z

Could the plugin simply not declare any peer deps at all ?

pgrzesik · 2022-04-08T14:38:50Z

This is a question to plugin author, but it's a good practice to declare supported versions of Framework by peer dependencies. It's very unusual however, that the plugin has a part that is supposed to be required on runtime - 99.9% of plugins are required only as dev dependencies.

activescott · 2022-04-21T05:15:19Z

I am the author of the serverless-aws-static-file-handler plugin. The plugin doesn't import or access serverless at runtime technically (only expects standard lambda/APIG payloads), but it seemed like a good practice to be explicit. Being explicit may be causing more trouble than its worth though. @pgrzesik do you have a recommendation or best practice based on your experience with other serverless plugins?

pgrzesik · 2022-04-21T09:02:03Z

Hello @activescott - first of all, thanks a lot for your work on the serverless-aws-static-file-handler plugin 🙇 I don't know the specifics of this particular plugin, but is it plugin more of a helper library that is only used in Lambda functions' code or is it actually hooks into the plugin engine (adds a new command, hooks into lifecycle methods)? If it's only used as a library, it would make sense to not treat it as plugin and probably listing serverless as peer dependency is no longer needed in that case.

activescott · 2022-04-21T15:08:33Z

Thanks for your advice @pgrzesik! I’m this case it’s more appropriate to be a library so we’ll try removing the peer dependency.

pgrzesik · 2022-04-22T11:21:58Z

No problem @activescott - once again thanks a lot for being a part of community.

I'm going to close this issue.

…less as a peer dependency * Removes serverless as a peer dependency (see serverless/serverless#10863) * Deletes package-lock.json * fixes #139 Co-authored-by: Scott Willeke <scott@willeke.com>

simon1389 · 2022-12-23T09:57:50Z

Hey guys!
I am currently facing a similiar / the same issue as the creator of that issue..
The difference is, that i am not doing the deploy via command on my machine, but i have created a ci/cd pipeline on app.serverless.com which is doing that.

When i push a commit with the only change in the serverless.yml being runtime: nodejs16.x then i get the following output on the build-log:

> node --version
v16.15.0
> npm --version
8.5.5
> npx serverless --version
Framework Core: 3.26.0 (local)
Plugin: 6.2.2
SDK: 4.3.2
> serverless-build --info
Runtime: nodejs16.x
Build: 9172070
Runner: 0.7.6
Provider: GITHUB
.....
.....
Uploading service darts-live-v2-nestjs.zip file to S3 (183.37 MB)
....

and the deployment fails due to the zip being too large (Resource handler returned message: "Unzipped size must be smaller than 262144000 bytes)

If i then push a change, just setting runtime: nodejs14.x in the serverless.yml it deploys without a problem having this log:

Build started at Fri Dec 23 2022 09:54:14 GMT+0000 (GMT)
> node --version
v14.21.1
> npm --version
6.14.17
> npx serverless --version
Framework Core: 3.26.0 (local)
Plugin: 6.2.2
SDK: 4.3.2
> serverless-build --info
Runtime: nodejs14.x
Build: 845d8ab
Runner: 0.7.6
Provider: GITHUB
...
...
Uploading service darts-live-v2-nestjs.zip file to S3 (30.16 MB)
...

So it seems that there is really something going wrong?

tomchiverton · 2022-12-23T20:16:49Z

Grab the before and after .zip and compare.

What's your package.json? Does your ci/cd start from scratch each time eg package-lock and node_modules wiped?

Note that updating your runtime and not dependancies may be sub optimal anyway. What happens if you also bump all of them?

pgrzesik added the needs feedback label Mar 17, 2022

tomchiverton mentioned this issue Apr 8, 2022

Support Serverless v3 activescott/serverless-aws-static-file-handler#139

Closed

pgrzesik closed this as completed Apr 22, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

npm 8 / Node 16 produces deployments zip's too large to deploy (50meg), npm 6 / Node 12 fine (12meg) #10863

npm 8 / Node 16 produces deployments zip's too large to deploy (50meg), npm 6 / Node 12 fine (12meg) #10863

tomchiverton commented Mar 17, 2022

pgrzesik commented Mar 17, 2022

tomchiverton commented Mar 17, 2022

pgrzesik commented Mar 17, 2022

tomchiverton commented Mar 17, 2022

pgrzesik commented Mar 17, 2022

tomchiverton commented Mar 17, 2022

pgrzesik commented Mar 17, 2022

tomchiverton commented Apr 8, 2022

tomchiverton commented Apr 8, 2022

pgrzesik commented Apr 8, 2022

tomchiverton commented Apr 8, 2022

pgrzesik commented Apr 8, 2022

tomchiverton commented Apr 8, 2022

pgrzesik commented Apr 8, 2022

tomchiverton commented Apr 8, 2022

tomchiverton commented Apr 8, 2022

pgrzesik commented Apr 8, 2022

tomchiverton commented Apr 8, 2022

pgrzesik commented Apr 8, 2022

activescott commented Apr 21, 2022

pgrzesik commented Apr 21, 2022

activescott commented Apr 21, 2022

pgrzesik commented Apr 22, 2022

simon1389 commented Dec 23, 2022

tomchiverton commented Dec 23, 2022

npm 8 / Node 16 produces deployments zip's too large to deploy (50meg), npm 6 / Node 12 fine (12meg) #10863

npm 8 / Node 16 produces deployments zip's too large to deploy (50meg), npm 6 / Node 12 fine (12meg) #10863

Comments

tomchiverton commented Mar 17, 2022

Are you certain it's a bug?

Is the issue caused by a plugin?

Are you using the latest version?

Is there an existing issue for this?

Issue description

Service configuration (serverless.yml) content

Command name and used flags

Command output

Environment information

pgrzesik commented Mar 17, 2022

tomchiverton commented Mar 17, 2022

pgrzesik commented Mar 17, 2022

tomchiverton commented Mar 17, 2022

pgrzesik commented Mar 17, 2022

tomchiverton commented Mar 17, 2022

pgrzesik commented Mar 17, 2022

tomchiverton commented Apr 8, 2022

tomchiverton commented Apr 8, 2022

pgrzesik commented Apr 8, 2022

tomchiverton commented Apr 8, 2022

pgrzesik commented Apr 8, 2022

tomchiverton commented Apr 8, 2022

pgrzesik commented Apr 8, 2022

tomchiverton commented Apr 8, 2022

tomchiverton commented Apr 8, 2022

pgrzesik commented Apr 8, 2022

tomchiverton commented Apr 8, 2022

pgrzesik commented Apr 8, 2022

activescott commented Apr 21, 2022

pgrzesik commented Apr 21, 2022

activescott commented Apr 21, 2022

pgrzesik commented Apr 22, 2022

simon1389 commented Dec 23, 2022

tomchiverton commented Dec 23, 2022