Package: Individually is adding all node_modules to each zip file #441
Labels
Comments
|
Same issue here |
|
@wowi42 Can you post your dev/dependencies from your package.json as well as your webpack config? |
|
Update: I've just cleared a bunch of memory on my laptop and run this again. I had 11gb free, the packaging service (52 functions) got me down to 185mb during Here's the output from webpack: When I look in my .zip files, each zip is about 50mb, and I can see that each function has a node_modules folder that unzips to 183mb, with the following packages inside: The biggest dependancies in those node modules are:
So 133mb is just these 10 dependancies Any ideas @HyperBrain ? I'm at a bit of a loss here. I've also reverted to using the webpack config from the readme (with the addition of the nodeExternals() external) |
jvgeee
changed the title
|
I'll check and reply tomorrow. Sorry for the delay
jascination <notifications@github.com> schrieb am Di., 4. Sep. 2018, 2:51
nachm.:
… Update: I've just cleared a bunch of memory on my laptop and run this
again. I had 11gb free, the packaging service (52 functions) got me down to
185mb during sls package. Then when zipping it all up it freed up space
so the zips only take up ~2gb.
Here's the output from webpack:
Serverless: Invoke webpack:package
Serverless: Package lock found - Using locked versions
Serverless: Packing external modules: babel-runtime@^6.26.0, zone.js@^0.8.18, fast-xml-parser@^2.7.4, request@^2.85.0, bluebird@^3.4.6, joi@^10.6.0, uuid@^3.1.0, dynogels@^8.0.1, firebase-admin@^5.12.1, @slack/client@^4.3.1, sharp@^0.20.2, swagger-ui-dist@^3.17.1, haversine@^1.1.0
Serverless: Packaging service...
Serverless: Invoke aws:package:finalize
Serverless: Invoke aws:common:moveArtifactsToPackage
When I look in my .zip files, each zip is about 50mb, and I can see that
each function has a node_modules folder that unzips to 183mb, with the
following packages inside:
@firebase colour gauge isobject optjs sntp
@Google-Cloud combined-stream gcp-metadata isstream os-homedir sort-keys
@mrmlnc component-emitter gcs-resumable-upload isurl os-locale source-map
@nodelib compressible get-stream items os-tmpdir source-map-resolve
@protobufjs concat-map get-value jmespath osenv source-map-url
@sindresorhus concat-stream getpass joi p-cancelable split-array-stream
@slack configstore github-from-package jsbn p-defer split-string
@types console-control-strings glob json-buffer p-finally sshpk
abbrev copy-descriptor glob-parent json-schema p-is-promise static-extend
acorn core-js glob-to-regexp json-schema-traverse p-queue stream-events
acorn-es7-plugin core-util-is globby json-stringify-safe p-retry stream-shift
ajv create-error-class google-auth-library jsonwebtoken p-timeout strict-uri-encode
ansi-regex cryptiles google-auto-auth jsprim pascalcase string-format-obj
aproba crypto-browserify google-gax jwa path-dirname string-width
are-we-there-yet crypto-random-string google-p12-pem jws path-is-absolute string_decoder
arr-diff dashdash google-proto-files keyv path-type stringifier
arr-flatten debug got kind-of performance-now stringstream
arr-union decamelize graceful-fs lcid pify strip-ansi
array-filter decode-uri-component grpc lodash posix-character-classes strip-json-comments
array-union decompress-response gtoken lodash.includes power-assert stubs
array-uniq deep-equal har-schema lodash.isboolean power-assert-context-formatter tar
array-unique deep-extend har-validator lodash.isinteger power-assert-context-reducer-ast tar-fs
arrify define-properties has lodash.isnumber power-assert-context-traversal tar-stream
ascli define-property has-symbol-support-x lodash.isplainobject power-assert-formatter through2
asn1 delay has-to-string-tag-x lodash.isstring power-assert-renderer-assertion timed-out
assert-plus delayed-stream has-unicode lodash.once power-assert-renderer-base to-buffer
assign-symbols delegates has-value log-driver power-assert-renderer-comparison to-object-path
async detect-libc has-values loglevel power-assert-renderer-diagram to-regex
async-limiter diff-match-patch hash-stream-validation long power-assert-renderer-file to-regex-range
asynckit dir-glob hawk lowercase-keys power-assert-util-string-width topo
atob dot-prop hoek lru-cache prebuild-install tough-cookie
aws-sdk duplexer3 http-cache-semantics make-dir prepend-http traverse
aws-sign2 duplexify http-parser-js map-cache process-nextick-args tslib
aws4 dynogels http-signature map-visit protobufjs tunnel-agent
axios eastasianwidth iconv-lite merge2 pseudomap tweetnacl
babel-runtime ecc-jsbn ieee754 methmeth pump type-name
balanced-match ecdsa-sig-formatter ignore micromatch pumpify typedarray
base empower ignore-walk mime punycode union-value
base64-js empower-core imurmurhash mime-db qs unique-string
base64url end-of-stream indexof mime-types query-string universal-deep-strict-equal
bcrypt-pbkdf ent inflight mimic-response querystring unset-value
bl es-abstract inherits minimatch rc urix
bluebird es-to-primitive ini minimist readable-stream url
boom espurify into-stream minipass regenerator-runtime url-join
brace-expansion estraverse invert-kv minizlib regex-not url-parse-lax
braces eventemitter3 is mixin-deep repeat-element url-to-options
buffer expand-brackets is-accessor-descriptor mkdirp repeat-string use
buffer-alloc expand-template is-arrayish modelo request util-deprecate
buffer-alloc-unsafe extend is-buffer ms resolve-url uuid
buffer-equal-constant-time extend-shallow is-callable nan responselike verror
buffer-fill extglob is-data-descriptor nanomatch ret websocket-driver
buffer-from extsprintf is-date-object needle retry websocket-extensions
bun fast-deep-equal is-descriptor node-abi retry-axios which-pm-runs
bytebuffer fast-glob is-extendable node-forge retry-request wide-align
cache-base fast-json-stable-stringify is-extglob node-pre-gyp rimraf window-size
cacheable-request faye-websocket is-fullwidth-code-point noop-logger safe-buffer wrap-ansi
call-me-maybe fill-range is-glob nopt safe-regex wrappy
call-signature finity is-number normalize-url safer-buffer write-file-atomic
camelcase firebase-admin is-obj npm-bundled sax ws
capture-stack-trace follow-redirects is-object npm-packlist semver xdg-basedir
caseless for-in is-odd npmlog set-blocking xml2js
chownr foreach is-plain-obj number-is-nan set-value xmlbuilder
class-utils forever-agent is-plain-object oauth-sign sharp xtend
cliui form-data is-regex object-assign signal-exit y18n
clone-response fragment-cache is-retry-allowed object-copy simple-concat yallist
co from2 is-stream object-keys simple-get yargs
code-point-at fs-constants is-stream-ended object-visit simple-swizzle zone.js
collection-visit fs-copy-file-sync is-symbol object.entries slash
color fs-minipass is-typedarray object.getownpropertydescriptors snakeize
color-convert fs.realpath is-windows object.pick snapdragon
color-name function-bind isarray object.values snapdragon-node
color-string functional-red-black-tree isemail once snapdragon-util
The biggest dependancies in those node modules are:
- aws-sdk (23M)
- babel-runtime (8.2M)
- core-js (7.3M)
- dynogels (11M)
- firebase (4.2M)
- google-p12-pem (1.8M)
- google-proto-files (2.9M)
- grpc (46M)
- lodash (4.8M)
- sharp (24M)
Any ideas @HyperBrain <https://github.com/HyperBrain> ? I'm at a bit of a
loss here. I've also reverted to using the webpack config from the readme
(with the addition of the nodeExternals() external)
// Version with promises
// webpack.config.js
const webpack = require('webpack');
const slsw = require('serverless-webpack');
const BbPromise = require('bluebird');
const nodeExternals = require('webpack-node-externals');
module.exports = BbPromise.try(() => {
return slsw.lib.serverless.providers.aws.getAccountId()
.then(accountId => ({
entry: slsw.lib.entries,
plugins: [
new webpack.DefinePlugin({
AWS_ACCOUNT_ID: `${accountId}`,
}),
],
devtool: 'source-map',
// webpack-node-externals
// don't bundle built in modules like 'fs', 'path' etc.
target: 'node',
// webpack-node-externals
// Define 'externals' which won't be included in the bundle that webpack generates.
// We're excluding everything in node_modules.
externals: [nodeExternals()],
// Run babel on all .js files and skip those in node_modules
module: {
rules: [{
test: /\.js$/,
loader: 'babel-loader',
include: __dirname,
exclude: /node_modules/
}]
}
}));
});
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#441 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AFRM3n5jWV5v-y2OAC1J3Oy_HfERYLuYks5uXndigaJpZM4WT5dU>
.
|
|
Did you sort this out @jascination? |
|
I ran into this same issue with dynogels. It appears all of dynogels deps are being included regardless of having peers at the top level. |
|
Here's what worked for me: |
|
Are there any other workarounds for this? I'm considering having a seperate serverless application for each of my functions as that currently seems to be the only reliable way of using webpack with node externals whilst avoiding huge package sizes |
|
I have a similar issue without webpack. 10 times faster to not package them individually ... |
|
I originally had this problem with a separate project, but now working on a new one it doesn't seem to happen anymore. I think I might have update my serverless in the mean time (currently running 1.38.0) Relevant config as follows: custom:
webpack:
webpackConfig: ./webpack.config.js
includeModules:
forceExclude:
- aws-sdk
packager: 'yarn'
package:
individually: true
functions:
foo:
handler: handlers/foo.handle
bar:
handler: handlers/bar.handlewebpack.config.js const path = require('path');
const slsw = require('serverless-webpack');
const nodeExternals = require('webpack-node-externals');
module.exports = {
entry: slsw.lib.entries,
target: 'node',
mode: slsw.lib.webpack.isLocal ? 'development' : 'production',
optimization: {
minimize: false
},
devtool: 'nosources-source-map',
externals: [nodeExternals()],
module: {
rules: [
{
test: /\.js$/,
exclude: /node_modules/,
use: [
{
loader: 'babel-loader'
}
],
}
]
},
output: {
libraryTarget: 'commonjs2',
path: path.join(__dirname, '.webpack'),
filename: '[name].js',
sourceMapFilename: '[file].map'
}
};It might be worth noting that each function has a separate file for its handler. I would imagine that if you had two handlers in one file for two functions (handler.foo, handler.bar as an example) then webpack wouldn't know how to separate out the deps and shake those out. I'm also skeptical whether this will tree shake a dependency that isn't used in a required file. e.g. lib/mylib.js const _ = require('lodash');
const foo = () => {
return _.min(1,2);
}
const bar = () => {
return 1;
}
export { foo, bar }If you were to require this above file and only use the bar function as an example, then it might still include lodash in your bundle, but I haven't tested this personally. |
|
Hi @nzchicken, thanks for updating us, going to close the issue since it seems resolved 👍
I'm pretty sure Webpack handles that fine, but perhaps @HyperBrain can confirm? 😄
IIRC this whole situation can be avoided by either installing specific Lodash functions i.e. ( |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
This is a Bug Report
Description
Trying to do
sls packagewith yourpackage: individually: trueflag set. When I tried this, it took about 30mins (for around 30 functions within 8 services) and, and ate up about 5gb of my memory.I looked inside the
.webpack/folder, and within each function thenode_modulesfolder was about 500mb.Additional Data
``
None available, although here is my webpack config:
And here are my dependancies:
What's the best way to debug this? I can see that
aws-cliis getting included in all of these lambdas, but this is only about 20mb. I've seen the threads on force-excluding AWScli too, but I think my issue is a little more sever than that.Any ideas what I'm doing wrong?
The text was updated successfully, but these errors were encountered: