Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Request all dependency package owners to add .npmignore #71

Open
pilwon opened this issue Nov 18, 2013 · 1 comment
Open

Request all dependency package owners to add .npmignore #71

pilwon opened this issue Nov 18, 2013 · 1 comment

Comments

@pilwon
Copy link
Owner

pilwon commented Nov 18, 2013

It's taking too long to install all dependencies. It appears that most of dependency packages do not contain .npmignore or files specified in package.json to cherry pick what files gets blacklisted or whitelisted in the published npm packages. It will be nice if we inform each one of them about this technique and reduce the overall size and time to install all dependencies.

@pilwon
Copy link
Owner Author

pilwon commented Feb 28, 2014

All modules that has a real size > 1mb...

where real size = total module directory size - sum of its sub-module directories

Real Size (MB) Size (MB) Name Version Path
10.8832960129 49.9334955215 barbeque 0.2.1 node_modules/barbeque
9.21323013306 9.59646987915 phantomjs 1.9.7-1 node_modules/karma-phantomjs-launcher/node_modules/phantomjs
8.07612419128 9.8537569046 less 1.6.3 node_modules/barbeque/node_modules/less-middleware/node_modules/less
7.89259338379 7.89259338379 zeparser 0.0.5 node_modules/barbeque/node_modules/socket.io/node_modules/socket.io-client/node_modules/active-x-obfuscator/node_modules/zeparser
7.89257717133 7.89257717133 zeparser 0.0.5 node_modules/karma/node_modules/socket.io/node_modules/socket.io-client/node_modules/active-x-obfuscator/node_modules/zeparser
7.89257717133 7.89257717133 zeparser 0.0.5 node_modules/ultimate/node_modules/socket.io/node_modules/socket.io-client/node_modules/active-x-obfuscator/node_modules/zeparser
5.59165477753 7.38679122925 less 1.5.1 node_modules/grunt-contrib-less/node_modules/less
3.54654598236 3.54654598236 esprima 0.9.9 node_modules/barbeque/node_modules/browserify-middleware/node_modules/browserify/node_modules/syntax-error/node_modules/esprima
3.54654598236 3.54654598236 esprima 0.9.9 node_modules/grunt-browserify2/node_modules/browserify/node_modules/syntax-error/node_modules/esprima
3.413438797 3.5336265564 multiparty 2.2.0 node_modules/barbeque/node_modules/express/node_modules/connect/node_modules/multiparty
2.40671348572 2.40671348572 dojo 2.0.0-dev node_modules/intern/node_modules/dojo
2.35101032257 2.35101032257 moment 2.4.0 node_modules/moment
2.21811771393 2.24058628082 useragent 2.0.7 node_modules/karma/node_modules/useragent
1.86213493347 11.7248077393 socket.io-client 0.9.16 node_modules/ultimate/node_modules/socket.io/node_modules/socket.io-client
1.861992836 11.7363100052 socket.io-client 0.9.16 node_modules/barbeque/node_modules/socket.io/node_modules/socket.io-client
1.86197662354 11.7244939804 socket.io-client 0.9.16 node_modules/karma/node_modules/socket.io/node_modules/socket.io-client
1.70005321503 1.70005321503 selenium-webdriver 2.37.0 node_modules/protractor/node_modules/selenium-webdriver
1.30572986603 1.30572986603 uglify-js 1.2.5 node_modules/barbeque/node_modules/socket.io/node_modules/socket.io-client/node_modules/uglify-js
1.30572986603 1.30572986603 uglify-js 1.2.5 node_modules/karma/node_modules/socket.io/node_modules/socket.io-client/node_modules/uglify-js
1.30571365356 1.30571365356 uglify-js 1.2.5 node_modules/ultimate/node_modules/socket.io/node_modules/socket.io-client/node_modules/uglify-js
1.25746059418 1.25746059418 esprima 1.1.0-dev node_modules/rfileify/node_modules/falafel/node_modules/esprima
1.21668815613 1.21668815613 bson 0.1.8 node_modules/ultimate/node_modules/connect-mongo/node_modules/mongodb/node_modules/bson
1.20090389252 1.20090389252 lodash 1.3.1 node_modules/lodash
1.16223812103 1.16223812103 esprima 1.0.2 node_modules/barbeque/node_modules/browserify-middleware/node_modules/browserify/node_modules/insert-module-globals/node_modules/lexical-scope/node_modules/astw/node_modules/esprima
1.16223812103 1.16223812103 esprima 1.0.2 node_modules/barbeque/node_modules/browserify-middleware/node_modules/browserify/node_modules/module-deps/node_modules/detective/node_modules/esprima
1.16223812103 1.16223812103 esprima 1.0.2 node_modules/grunt-browserify2/node_modules/browserify/node_modules/insert-module-globals/node_modules/lexical-scope/node_modules/astw/node_modules/esprima
1.16223812103 1.16223812103 esprima 1.0.2 node_modules/grunt-browserify2/node_modules/browserify/node_modules/module-deps/node_modules/detective/node_modules/esprima
1.06831932068 1.06831932068 bson 0.2.5 node_modules/winston-mongodb/node_modules/mongodb/node_modules/bson
1.06285095215 1.06285095215 requirejs 2.1.11 node_modules/requirejs

Python script used:

#!/usr/bin/env python

import json
import os
from operator import itemgetter

ROOT_DIR = '<absolute-path-of-ultimate-seed-root-dir>'
DATA = []

def get_dir_size(dir):
    dir_size = 0
    for (path, dirs, files) in os.walk(dir):
        for file in files:
            filename = os.path.join(path, file)
            dir_size += os.path.getsize(filename)
    return dir_size / 1024.0 / 1024.0

for root, dirnames, filenames in os.walk(ROOT_DIR):
    if not root.endswith('node_modules'): continue
    for dirname in dirnames:
        module_name = dirname
        module_path = os.path.join(root, dirname)
        pkg_json_path = os.path.join(module_path, 'package.json')
        if dirname == '.bin': continue
        if not os.path.isfile(pkg_json_path): continue
        module_size = get_dir_size(module_path)
        submodules_size = get_dir_size(os.path.join(module_path, 'node_modules'))
        real_size = module_size - submodules_size
        pkg_json_file = open(pkg_json_path)
        pkg_json = json.load(pkg_json_file)
        pkg_json_file.close()
        if 'version' not in pkg_json: continue
        module_version = pkg_json['version']
        DATA.append((real_size, module_size, module_name, module_version, module_path))

print '| Real Size (MB) | Size (MB) | Name | Version | Path |'
print '| --------- | ---- | ---- | ------- | ---- |'
for row in sorted(DATA, key=itemgetter(0), reverse=True):
    print '| ',
    print row[0], ' | ',
    print row[1], ' | ',
    print row[2], ' | ',
    print row[3], ' | ',
    print row[4].replace(ROOT_DIR + '/', ''),
    print '|'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants