A CLI tool to lock all node_modules dependencies to a separate git repository.
Read a post why you may need it.
- Tracks changes in package.json file
- When a change is found makes a clean install of all dependencies and commits and pushes node_modules to a remote repository
- Works independently from your npm workflow and may be used on a CI server only keeping your dev environment simpler
How to use
sudo npm install -g npm-git-lock cd [your work directory] npm-git-lock --repo [email@example.com:your/dedicated/node_modules/git/repository.git] -v
If you don't want to depend on NPM connectivity when installing this module, you can install directly from github:
sudo npm install -g https://raw.githubusercontent.com/bestander/npm-git-lock/master/npm-git-lock-latest.tgz
Beware of possible breaking changes in the future, if you seek stability, obtain a link to a particular commit with the .tgz file on GitHub.
--verbose [-v] Print progress log messages --repo Git URL to repository with node_modules content [required] --cross-platform Run in cross-platform mode (npm 3 only) --incremental-install Keep previous modules instead of always performing a fresh npm install (npm 3 only) --production Runs npm install with production flag --check-all-json-elements Sha-1 calculated from all elements in package.json instead of only dependencies and devDependencies
npm-git-lock works with both npm 2 and 3, although the options
--incremental-install are only supported on npm 3.
Why you need it
You need it to get reliable and reproducible builds of your Node.js/io.js projects.
is the recommended option to "lock down" dependency tree of your application.
I have been using it throughout 2014 and there are too many inconveniences that accompany this technique:
- Dependency on npm servers availability at every CI build. NPM availability is quite good in 2015 watch a good talk but you don't want to have another moving part when doing an urgent production build.
- Managing of npm-shrinkwrap.json is not straightforward as of firstname.lastname@example.org. It is promising to improve though.
- Even though npm does not allow force updating packages without changing the version, packages can still be removed from the repository and you don't want to find that out when doing a production deployment.
- There are lots of other complex variable things about shrinkwrapping like optional dependencies and the coming changes in email@example.com like flat node_modules folder structure.
to your source version control system was recommended before shrinkwrapping but it is not anymore.
Nonetheless I think it is a more reliable option though with a few annoying details:
- A change in any dependency can generate a humongous commit diff which may get your Pull Requests unreadable
- node_modules often contains binary dependencies which are platform specific and large and don't play well across dev and CI environments
npm-git-lock is like committing your dependencies to git but without the above disadvantages.
How it works
The algorithm is simple:
- Check if node_modules folder is present in the working directory
- If node_modules exists check if there is a separate git repository in it
- Calculate sha1 hash from package.json in base64 format
- If remote repo from  has a commit tagged with sha1 from  then check it out clean, no
npm installis required
- Otherwise remove everything from node_modules (unless
--incremental-installis set, in which case only uncommitted changes will be stashed away), do a clean
npm install, commit, tag with sha1 from  and push to remote repo
- Next time you build with the same package.json, it is guaranteed that you get node_modules from the first run
After this you end up with a reliable and reproducible source controlled node_modules folder.
If there is any change in package.json, a fresh
npm install will be done once.
If there is no change, npm command is not touched and your CI build is fast.
npm-git-lock is run with the
--cross-platform option, it does not commit "platform-specific" build artifacts into the remote repository. Instead, it builds them using
npm rebuild when checking out the repository (step 4) or when doing a clean
npm install (step 5). Platform-specific files are taken to be those files that are generated by build scripts.
Inspired by this post, this is how step 5 is modified in cross-platform mode:
npm install --ignore-scriptsto prevent platform-specific compilation of any files.
git add .to capture the current "clean" cross-platform state.
npm rebuildto create any platform-specific files.
git status --untracked-files=allto list all files that have been generated in the previous step. Add these files to
--cross-platform is only supported on
npm version >= 3, since npm 2 doesn't run custom install scripts as it should during
npm rebuild (cf. this CI failure).
npm-git-lock will perform a completely fresh
npm install whenever there is any change to package.json (i.e., there is no commit in the node_modules repository tagged with the sha1 of package.json). However, that might not always be desired, since all dependencies might change (as long as their version is still within the range specified in package.json).
To get a behavior more similar to
npm shrinkwrap, you can use the option
--incremental-install. When installing modules, it will reuse modules that have already been committed to the node_modules repository and only run
npm install "on top of them".
A potential caveat is that modules are always fetched from the latest state (the master branch) of the node_modules repository. If there are dependencies introduced in a previous commit but not on the latest master HEAD, they will be freshly installed.
Example: In your project, you work on a branch A that declares a new dependency depA in package.json. In parallel, you have a branch B declaring a new dependency depB. Then you merge them both back into master. Assuming you run
npm-git-lock in each step, the history of your central node_modules repository will look like this (from recent to older):
- [commit3, master HEAD] depA + depB
- [commit2] depB
- [commit1] depA
Note that when running
npm-git-lock after the merge (to produce commit3), only depB was fetched from the previous state. For depA, a fresh install was performed.
With this package you get:
- Minimum dependency on npm servers availability for repeated builds which is very common for CI systems.
- No noise in your main project Pull Requests, all packages are committed to a separate git repository that does not need to be reviewed or maintained.
- If the separate git repository for packages gets too large and slows down your builds after a few years, you can just create a new one, saving the old one for patches if you need.
- Using it does not interfere with the recommended npm workflow, you can use it only on your CI system with no side effects for your dev environment or mix it with shrinkwrapping.
- You can have different node_modules repositories for different OS. Your CI is likely to be linux while your dev machines may be mac or windows. You can set up 3 repositories for them and use them independently.
- And it is blazing fast.
If you see this kind of error in your CI:
Cloning into 'node_modules'... done. *** Please tell me who you are. Run git config --global user.email "firstname.lastname@example.org" git config --global user.name "Your Name" to set your account's default identity. Omit --global to set the identity only in this repository. fatal: empty ident name (for <email@example.com>) not allowed
You need to configure
user.name in the environment as shown in the error message.
Just add those two commands before
Please give me your feedback and send Pull Requests.
Unit tests rely on
require(`child_process`).execSync command that works in node 0.11+.
Future plans (up for grabs)
- Replace .es6 extension with .js
- Switch to shelljs from promises API. Promises are still too heavy for such a file oriented CLI tool
3.5.0 - 2016-06-30
- Feature support --check-all-json-elements
3.3.0 - 2016-04-26
- Feature support --production
3.2.1 - 2016-04-14
- Fixed support for Node 0.12
3.2.0 - 2016-03-24
- Feature run
postinstallscripts even in
3.1.1 - 2016-03-17
loglevelargument for npm commands
3.0.0 - 2016-02-18
- The hashing algorithm has changed due to a bug that caused different hashes to be generated on different platforms. This means hashes generated by 3.0.0+ are not compatible with older versions. Make sure you use the same version in all your environments! (
git install -g firstname.lastname@example.org your friend)