An experiment in bringing the
go mod vendor
experience to thenpm
ecosystem. See the blog post.
$ go install github.com/hardfinhq/npm-mod/cmd/npm-mod@v1.20220526.1
For the purposes of demonstration, we'll be running subcommands in an
application (called sample
) created with the create-react-app
tool.
Running the tidy
subcommand we see a new file and two changed files:
$ npm-mod tidy
$ git status
On branch main
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git restore <file>..." to discard changes in working directory)
modified: package-lock.json
modified: package.json
Untracked files:
(use "git add <file>..." to include in what will be committed)
.npm-mod.tidy.json
no changes added to commit (use "git add" and/or "git commit -a")
Digging in a bit to see what's changed. The .npm-mod.tidy.json
captures
the contents of the modified files so we can revert easily if need be and
also tracks the URLs and file integrity for all packages referenced:
$ cat .npm-mod.tidy.json
{
"version": "22.05",
"package.json": "ewogICJuYW1lIjogInNhbXBsZSIsCiAg...",
"package-lock.json": "ewogICJuYW1lIjogInNhbXBsZSIsCiAg...",
"packages": [
{
"url": "https://registry.npmjs.org/@ampproject/remapping/-/remapping-2.1.2.tgz",
"algorithm": "sha512",
"hash": "hoyByceqwKirw7w3Z7gnIIZC3Wx3J484Y3L/cMpXFbr7d9ZQj2mODrirNzcJa+SM3UlpWXYvKV4RlRpFXlWgXg=="
},
{
"url": "https://registry.npmjs.org/@apideck/better-ajv-errors/-/better-ajv-errors-0.3.3.tgz",
"algorithm": "sha512",
"hash": "9o+HO2MbJhJHjDYZaDxJmSDckvDpiuItEsrIShV0DXeCshXWRHhqYyU/PKHMkuClOmFnZhRd6wzv4vpDu/dRKg=="
},
...
The package.json
has had every semver version range swapped for an explicit
local file reference:
diff --git a/package.json b/package.json
index 82115d2..753457c 100644
--- a/package.json
+++ b/package.json
@@ -3,13 +3,13 @@
"version": "0.0.1",
"private": true,
"dependencies": {
- "@testing-library/jest-dom": "^5.16.4",
- "@testing-library/react": "^13.1.1",
- "@testing-library/user-event": "^13.5.0",
- "react": "^18.0.0",
- "react-dom": "^18.0.0",
- "react-scripts": "5.0.1",
- "web-vitals": "^2.1.4"
+ "@testing-library/jest-dom": "file:vendor/testing-library__jest-dom-5.16.4.tgz",
+ "@testing-library/react": "file:vendor/testing-library__react-13.1.1.tgz",
+ "@testing-library/user-event": "file:vendor/testing-library__user-event-13.5.0.tgz",
+ "react": "file:vendor/react-18.0.0.tgz",
+ "react-dom": "file:vendor/react-dom-18.0.0.tgz",
+ "react-scripts": "file:vendor/react-scripts-5.0.1.tgz",
+ "web-vitals": "file:vendor/web-vitals-2.1.4.tgz"
},
"scripts": {
"start": "react-scripts start",
and the package-lock.json
has had a similar transformation, this time
swapping URLs for local file references:
diff --git a/package-lock.json b/package-lock.json
index 5303979..02c30e9 100644
--- a/package-lock.json
+++ b/package-lock.json
@@ -18,8 +18,8 @@
}
},
"node_modules/@ampproject/remapping": {
- "version": "2.1.2",
- "resolved": "https://registry.npmjs.org/@ampproject/remapping/-/remapping-2.1.2.tgz",
+ "version": "file:vendor/ampproject__remapping-2.1.2.tgz",
+ "resolved": "file:vendor/ampproject__remapping-2.1.2.tgz",
"integrity": "sha512-hoyByceqwKirw7w3Z7gnIIZC3Wx3J484Y3L/cMpXFbr7d9ZQj2mODrirNzcJa+SM3UlpWXYvKV4RlRpFXlWgXg==",
"dependencies": {
"@jridgewell/trace-mapping": "^0.3.0"
@@ -29,8 +29,8 @@
...
Just checking in the changes from npm-mod tidy
is insufficient; the
application will be broken because all of the file:vendor/...
references
don't point anywhere. To fix this, the npm-mod vendor
subcommand will
download all packages referenced in .npm-mod.tidy.json
:
$ npm-mod vendor
Saved babel__helper-builder-binary-assignment-operator-visitor-7.16.7.tgz
Saved babel__generator-7.17.9.tgz
...
Saved yargs-16.2.0.tgz
Saved yocto-queue-0.1.0.tgz
$
$
$ git status
On branch main
Untracked files:
(use "git add <file>..." to include in what will be committed)
vendor/
nothing added to commit but untracked files present (use "git add" to track)
$
$
$ ls -1 vendor/
abab-2.0.6.tgz
accepts-1.3.8.tgz
...
yargs-parser-20.2.9.tgz
yocto-queue-0.1.0.tgz
$ ls -1 vendor/ | wc -l
1135
Using the package.json
and package-lock.json
fields in .npm-mod.tidy.json
the unvendor
can restore the original semver ranges. This would enable a
workflow using the npm-mod
that can switch from "native npm
" mode back to
"vendor-style packages checked into source tree". With this workflow, updating
a dependency could be done as follows:
- Run
npm-mod unvendor
subcommand to "go back" to standard form forpackage.json
- Modify
package.json
semver ranges to prepare for any new packages or package version updates needed - Delete any or all of
vendor/
,node_modules/
,package-lock.json
and.npm-mod.tidy.json
to ensure all desired package updates happen - Run
npm-mod tidy
again to switch back tofile:vendor/...
references - Run
npm-mod unvendor
again to download newly added or changed packages
The primary goal of this project is to enable an experiment in npm
packaging. As a result, the CLI may have incomplete support for all of the
various shapes a package-lock.json
file can exhibit.
Some things we explicitly don't support, but may choose to expand support for over time:
- Providing a
yarn-mod
equivalent (orpnpm-mod
for that matter) - Support
lockfileVersion=1
for thenpm
package lock format (onlylockfileVersion=2
is supported)
This tool is not intended to improve performance in the steady state case.
The performance boost from "no network required" is roughly equivalent to
the performance boost from using ~/.npm/_cacache
.
Starting with the ~/.npm/_cacache
directory empty, installing the
packages needed for the basic create-react-app
sample takes around 10.5
seconds:
$ rm -fr ~/.npm/_cacache/
$ time npm ci
...
added 1393 packages, and audited 1394 packages in 10s
...
real 0m10.370s
user 0m13.952s
sys 0m6.959s
On a second run, the global npm
cache reduces the time to around 4.5 seconds:
$ rm -fr ./node_modules/
$ time npm ci
...
added 1393 packages, and audited 1394 packages in 4s
...
real 0m4.318s
user 0m4.809s
sys 0m5.909s
After running npm-mod tidy
and vendor
, the install time is roughly
the same:
$ rm -fr ./node_modules/
$ npm-mod tidy
$ npm-mod vendor
$ time npm ci
...
added 1393 packages, and audited 1394 packages in 4s
...
real 0m4.456s
user 0m5.112s
sys 0m5.802s