Add support for npm5

[JamieMason]

npm install -g npm5 has a different cache structure so shrinkpack will need updating to support that, while being backwards compatible for older versions.

[zkat]

So, there's two ways you can go about this: The way I think I most recommend is to run npm pack pkg@1.2.3 pkg2@3.4.5 other-pkg@1 ..., according to the versions in npm-shrinkwrap.json or package-lock.json. This way is compatible with just about any version of npm I know of, and does not even require a warm cache to work: it'll use the cache if needed, but will otherwise make sure whatever you need to fetch gets fetched before it's copied to the cwd.

The other way, which will be only compatible with npm@>=5, is to use cacache in conjunction with the content addresses in package-lock.json or npm-shrinkwrap.json. That is, cacache.get.stream.byDigest('~/.npm/_cacache', locked.dependencies[pkg].integrity).pipe(fs.createWriteStream(locked.dependencies[pkg].name + '.tgz'). Obvs with whatever other stuff you wanna add to this. Note that doing it this way is not all that different from going the npm pack route, except this version allows you to use/verify integrity hashes when available.

lmk if you have any questions about the cache format or the tools around it. shrinkpack is real cool. :)

[JamieMason]

Thanks @zkat, I'll most likely go the npm pack route 👍

[timdp]

In the meantime, it might be helpful to pass {stdio: ['pipe', 'pipe', 'inherit']} when you spawn npm so stderr doesn't get swallowed. Anyway, I appreciate the effort on this. 👍

[JamieMason]

Thanks a lot for the tip @timdp

[JamieMason]

@zkat should I consider rewriting the paths in package-lock.json in the same way I do npm-shrinkwrap.json (from URLs to their local .tar path) or would you advise against that?

[zkat]

@JamieMason If you do that, it HAS to be: file:relative/path/to.tar, and you have to add the sha512sum of the .tar to integrity (NOT replace it -- it needs to have both sha512s in that string, separated by a space. See https://npm.im/ssri)

I haven't tried this myself yet, but I would hope it would work. It should. If not, that's a bug I gotta fix.

So, to summarize:

"foo": {
  "version": "1.2.3",
  "resolved": "https://registry.npmjs.org/mypkg/-/mypkg-1.2.3.tgz",
  "integrity": "sha512-deadbeef"
}

becomes

"foo": {
  "version": "1.2.3",
  "resolved": "file:package.mirror/mypkg-1.2.3.tar",
  "integrity": "sha512-deadbeef sha512-c0ffee"
}

If the integrity hash doesn't match resolved, you'll get an integrity error.

[pietermees]

@JamieMason just wondering, any progress on this? Thanks!

[JamieMason]

Hey @zkat,
I've noticed a few mentions from npm peeps that there's a plan to ~implement shrinkpack in npm, is that right? If so, would there be any need to add npm v5 support to shrinkpack if it will be available without a 3rd party package anyway?

Thanks a lot,

Jamie

[zkat]

@JamieMason doing that is a ways away. You're free to implement it shrinkpack-side in he meantime -- it's gone pretty far down the queue in the past couple of months, and idk when we'd even get around to it, rn.

[JamieMason]

righto, thanks a lot @zkat

[cmandlbaur]

Is this something that is in progress?

[JamieMason]

Hey again @zkat, hope you like @ mentions.

If I wanted to keep the ↓ part of the +110 -0 ↓50 →60 ✓0 summary shown after shrinkpack has run, am I right that for npm >= 5 cacache.get.hasContent(cache, integrity) is the method I would want to use?

I'm thinking I would use that to report on whether the package was downloaded, but run npm pack to do the ✨🦄 ✨.

@pietermees @cmandlbaur It hasn't been possible to work on shrinkpack up until now, from next week I will be able to hopefully make a start.

If anyone knows of anything I can do to help make it easier for people to contribute, please let me know. I've spent a lot of time on the readme.md and contributing.md etc but there's probably all kinds of stuff that should be clear to people but really isn't.

[pietermees]

That's great news @JamieMason!

I'm not entirely sure what would make contributing easier. I think there's a general sense (at least on my end) that I don't fully understand all the moving parts. And seeing that even you - after all this amazing work - still seem to rely on zkat's knowledge from time to time, makes me feel like the learning curve is relatively long and steep ;)

On the one hand, having a number of smaller tasks that people can tackle to learn seems like a good idea, on the other hand people probably run into specific issues from time to time that they care about a lot (like this one) and are very happy the rest of the time. So not many people might pick up the smaller tasks.

[zkat]

@JamieMason yeah. Although note that the cacache directory is inside ~/.npm/_cacache, when you point it at a cache

[JamieMason]

Thanks again 👊

[JamieMason]

Sorry @zkat, can I nag you again please?

1

I've noticed that sometimes entries have a semver string for version and a location for resolved
and other times a location for version and no resolved field, like so;

EDIT: package-lock.json on its own looks as expected (I'm on npm 5.3.0) but the following is seen if the user runs npm shrinkwrap --dev over it.

// familar, with version semver and resolved location
"enhanced-resolve": {
  "version": "3.4.1",
  "resolved": "https://registry.npmjs.org/enhanced-resolve/-/enhanced-resolve-3.4.1.tgz",
  "integrity": "sha1-BCHjOf1xQZs9oT0Smzl5BAIwR24=",
  "dev": true,
  "requires": {
    "graceful-fs": "https://registry.npmjs.org/graceful-fs/-/graceful-fs-4.1.11.tgz",
    "memory-fs": "0.4.1",
    "object-assign": "https://registry.npmjs.org/object-assign/-/object-assign-4.1.1.tgz",
    "tapable": "0.2.8"
  }
},
// new, with location version and no resolved
"ent": {
  "version": "https://registry.npmjs.org/ent/-/ent-2.2.0.tgz",
  "integrity": "sha1-6WQhkyWiHQX0RGai9obtbOX13R0=",
  "dev": true
},

When running npm pack, should I eg. npm pack ${node.resolved || node.version}? Can you tell me
anything more about the rules behind what's happening here?

2

When I come to update the lockfile, what should the resulting version be in these cases? eg.

// input
"ent": {
  "version": "https://registry.npmjs.org/ent/-/ent-2.2.0.tgz",
  "integrity": "sha1-6WQhkyWiHQX0RGai9obtbOX13R0=",
  "dev": true
},

Would npm be able to resolve the dependency if that became this?

// possible output 1
"ent": {
  "version": "file:node_shrinkwrap/some-file.tar",
  "integrity": "sha1-6WQhkyWiHQX0RGai9obtbOX13R0= some-new-sha-for-the-local-file",
  "dev": true
},

Or should it maybe become this?

// possible output 2
"ent": {
  "version": "https://registry.npmjs.org/ent/-/ent-2.2.0.tgz",
  "resolved": "file:node_shrinkwrap/some-file.tar",
  "integrity": "sha1-6WQhkyWiHQX0RGai9obtbOX13R0= some-new-sha-for-the-local-file",
  "dev": true
},

3

Can you point me to anything you'd recommend I should use to generate the sha512s please? also,
what should I feed it? the contents of the new tarball? and/or the location? This isn't something
I've done before.

4

EDIT: Following on from question 1, I wonder if I'll be able to maintain the file names of node_shrinkwrap/<name>-<semver>.tgz without opening the packages?

Sorry this is a huge list, thanks for all your help.

[zkat]

1. See the spec for package-lock's version field here: https://github.com/npm/npm/blob/latest/doc/spec/package-lock.md#version-changed. Also, you should keep requires in there.

2. Thinking about this makes me wonder if npm5 can actually do this sort of thing right now. URL specs don't match semver specs, so npm would (I believe) immediately NOPE out and force a full reinstallation because the package-lock would not be considered to match the package.json. I think the right thing here is to keep the original version field value and change resolved, but I think npm insists on resolved being web URLs right now. I really just don't know and you'll have to experiment here and maybe help patch npm to make it support this.

3. https://npm.im/ssri is a full toolset for working with SRI strings, and is what npm itself uses

4. I don't really understand this or why it's a problem.

[JamieMason]

Thanks a lot

[rockerest]

Just wanted to leave a note that I'm pleased that you've found time to work on this.

Like @pietermees:

there's a general sense (at least on my end) that I don't fully understand all the moving parts. And seeing that even you - after all this amazing work - still seem to rely on zkat's knowledge from time to time, makes me feel like the learning curve is relatively long and steep ;)

So I feel indebted to you for the work already done, and for the work you've now picked up again.

I'm cheering for you, and I deeply appreciate your time and energy on this project. 🍻 🎉

[pietermees]

I second that!

[JamieMason]

That's amazing, thanks so much. Just a note to people dropping in that I'm travelling now so won't be able to pick up again until mid-late next week.

😊👍

[JamieMason]

Some learnings today, I've created 4 branches which show different structures of npm-shrinkwrap.json and package.json.

• format 1 ❌
• format 2 ❌
• format 3 ✅
• format 4 ❌

Possible things to try next are updating the requires fields as follows...

"react": {
  "version": "file:node_shrinkwrap/react-15.6.1.tgz",
  "integrity": "sha1-uqhDTsZ4C96ZfNw4C3nNM7ljk98=",
  "requires": {
    "create-react-class": "file:node_shrinkwrap/create-react-class-15.6.0.tgz",
    "fbjs": "file:node_shrinkwrap/fbjs-0.8.14.tgz",
    "loose-envify": "file:node_shrinkwrap/loose-envify-1.3.1.tgz",
    "object-assign": "file:node_shrinkwrap/object-assign-4.1.1.tgz",
    "prop-types": "file:node_shrinkwrap/prop-types-15.5.10.tgz"
  }
}

...as that is what was produced when running the following to see how npm itself would generate a localised npm-shrinkwrap.json;

npm install --save ./node_shrinkwrap/angular-1.6.5.tgz
npm install --save ./node_shrinkwrap/asap-2.0.6.tgz
npm install --save ./node_shrinkwrap/core-js-1.2.7.tgz
npm install --save ./node_shrinkwrap/create-react-class-15.6.0.tgz
npm install --save ./node_shrinkwrap/encoding-0.1.12.tgz
npm install --save ./node_shrinkwrap/fbjs-0.8.14.tgz
npm install --save ./node_shrinkwrap/iconv-lite-0.4.18.tgz
npm install --save ./node_shrinkwrap/is-stream-1.1.0.tgz
npm install --save ./node_shrinkwrap/isomorphic-fetch-2.2.1.tgz
npm install --save ./node_shrinkwrap/js-tokens-3.0.2.tgz
npm install --save ./node_shrinkwrap/lodash-4.17.4.tgz
npm install --save ./node_shrinkwrap/lodash.assign-4.2.0.tgz
npm install --save ./node_shrinkwrap/loose-envify-1.3.1.tgz
npm install --save ./node_shrinkwrap/node-fetch-1.7.2.tgz
npm install --save ./node_shrinkwrap/object-assign-4.1.1.tgz
npm install --save ./node_shrinkwrap/promise-7.3.1.tgz
npm install --save ./node_shrinkwrap/prop-types-15.5.10.tgz
npm install --save ./node_shrinkwrap/react-15.6.1.tgz
npm install --save ./node_shrinkwrap/setimmediate-1.0.5.tgz
npm install --save ./node_shrinkwrap/ua-parser-js-0.7.14.tgz
npm install --save ./node_shrinkwrap/whatwg-fetch-2.0.3.tgz
npm shrinkwrap --dev

that produced format 3 which is the only format which will let you install while offline and with an empty npm/cacache cache. But it has the following in package.json which will not be acceptable;

  "dependencies": {
    "angular": "file:node_shrinkwrap/angular-1.6.5",
    "asap": "file:node_shrinkwrap/asap-2.0.6",
    "core-js": "file:node_shrinkwrap/core-js-1.2.7",
    "create-react-class": "file:node_shrinkwrap/create-react-class-15.6.0",
    "encoding": "file:node_shrinkwrap/encoding-0.1.12",
    "fbjs": "file:node_shrinkwrap/fbjs-0.8.14",
    "iconv-lite": "file:node_shrinkwrap/iconv-lite-0.4.18",
    "is-stream": "file:node_shrinkwrap/is-stream-1.1.0",
    "isomorphic-fetch": "file:node_shrinkwrap/isomorphic-fetch-2.2.1",
    "js-tokens": "file:node_shrinkwrap/js-tokens-3.0.2",
    "lodash": "file:node_shrinkwrap/lodash-4.17.4",
    "lodash.assign": "file:node_shrinkwrap/lodash.assign-4.2.0",
    "loose-envify": "file:node_shrinkwrap/loose-envify-1.3.1",
    "node-fetch": "file:node_shrinkwrap/node-fetch-1.7.2",
    "object-assign": "file:node_shrinkwrap/object-assign-4.1.1",
    "promise": "file:node_shrinkwrap/promise-7.3.1",
    "prop-types": "file:node_shrinkwrap/prop-types-15.5.10",
    "react": "file:node_shrinkwrap/react-15.6.1",
    "setimmediate": "file:node_shrinkwrap/setimmediate-1.0.5",
    "ua-parser-js": "file:node_shrinkwrap/ua-parser-js-0.7.14",
    "whatwg-fetch": "file:node_shrinkwrap/whatwg-fetch-2.0.3"
  }

• format 4 is format 3 but with the package.json put back to a more typical format;

  "dependencies": {
    "angular": "1.6.5",
    "asap": "2.0.6",
    "core-js": "1.2.7",
    "create-react-class": "15.6.0",
    "encoding": "0.1.12",
    "fbjs": "0.8.14",
    "iconv-lite": "0.4.18",
    "is-stream": "1.1.0",
    "isomorphic-fetch": "2.2.1",
    "js-tokens": "3.0.2",
    "lodash": "4.17.4",
    "lodash.assign": "4.2.0",
    "loose-envify": "1.3.1",
    "node-fetch": "1.7.2",
    "object-assign": "4.1.1",
    "promise": "7.3.1",
    "prop-types": "15.5.10",
    "react": "15.6.1",
    "setimmediate": "1.0.5",
    "ua-parser-js": "0.7.14",
    "whatwg-fetch": "2.0.3"
  }

Strangely though, this version does not work.

I'll post more as I find it, thanks all.

[rockerest]

Thanks for the continuing investigation and updates!

[JamieMason]

Out of time for today but I have this branch https://github.com/JamieMason/shrinkpack/tree/npm5 which is a thin wrapper for my-precious. All seems to work as expected but I've yet to get the e2e tests passing. Once I can figure this out I'll have more confidence to release it.

EDIT: Apologies for the email spam subscribers probably got thanks to my repeated pushes to Travis ☝️

[peteward44]

Any progress on this? I don't mind having a look if you're not able to work on it

[JamieMason]

Thanks @peteward44 I appreciate the offer.

I have to grudgingly admit that this issue has worn me down and by now I really struggle to find the energy and motivation to get it through. Kat and Rebecca from npm have been amazing and we couldn't ask any more from them, but in two days this issue will be a year old. That's a long slog for one person and over that time each fix has uncovered new issues which took time to resolve before continuing on again.

The under-development version of shrinkpack works, at least on paper, the overall ongoing issue has been around gaining support in npm >=5 for fully offline installations using only a lockfile containing file: paths, as was the case in npm 3 and 4. Again, the npm devs are swamped and it's no simple task.

There is some good news though that npm have announced they will be:

Integrating the functionality of shrinkpack into npm directly. This would allow for entirely offline deploys even without a cache. It will also let you deploy with git dependencies without having git installed on your production machines.
– https://blog.npmjs.org/post/173239798780/beyond-npm6-the-future-of-the-npm-cli

In light of this, I think it would be better to wait until npm archive is released in npm@7. No third party tooling (shrinkpack) will be needed and it's also ridiculously fast compared to shrinkpack.

Sorry everyone for such a long wait, but as a result we will get npm archive so it will be worth it.

[peteward44]

OK fair enough. A shame as shrinkpack is a great tool, but native support for offline installs does make a lot more sense.

[mobidev111]

npm taking over is definitely good - giving it a future as first class citizen.

This probably only arrives in half a year or so. In the blog post they state "late summer and fall of 2018" and given they're loaded with work one could guess that the timeline for this feature slips further.

I would welcome a quicker (interim) solution - and it seems we are already very close with shrinkpack working at the core.

For folks who want to help and take over your excellent work to get it finally out of the door - what would be the next steps to be taken?

To me, the next steps look like:

• set up & fix tests in the npm5 branch - any new ones?
• Report the progress on tests, so that others can chime on problems
• Identify potentially further problems in npm
• create a pre-release version to gather feedback from users

[zkat]

@mobidev111 the native npm version is pretty much already implemented and mostly needs attention and polish. You can try it right now with $ npx npmc archive. Regular npm install as of npm@6 should be able to install off that archive. If you want to add/remove packages, though, you'll need to use $ npx npmc add <pkg> and $ npx npmc rm <pkg> so edits to package-lock.json happen correctly.

[zkat]

The reason it isn't shipped is because I need to take it over the finish line, make sure corner cases are covered, add tests, etc -- but I won't have time allocated to do this for the next couple of months, probably.

[zkat]

(p.s. $ npx npmc unarchive will return your repo to its original pre-archive format)

[peteward44]

Great news @zkat. I'm not brave enough to use this in my production tool chain just yet, but i look forward to the official release 👍 We've been stuck on npm 4 due to this for a long time now and really can't wait to upgrade

[JamieMason]

I have published npm install -g shrinkpack@next which contains the current npm5 branch as 1.0.0-alpha: https://github.com/JamieMason/shrinkpack/blob/npm5/CHANGELOG.md.

It is a thin wrapper for my-precious rather than the ongoing work I did in the dev branch, I expect there may be issues but please let us know how it goes for you 👍

Thanks all for your comments, @mobidev111 thanks a lot for that useful comment and organisation.

[behoyh]

Hi @JamieMason,

Thanks for all your work, I know firsthand how building an open source project can be disorienting and tiring, and I just wanted to say that your efforts are very much appreciated and needed!

Congrats on getting your code into npm archive!

[BogdanDumitruMarian]

Hi. @JamieMason
I'm getting the infamous

[ERROR] npm WARN optional SKIPPING OPTIONAL DEPENDENCY: aggregate error
[ERROR]
[ERROR] npm ERR! aggregate error

when running npm install

I'm using node v9.11.1 / npm 5.6.0 and shrinkpack 1 - alpha.
Funky enough, it only happens on the Jenkins Linux machine.
I've tried running npm i against the archived-packages locally, with the internet connection turned off, and no problem whatsoever occured.

[themoonrat]

npm/cli#1 got closed down :(
Looks like yarn is the only solution, other than staying with vastly slower npm v4

[zkat]

@themoonrat we're looking into ways to reintegrate this and see where it fits into our product plans for the CLI. In the meantime, the core support for this kind of thing did land in npm. Any tool that does the same kind of rewriting should provide similar support. This means you can use $ npx my-precious or shrinkpack can be updated to do things with libprecious itself, and it should be way faster, and npm will do installations off the local tarballs very very quickly. :) I'm also still around to help integrate things if shrinkpack is interested in moving forward with this while we figure out how and whether to land this in npm proper.

[thomashardwick]

So it's been while.
I'm still using shrinkpack and npm@4 since it still seems to be the only way to do fully offline installs without zipping up the node_modules folder but it is steadily becoming problematic to be on such an old version.
Has there been any movement on getting npm able to install on an offline air-gapped machine or with shrinkpack using newer versions of npm?

[JamieMason]

Hey @thomashardwick, check out #83 (comment) as I think it's still about right as a summary of where shrinkpack is at. Quite some time has passed though and I've lost touch with what npm provides out of the box in this area, there was talk of an npm archive command but again I don't know the latest.

[rockerest]

@thomashardwick I know this is a very different solution, but have you tried something like https://www.snowpack.dev/ or my own much simpler (better, in my opinion) https://www.npmjs.com/package/@thomasrandolph/icepack?

In both cases, instead of saving a local copy of the full dependency tree, those tools build a bundled copy of the appropriate dependencies. You would do that once, while you have an internet connection. Then, all future references to the dependency just reference the bundled file. One key part of both of these tools is the bundled version is always browser-safe, too, so you can run either tool ahead of time and then you have a set of dependencies that you never have to touch again, including compiling (if you ship ES6 modules to the browser).

Again, I know this is a really big deviation from shrinkpack, but I've found it's been extremely helpful to distance myself as far as possible from the npm ecosystem.

[peteward44]

I've since changed to use Yarn and it's offline mirror, which works pretty much like shrinkpack does.

https://classic.yarnpkg.com/blog/2016/11/24/offline-mirror/

It would seem NPM has lost interest in allowing users to offline dependencies as that goes against it's business model. A shame

[JamieMason]

Keep an eye on yarnpkg/yarn#541 @peteward44 as Yarn's offline mirror can gradually harm the performance of your Git repo over time.

[iarna]

I've got no idea what npm's plans are these days, but they do have an active rfc process going, and it may be useful to bring this up there again.

[JamieMason]

shrinkpack@0.19.0 has been released which adds support for npm 7 and up. Thanks again to Kat and Rebecca for all of your help during your time at npm ❤️ .

[behoyh]

This is amazing