Canonicalize npm-shrinkwrap.json file contents for easier diffing

[derekcicerone-zz]

The shrinkwrap.json files currently seem to output modules somewhat randomly. It would be nice to have them canonicalized somehow (perhaps just by sorting the modules alphabetically by name) so that the file can be easily diffed to prior versions when the dependencies are updated.

[domenic]

In my experience the order stays the same; the issues come when upgrading between versions of npm, all of which seem to store slightly different metadata, thus making diffs generated between versions very differently.

Has your experience been that the order, specifically, is rearranged?

[derekcicerone-zz]

I've seen reordering but perhaps my process for updating is wrong. When we want to update all of our dependencies to the latest versions I delete the node_modules directory and npm-shrinkwrap.json, then run "npm install" followed by "npm shrinkwrap". The resulting shrinkwrap file is then diffed and checked in.

[domenic]

Hmm OK, that sounds reasonable. I guess I just haven't seen that yet. In which case, +1, this would make a lot of sense.

[isaacs]

@derekcicerone What was the before and after in this case?

[derekcicerone-zz]

The before and after? I'm not sure I understand your question. The shrinkwrap file lists out a bunch of modules and the ordering of the modules was significantly different when I last updated the file so comparing them wasn't very useful. If the modules were ordered by name, then the diff would likely be much more useful.

[domenic]

This is probably related to the fact that the shinkwrap file uses a hash instead of an array, and key order in hashes is not guaranteed in JavaScript (in particularly not guaranteed in JSON.stringify).

So this would kind of be a big change, requiring revamping the shrinkwrap format to use arrays instead of hashes :(.

[derekcicerone-zz]

I see, that makes sense - its a limitation of the JSON stringification method being used. Sounds like its related to this: http://stackoverflow.com/questions/8931967/is-there-a-deterministic-equivalent-of-json-stringify

[edef1c]

@domenic In practice, insertion order is guaranteed.
Something along the lines of var newObj = {}; Object.keys(obj).sort().forEach(function(key) { newObj[key] = obj[key] }) would work.

[OliverJAsh]

I don't seem to have an issue with ordering, but the metadata does appear to change every time someone on the project runs shrinkwrap, making it very difficult to read the diff. This time it was from changing from "serve-static@1.2.3" to "https://registry.npmjs.org/serve-static/-/serve-static-1.2.3.tgz".

[pseidemann]

my npm version is 2.15.1 with node 4.4.2 and if I just run npm shrinkwrap twice without changing anything, the file will be different every time.
it seems like the order of the packages is random and need some clear sorting. this makes the whole feature almost useless because it's impossible to follow changes properly in git.
would it be possible to implement some node script which will sanitize the json file after shrinkwrap has run?

[othiym23]

As of the most recent version of npm@3 (going into npm@next today), we should have eliminated the last source of nondeterminism in npm-shrinkwrap.json across runs. We probably won't ever backport this feature to npm@2, because it's merely churn in a generated file and not an actual bug or security flaw; at this point, if you want these kinds of fixes, you need to update to npm@3.

That said, there will be some further changes to shrinkwrap soon to incorporate support for shasums, and when we do that we may eliminate one or both of from and version as redundant if resolved and shasum are available (not to say that we won't use them if they're there). However, the team believes that with the changes in npm@3, this issue is resolved. Thanks to all for your time and patience!