why is package-lock being ignored?

[kidd3]

I'm opening this issue because:

I had been using npm version 5.0.3 in my project and the package-lock.json file had been working exactly as I expected. However, upon updating my npm version to 5.3.0, my package-lock.json no longer works.

Example:

package.json:
"Package-A": "^v1.0.0"

package-lock.json:
"Package-A": { version: 1.0.0 }

When I have no node_modules folder and I attempt to do a fresh npm install, previously in npm version 5.0.3 this would install version 1.0.0 (as this is what the lock file states). However, now on npm version 5.3.0, a fresh install will cause any version from the range ^v1.0.0 to be installed, completely ignoring the lock file.

I want my project to be installed identical every time based on the lock file, this surely is what a lock file is supposed to do?

How can I make my project useable in a projection environment to ensure that the lock file is the only point of reference for npm install?

[kidd3]

Is anyone able to give me some help with this please? I am using npm 5.3.0 and I need my production environment to be the same overtime, which is why I was originally using the package-lock.json. Please can someone give me some information on how I should now have my project set-up as package-lock.json is now updated every time and not locked at all.

[jotaen]

The concept of the package-lock.json is either utterly confusing or not really thought through yet.

This document states that npm would install dependencies as they are specified in the lock file. However, as you observed, this is not true at all and I can reproduce the exact same behaviour with npm 5.3.0.

As a current workaround, I pin my dependencies in package.json, so I write 1.2.0 instead of ~1.2.0 or ^1.2.0. I commit the auto-generated package-lock.json and keep it up to date, but I’m not exactly sure what npm does with it under the hood. At least that strategy seems to yield reproducible builds then.

@jakeNiemiec instead of just voting this question down it would be nice if you would provide any helpful explanation or guidance instead. Lock files are anything else then transparent at the moment and infact behave fundamentally different as they do on other platforms, such as PHP/composer for instance. The confusion here is more then understandable.

[jotaen]

Btw., see this Stack Overflow discussion, which also raises more questions than it solves: https://stackoverflow.com/questions/45022048/why-does-npm-install-rewrite-package-lock-json

[kidd3]

Thank you @jotaen, I have been trying to highlight this issue for over a week now. Currently I have had to rollback to npm 5.0.3 as this issue doesn't seem to occur there. I looked at the change log and saw that when npm 5.1.0 was released it allowed the package.json to trump the lock file, so it seems suspicious that the issue started happening after that.
Currently I have no way to create a static, reproducible production environment for testing, because every time I do a fresh npm install with version 5.3.0 it ignores the lock file and installs the newest versions of packages.

[jotaen]

Thanks for pointing to the Changelog: the root cause is #16866, which is the proof that the current behaviour is indeed intended. There was already a lengthy discussion happening in this PR, however, I doubt that the documentation was properly updated after the change being applied.

In my eyes #16866 is not a good decision, because the introduced behaviour is not predictable and transparent to the user. Also apparently, the naming lock doesn’t reflect the actual purpose of the file anymore, because it’s no longer a lock file, but rather just a dependency snapshot that is taken into consideration in particular cases.

I doubt that this design decision is reverted, so I guess we now have to deal with it (or switch to yarn or whatever). At least the documentation should be updated properly to allow users to understand the current behaviour.

Last note: #16866 is a breaking, non-backward compatible change, which shouldn’t have been introduced as minor version upgrade.

[kidd3]

@jotaen this is indeed what I have been trying to highlight with this question here and my posts on the other feed. The fact that now the lock file no longer locks means it has no purpose in terms of locking packages. A suggestion to have an additional flag (i.e. npm install ---lock-file) to enforce an installation ONLY from the lock file is a decent one, but I do not know where to officially make this request.
Currently version 5.3.0 is unusable when trying to create a static, repeatable environment.

@kidd3 This might be helpful:
https://github.com/npm/npm/blob/latest/CONTRIBUTING.md

I am neither a collaborator, nor an employee of npm. But based on my experience you can either:

1. make the changes yourself and make a pull request (if you have the time and feel comfortable enough to change the code).
2. Or wait for a collaborator or a npm employee to do so. But considering the number of issues and threads on this repo, it would be very unlikely for the latter to happen, unless a huge number of people are experiencing the same issue. Which in this case, unfortunately, I don't think is happening.

So your best chance might be to make the changes in the code yourself, reference it in this issue, and hope the PR is accepted.

P.S: Just as a pointer, you will want to start with making your changes to this file, if you decide to go with the DIY option:
https://github.com/npm/npm/blob/latest/lib/install.js

[jotaen]

Thanks for pointing that out. The only thing that is a bit unfortunate though, is that such a PR would basically be a revert of #16866, which makes it quite unlikely to be accepted. This whole topic has already been discussed back and forth in that PR.

The missing bit for me is now to update the documentation accordingly, which is misleading if not incorrect. On the page about package locks it says:

The presence of a package lock changes the installation behavior such that:

1. The module tree described by the package lock is reproduced. This means reproducing the structure described in the file, using the specific files referenced in "resolved" if available, falling back to normal package resolution using "version" if one isn't.

This is not true, as the generated module tree might be (as of v5.1.0) a combined result of both package.json and package-lock.json.

Also the documentation should point out a way how to perform truly reproducible builds. That was the reason why package locks have been introduced in the first place and that was also what a lot of people obviously had relied on since package-lock.json had been introduced.

@jotaen Very true. That exact part of the documentation is a bit misleading. It might be a good idea to change that as well.
But this PR does not need to be a revert of #16866 since we are not changing the default behavior of npm install.
npm install can still do exactly what it is doing right now, so this PR won't affect anyone who is happily using it now. We can just add another option (E.g. --lock or --read-only or --lock-file) that reads the tree from lock file and does not heed the differences between the lock file and package.json, whatsoever.

[jotaen]

@Maziar-Fotouhi that makes sense, thanks for clarifying.

Maybe it’s an idea to extract the outcome of the discussion here into separate issues. I doubt that this issue here deserves the right attention, since the title is rather suggesting a support request.

I opened #18103 for changing/updating the documentation.

@kidd3 Maybe you could also open a new issue in which you present the idea of introducing some sort of CLI option. (Or, if you have the time and feel confident, you could directly work on a PR.) If you think that your needs are covered thereby, this issue here could be closed then.

[kidd3]

@Maziar-Fotouhi thanks for the information, but unfortunately I don't have enough time right now to attempt to fix this issue myself, I have already had to rollback to v5.0.3 to continue my own work.
Considering it didn't exist prior to #16866 I can only assume people haven't encountered this issue as they aren't needing to have a repeatable, static install for an environment for testing. Otherwise I really don't understand why #16866 was allowed to happen, as it completely breaks npm and the lock file - which as @jotaen has pointed out, no longer matches the documentation which is clearly still a version of docs from before v5.1.0.
As it currently stands, npm v5.3.0 is completely unusable if you are attempting to use the package-lock.json file as an actual lock file. So I would really hope someone in the npm community can add a fix to the workflow soon, or at least give some indication of a timeframe for a fix.