package-lock.json and optional packages

[Strate]

I'm opening this issue because:

• npm is doing something I don't understand.

What's going wrong?

Npm accidently make changes to package-lock.json

How can the CLI team reproduce the problem?

Assume you have a 2 developers, one on mac, and one on linux. You use npm@5.1 and your project depends on chokidar package. That package has optional dependency of fsevents, which is useful only for mac. So, you are on linux, and do npm i chokidar. npm generates package-lock.json withous fsevents, because it is useless on linux. You commit that generated file.

Your teammate pulls your changes, and do npm i, to get node_modules in sync with package-lock.json. Npm installs fsevents, and write it to package-lock.json. What should mac user to do? commit that file?

Assume that mac user commits file. Linux user pulls it, and make npm install. Npm does not install fsevents, and remove it from package-lock.json. And this become annoying very quickly.

How lock files could be used in this case?

supporting information:

• npm -v prints: 5.1.0
• node -v prints: v6.10.0
• npm config get registry prints: undefined
• Windows, OS X/macOS, or Linux?: max + linux
• Network issues:
  • Geographic location where npm was run:
  • I use a proxy to connect to the npm registry.
  • I use a proxy to connect to the web.
  • I use a proxy when downloading Git repos.
  • I access the npm registry via a VPN
  • I don't use a proxy, but have limited or unreliable internet access.
• Container:
  • I develop using Vagrant on Windows.
  • I develop using Vagrant on OS X or Linux.
  • I develop / deploy using Docker.
  • I deploy to a PaaS (Triton, Heroku).

[mborisv]

👍

[topheman]

We have the exact same problem. Developers use MacOS and Linux across our team and a simple npm install might update the package-lock.json (adding or removing fsevents) (according on the plaform this file was created first) - even though there wasn't any package added.

Is there a flag to do the opposite of npm install --no-optional ? (though, this might not work, since the platform will override anyway)

[Strate]

We also can not force to use fsevents package, because it has strict requirement of os.

[topheman]

@Strate I agree, it can also be the other way around - it might even make more sense: treat optionalDependencies as side effects (since package.json ➡️ package-lock.json should be pure), but some people will raise the fact that this isn't a 100% reproducible build.

At the end this is what we want, using package-lock.json: reproducible builds.

How is that issue managed using yarn ? (I don't remember having this kind of problem - I'll have to check).

Edit: On the other hand, if you use the --no-optional flag, on a MacOS computer, you wont be able to benefit from fsevents ...

[usandfriends]

Hey team, any updates on this? We're getting really close to having npm-shrinkwrap.json working 100% between our Linux CI server and our local OS X machines, just have this last issue. Thanks!

[wojtekk]

We have this issue on Travis CI with package fsevents - we are working on Macs, Travis CI uses Ubuntu images.

To prevent the issue we install packages with flag: --no-save in a .travis.yml:

npm install --no-save

[VincentBailly]

We have the same issue in my team, I don't understand why "npm install" on windows/linux removes fsevents from package-lock.json. The documentation states that it should be there always: https://docs.npmjs.com/files/package-lock.json

[mtibben]

Using npm install --no-save works, but isn't ideal in a CI environment. In a CI environment you want to ensure you have a reproducible build from the lockfile, but using --no-save can result in a node_modules different to your lockfile if your package.json specifies something different. You want to get feedback (ideally a hard fail) if node_modules and package-lock.json is out of sync after an npm install. And this is just not always possible if the lock file is flapping between architectures.

So I would really like to see a flag to assert that the lockfile and node_modules is in sync, and incorporated into the install command. Perhaps something like npm install --from-lockfile-only which then hard fails if package.json and the lockfile are out of sync

[00dani]

An added flag like npm install --from-lockfile-only would be perfect for CI and the like, but it doesn't help so much when your development machines have different architectures, as ours do.

Perhaps the lockfile format should be extended so that optional dependencies are annotated as such - "fsevents": {"version": "1.1.1", "optional": true, ...} - with npm interpreting the optional flag as specifying something like "if this package is installed at all, it must be the version specified, but if it's not installed at all don't worry about it"? I believe currently the lockfile is directly derived from the contents of node_modules, which would be why optional packages vanish from it if not installed, so that would need to change.

… and then I noticed that the lockfile does have "optional": true already. Alright. So, maybe npm should simply copy across optional packages into the new lockfile?

[gfzabarino]

👍

[desfero]

Our team also have the same problem which is really annoying, because each reviewer should check what was changed in package-lock. Maybe like @00dani propose add all optionals with a flag?

[CristovaoHonorato]

Is there a workaround for this issue, other than using --no-save?

[DmitryKorlas]

As workaround I see the only solution - not to commit package-lock.json if you don't change package.json

[impaler]

If something is non optional and platform specific it doesn't seem enough to describe that with "optionalDependencies". Until then we will see scripts & solutions like this https://github.com/bertofer/npm-platform-dependencies. If the lock-file is supposed to be an immutable representation of an install to reproduce its not entirely possible if the os isn't recorded as well. Maybe we need platform specfic lock files or an addition in there that is platform specific?

Ideally I'd like to see cross platform development as frictionless as possible. Platform specific dependencies should be described as what they are. imo:

"fs-events is a mac & development only requirement" !== "optionalDependencies"

Also --no-save is useful in itself 👍

[00dani]

Making the whole lockfile platform-specific - package-lock.darwin.json or whatever - makes it harder to guarantee that the packages you do install are at exactly the same versions. The majority of packages installed in a typical project should be identical across platforms.

Pushing platform-specific packages into a special subtree of the lockfile would be fine, though. How 'bout this?

"dependencies": {"some-package": {...}, "some-other-package": {...}, ...},
"osDependencies": {
  "darwin":  {"some-mac-package": {...}, ...},
  "linux": {"some-linux-package": {...}, ...}
}

If added, osDependencies probably should also be supported in package.json.

[impaler]

Yes thats what I had in mind @00dani and it would probably need to support dev dependencies as well.

[00dani]

The lockfile would handle dev dependencies just fine with just the keys above, since currently it handles dev dependencies simply by marking them with a "dev": true flag rather than storing them in a separate section.

It's trickier for package.json, however, since that file has actual separate sections for devDependencies and optionalDependencies. So as an alternative proposal, perhaps we could repurpose package.json's existing os key? Currently that key can only contain an array of supported platforms - keeping that behaviour unchanged, we could alternatively allow an object:

{
  "name": "chokidar",
  "dependencies": {"anymatch": "^1.3.0", ...},
  "os": {
    "darwin": {"dependencies": {"fsevents": "^1.0.0", ...}}
  }
}

[acutmore]

Split optional dependencies into separate folders with separate package.json files.

Not the cleanest solution but very simple.

Can have a different folder for mac/Linux/windows/developers/CI.