This repository has been archived by the owner. It is now read-only.

RFC: file: specifier changes #15900

Closed
wants to merge 210 commits into
from

Conversation

Projects
None yet
@iarna
Member

iarna commented Feb 28, 2017

Easy reading link for the new specification. This also introduces written specifications for npm features, something we intend to extend out to all of npm's activity.

The Problem We're Solving

File type dependency specifiers are confusing. How they interact across npm's many varied commands (npm install --save, npm update, npm outdated, npm shrinkwrap) has never been defined in one place. Most of the behaviors "happened by default". That is, the minimum implementation was done to make them install ok and the rest was left as a side effect.

Currently npm update and npm outdated simply don't do anything for local dependencies. If you've updated the source you have to manually reinstall to get a copy of it. How npm shrinkwrap saves local dependencies to npm-shrinkwrap.json and how it resolves them has varied over time as well.

Root Causes

When local dependencies were added, we didn't have any process around defining behavior or ensuring that all use cases were specified.

Proposal

In addition to actually specifying behavior (often simply writing down what
things currently do), we propose in important breaking change:

  • file:-type specifiers that refer to directories will be soft deprecated, their behavior being identical to the new link: specifier and their existence becoming a footnote in the documentation.
  • A new specifer type, link:, will be introduced for linking local directories. For the duration of npm@5,file:specifiers that refer to directories will be treated identically tolink:` specifiers.
  • link: specifiers will result in a symlink or junction made from the specifier path into your node_modules. On Windows try a junction and if that fails, try a symlink. If both fail, the error from the junction should be used.

This RFC essentially brings linklocal's bevior into core.

RISKS

  • Diagnostic information changes need to be handle delicately in order to not increase user confusion.
  • Some users may be unhappy with the changes to file: semantics.

@iarna iarna added the breaking label Feb 28, 2017

@timoxley

This comment has been minimized.

Show comment
Hide comment
@timoxley

timoxley Mar 1, 2017

Member

Nice! Why new link: vs just using file:?

Member

timoxley commented Mar 1, 2017

Nice! Why new link: vs just using file:?

@iarna

This comment has been minimized.

Show comment
Hide comment
@iarna

iarna Mar 1, 2017

Member

The reason for introducing the new specifier are:

  1. Have a specifier that immediately tells someone reading a package.json what it will do.
  2. Not have fights over what a valid file: URL looks like.

Originally there was a plan to actually hard deprecate the file: specifier and remove them, but we decided that was pointlessly disruptive. Instead file: will be an alias for link:.

Member

iarna commented Mar 1, 2017

The reason for introducing the new specifier are:

  1. Have a specifier that immediately tells someone reading a package.json what it will do.
  2. Not have fights over what a valid file: URL looks like.

Originally there was a plan to actually hard deprecate the file: specifier and remove them, but we decided that was pointlessly disruptive. Instead file: will be an alias for link:.

@timoxley

This comment has been minimized.

Show comment
Hide comment
@timoxley

timoxley Mar 1, 2017

Member

@iarna Sounds reasonable 👍

Have you considered the impact of having different module resolution paths when using symlinks?
I currently only use symlinks in development, while in production I let the current file: behaviour do its thing. Without the symlinks npm is able to move many shared packages to the top level, decreasing overall install time. This could have significant impact on deploy times if local dependencies share a lot of binary dependencies.

Member

timoxley commented Mar 1, 2017

@iarna Sounds reasonable 👍

Have you considered the impact of having different module resolution paths when using symlinks?
I currently only use symlinks in development, while in production I let the current file: behaviour do its thing. Without the symlinks npm is able to move many shared packages to the top level, decreasing overall install time. This could have significant impact on deploy times if local dependencies share a lot of binary dependencies.

@iarna

This comment has been minimized.

Show comment
Hide comment
@iarna

iarna Mar 1, 2017

Member

@timoxley It's a fair question. As it's specced right now it would do that. Maybe, if the linked module is inside the top level install root we should flatten its deps down, instead of keeping them inside the linked module. The module loader would still be able to pick them up and it'd not increase disk usage.

Member

iarna commented Mar 1, 2017

@timoxley It's a fair question. As it's specced right now it would do that. Maybe, if the linked module is inside the top level install root we should flatten its deps down, instead of keeping them inside the linked module. The module loader would still be able to pick them up and it'd not increase disk usage.

@jamiebuilds jamiebuilds referenced this pull request Mar 1, 2017

Merged

:link dependencies #34

@timoxley

This comment has been minimized.

Show comment
Hide comment
@timoxley

timoxley Mar 1, 2017

Member

if the linked module is inside the top level install root we should flatten its deps down

@iarna that sounds good. Being aware of where symlinks resolve to perhaps could be considered a good general improvement to the dedupe/flattening algorithm.

Member

timoxley commented Mar 1, 2017

if the linked module is inside the top level install root we should flatten its deps down

@iarna that sounds good. Being aware of where symlinks resolve to perhaps could be considered a good general improvement to the dedupe/flattening algorithm.

Show outdated Hide outdated doc/spec/file-and-link-specifiers.md
* Attempting to install a specifer that has a windows drive letter will
produce an error on non-Windows systems.
* A valid `file:` specifier points is:

This comment has been minimized.

@thefourtheye

thefourtheye Mar 1, 2017

Contributor

Did you mean "A valid file: specifier also is" like in the next item? I am finding it difficult to understand this line.

@thefourtheye

thefourtheye Mar 1, 2017

Contributor

Did you mean "A valid file: specifier also is" like in the next item? I am finding it difficult to understand this line.

This comment has been minimized.

@legodude17

legodude17 Mar 1, 2017

Contributor

I would say it should be "Also, a valid file: specifier is:"

@legodude17

legodude17 Mar 1, 2017

Contributor

I would say it should be "Also, a valid file: specifier is:"

Show outdated Hide outdated doc/spec/file-and-link-specifiers.md
Historically, these ambiguous specifiers were also allowed in the
`package.json`. Starting in `npm@5` using an ambiguous specifier in your
shrinkwrap will be depricated and will warn. In `npm@6` it will be an

This comment has been minimized.

@mantoni

mantoni Mar 1, 2017

Contributor

Typo: depricated -> deprecated

@mantoni

mantoni Mar 1, 2017

Contributor

Typo: depricated -> deprecated

@mantoni

This comment has been minimized.

Show comment
Hide comment
@mantoni

mantoni Mar 1, 2017

Contributor

This is great! I would like to know if you considered supporting the current npm ln use-case? I'm thinking of link:module-name to resolve to the "global" link which in turn links to the actual module directory. Contrary to the relative link paths, this wouldn't enforce a specific directory structure. What do you think?

Contributor

mantoni commented Mar 1, 2017

This is great! I would like to know if you considered supporting the current npm ln use-case? I'm thinking of link:module-name to resolve to the "global" link which in turn links to the actual module directory. Contrary to the relative link paths, this wouldn't enforce a specific directory structure. What do you think?

@legodude17

This seems like it would really help a lot of people. 🎆

Show outdated Hide outdated doc/spec/file-and-link-specifiers.md
* Attempting to install a specifer that has a windows drive letter will
produce an error on non-Windows systems.
* A valid `file:` specifier points is:

This comment has been minimized.

@legodude17

legodude17 Mar 1, 2017

Contributor

I would say it should be "Also, a valid file: specifier is:"

@legodude17

legodude17 Mar 1, 2017

Contributor

I would say it should be "Also, a valid file: specifier is:"

Show outdated Hide outdated doc/spec/file-and-link-specifiers.md
* A valid `file:` specifier points is:
* a valid package file. That is, a `.tar`, `.tar.gz` or `.tgz` containing
`<dir>/package.json`.
* OR, a directory that contains a `package.json`

This comment has been minimized.

@legodude17

legodude17 Mar 1, 2017

Contributor

Should that 'OR' be removed or should all of these start with 'OR'? Consistency.

@legodude17

legodude17 Mar 1, 2017

Contributor

Should that 'OR' be removed or should all of these start with 'OR'? Consistency.

This comment has been minimized.

@iarna

iarna Mar 2, 2017

Member

There are only two of them and the OR separates the two. So I'm not sure what you mean by "all of these".

@iarna

iarna Mar 2, 2017

Member

There are only two of them and the OR separates the two. So I'm not sure what you mean by "all of these".

Show outdated Hide outdated doc/spec/file-and-link-specifiers.md
* a valid package file. That is, a `.tar`, `.tar.gz` or `.tgz` containing
`<dir>/package.json`.
* OR, a directory that contains a `package.json`
* And a valid `file:` specifier also is:

This comment has been minimized.

@legodude17

legodude17 Mar 1, 2017

Contributor

Shouldn't this not be a bullet? Or just remove it.

@legodude17

legodude17 Mar 1, 2017

Contributor

Shouldn't this not be a bullet? Or just remove it.

This comment has been minimized.

@iarna

iarna Mar 2, 2017

Member

It should. It's part of the top level list.

@iarna

iarna Mar 2, 2017

Member

It should. It's part of the top level list.

Show outdated Hide outdated doc/spec/file-and-link-specifiers.md
The `preinstall` for file-type specifiers MUST be run AFTER the
`finalize` phase as the symlink may be a relative path reaching outside the
current project root and a symlink that resolves in `.staging` won't resolve
in the package's final resting place.

This comment has been minimized.

@legodude17

legodude17 Mar 1, 2017

Contributor

Would the preinstall for a package depended on via link: be run from it's original folder, or somewhere else?

@legodude17

legodude17 Mar 1, 2017

Contributor

Would the preinstall for a package depended on via link: be run from it's original folder, or somewhere else?

This comment has been minimized.

@iarna

iarna Mar 2, 2017

Member

Basically this is me trying to not screw people who are using pwd (which would be relative to where your package root's node_modules).

Folks who are using cwd would always get the same answer (the destination of the link).

@iarna

iarna Mar 2, 2017

Member

Basically this is me trying to not screw people who are using pwd (which would be relative to where your package root's node_modules).

Folks who are using cwd would always get the same answer (the destination of the link).

Show outdated Hide outdated doc/spec/file-and-link-specifiers.md
```
example-package@1.0.0 /path/to/example-package
+-- a -> Copied from: link:../a
```

This comment has been minimized.

@legodude17

legodude17 Mar 1, 2017

Contributor

Again, is this repetition intentional?

@legodude17

legodude17 Mar 1, 2017

Contributor

Again, is this repetition intentional?

Show outdated Hide outdated doc/spec/file-and-link-specifiers.md
```
example-package@1.0.0 /path/to/example-package
+-- a -> link:../a
```

This comment has been minimized.

@legodude17

legodude17 Mar 1, 2017

Contributor

Why are there two parts with the same thing? Is that intentional?

@legodude17

legodude17 Mar 1, 2017

Contributor

Why are there two parts with the same thing? Is that intentional?

@iarna

This comment has been minimized.

Show comment
Hide comment
@iarna

iarna Mar 2, 2017

Member

So after some discussion in air between @zkat and I, we're of a consensus that we should go ahead with just file: and not add a new specifier. I'll update the RFC as appropriate.

Member

iarna commented Mar 2, 2017

So after some discussion in air between @zkat and I, we're of a consensus that we should go ahead with just file: and not add a new specifier. I'll update the RFC as appropriate.

@coveralls

This comment has been minimized.

Show comment
Hide comment
@coveralls

coveralls Mar 2, 2017

Coverage Status

Coverage remained the same at 86.013% when pulling 43db052 on link-specifier into a08189f on latest.

Coverage Status

Coverage remained the same at 86.013% when pulling 43db052 on link-specifier into a08189f on latest.

Show outdated Hide outdated doc/spec/file-and-link-specifiers.md
```
example-package@1.0.0 /path/to/example-package
+-- a -> file:../a

This comment has been minimized.

@legodude17

legodude17 Mar 2, 2017

Contributor

Still, is this repetition intentional?

@legodude17

legodude17 Mar 2, 2017

Contributor

Still, is this repetition intentional?

This comment has been minimized.

@zkat

zkat Mar 2, 2017

Member

what repetition? directories and package names can be different: a -> file:../b is perfectly valid

@zkat

zkat Mar 2, 2017

Member

what repetition? directories and package names can be different: a -> file:../b is perfectly valid

This comment has been minimized.

@iarna

iarna Mar 2, 2017

Member

One of these is with unicode enabled, one without. I wanted to specify both.

@iarna

iarna Mar 2, 2017

Member

One of these is with unicode enabled, one without. I wanted to specify both.

This comment has been minimized.

@legodude17

legodude17 Mar 2, 2017

Contributor

Oh. Sorry. I didn't notice that.

@legodude17

legodude17 Mar 2, 2017

Contributor

Oh. Sorry. I didn't notice that.

Show outdated Hide outdated doc/spec/file-and-link-specifiers.md
```
Package Current Wanted Latest Location
a MISSING LOCAL LOCAL example-package

This comment has been minimized.

@legodude17

legodude17 Mar 2, 2017

Contributor

I would suggest that it should also log where it is looking for the package in.

@legodude17

legodude17 Mar 2, 2017

Contributor

I would suggest that it should also log where it is looking for the package in.

This comment has been minimized.

@iarna

iarna Mar 2, 2017

Member

As zkat says below, changing outdated to be more useful is really a separate RFC.

@iarna

iarna Mar 2, 2017

Member

As zkat says below, changing outdated to be more useful is really a separate RFC.

This comment has been minimized.

@legodude17

legodude17 Mar 2, 2017

Contributor

Ok, cool. Sounds like a really good thing.

@legodude17

legodude17 Mar 2, 2017

Contributor

Ok, cool. Sounds like a really good thing.

@zkat

pretty excited about this. Thanks for writing the spec up <3

Show outdated Hide outdated doc/spec/file-and-link-specifiers.md
#### File type specifers pointing at directories
File-type specifiers that point at directories will necessarily not do
anything for `fetch` and `extract` phases.

This comment has been minimized.

@zkat

zkat Mar 2, 2017

Member

How do you feel about pacote doing this automagically on pacote.extract?

@zkat

zkat Mar 2, 2017

Member

How do you feel about pacote doing this automagically on pacote.extract?

This comment has been minimized.

@iarna

iarna Mar 2, 2017

Member

Hrm. I don't think pacote has quite enough information currently to resolve this sort of thing. The specifier and the target destination aren't enough. You also need to know the location of the module that required it, because the specifier is relative to THAT.

@iarna

iarna Mar 2, 2017

Member

Hrm. I don't think pacote has quite enough information currently to resolve this sort of thing. The specifier and the target destination aren't enough. You also need to know the location of the module that required it, because the specifier is relative to THAT.

This comment has been minimized.

@zkat

zkat Mar 2, 2017

Member

Can/should we add this to realize-package-specifier? We use where there already.

@zkat

zkat Mar 2, 2017

Member

Can/should we add this to realize-package-specifier? We use where there already.

This comment has been minimized.

@iarna

iarna Mar 2, 2017

Member

Maybe, I mean, we might actually already be resolving come to think. It just requires some end-to-end care.

@iarna

iarna Mar 2, 2017

Member

Maybe, I mean, we might actually already be resolving come to think. It just requires some end-to-end care.

Show outdated Hide outdated doc/spec/file-and-link-specifiers.md
`file:///foo/bar` reference the same package.
* … or a relative path (eg `../path/to/thing`, `path\to\subdir`). Leading
slashes on a file specifier will be removed, that is 'file://../foo/bar`
references the same package as same as `file:../foo/bar`. The latter is

This comment has been minimized.

@zkat

zkat Mar 2, 2017

Member

Can you add a quick clarification that file://foo/bar is considered relative, too? At least that's what I assume from reading this. It's a weird, sandwichy corner case, but it's probably worth actually specifying:

  • file:///foo -> Absolute /foo
  • file://foo -> Relative, same as ./foo
  • file:/foo -> Absolute /foo
@zkat

zkat Mar 2, 2017

Member

Can you add a quick clarification that file://foo/bar is considered relative, too? At least that's what I assume from reading this. It's a weird, sandwichy corner case, but it's probably worth actually specifying:

  • file:///foo -> Absolute /foo
  • file://foo -> Relative, same as ./foo
  • file:/foo -> Absolute /foo

This comment has been minimized.

@iarna

iarna Mar 2, 2017

Member

Number of leading slashes changes nothing:

file:///foo -> /foo
file://foo -> /foo
file:/foo -> /foo
With no leading slashes, relative paths are evident:

file:foo -> ./foo
file:../ -> ../foo

I'm proposing that if you have .. as your first path element then we ignore any leading slashes:

file:/../foo -> ../foo
file://../foo -> ../foo
file:///../foo -> ../foo

I think this would eliminate whole classes of likely errors, particularly around users who are trying to make these things work like URLs. Also /../ is not a construct that would ever normally exist. It's a nonsensical noop in and of itself.

@iarna

iarna Mar 2, 2017

Member

Number of leading slashes changes nothing:

file:///foo -> /foo
file://foo -> /foo
file:/foo -> /foo
With no leading slashes, relative paths are evident:

file:foo -> ./foo
file:../ -> ../foo

I'm proposing that if you have .. as your first path element then we ignore any leading slashes:

file:/../foo -> ../foo
file://../foo -> ../foo
file:///../foo -> ../foo

I think this would eliminate whole classes of likely errors, particularly around users who are trying to make these things work like URLs. Also /../ is not a construct that would ever normally exist. It's a nonsensical noop in and of itself.

Show outdated Hide outdated doc/spec/file-and-link-specifiers.md
* Attempting to install a specifer that has a windows drive letter will
produce an error on non-Windows systems.
* A valid `file:` specifier points is:
* a valid package file. That is, a `.tar`, `.tar.gz` or `.tgz` containing

This comment has been minimized.

@zkat

zkat Mar 2, 2017

Member

We should make sure the error message from this is VERY CLEAR that we determine tarballness based on file extension. I know we talked about this a lot, but I still want to avoid linuxy/unixy folks pulling their hair out if they, say, try to install a tarball they downloaded from a webservice that does not specify a filename and they end up with an unsuffixed filename that is still technically a tarball.

Or, for example, if someone tries to npm install ~/.npm/.pacote/content/deadbeef, which do not have .tgz suffixes.

@zkat

zkat Mar 2, 2017

Member

We should make sure the error message from this is VERY CLEAR that we determine tarballness based on file extension. I know we talked about this a lot, but I still want to avoid linuxy/unixy folks pulling their hair out if they, say, try to install a tarball they downloaded from a webservice that does not specify a filename and they end up with an unsuffixed filename that is still technically a tarball.

Or, for example, if someone tries to npm install ~/.npm/.pacote/content/deadbeef, which do not have .tgz suffixes.

Show outdated Hide outdated doc/spec/file-and-link-specifiers.md
#### File type specifiers pointing at tarballs
File-type specifiers pointing at a `.tgz` or `.tar.gz or `.tar` file will

This comment has been minimized.

@zkat

zkat Mar 2, 2017

Member

missing backtick (`) here after .tar.gz

@zkat

zkat Mar 2, 2017

Member

missing backtick (`) here after .tar.gz

Show outdated Hide outdated doc/spec/file-and-link-specifiers.md
dependencies of the linked package will be hoisted to the top level as usual.
If the module is outside the package root then dependencies will be installed inside
the linked module's `node_modules` folder.

This comment has been minimized.

@zkat

zkat Mar 2, 2017

Member

Is this going to be done as a separate npm run within that project folder (possibly a subprocess?) or will we build a single ideal tree with some special semantics, and do it in a single npm run? This seems trickier to me than what this single sentence describes. 🤔

@zkat

zkat Mar 2, 2017

Member

Is this going to be done as a separate npm run within that project folder (possibly a subprocess?) or will we build a single ideal tree with some special semantics, and do it in a single npm run? This seems trickier to me than what this single sentence describes. 🤔

This comment has been minimized.

@iarna

iarna Mar 2, 2017

Member

Nah, single thing, it doesn't even require much in the way of special semantics. It's basically the same logic as how we define "global" mode currently.

@iarna

iarna Mar 2, 2017

Member

Nah, single thing, it doesn't even require much in the way of special semantics. It's basically the same logic as how we define "global" mode currently.

Show outdated Hide outdated doc/spec/file-and-link-specifiers.md
```
Package Current Wanted Latest Location
a MISSING LOCAL LOCAL example-package

This comment has been minimized.

@zkat

zkat Mar 2, 2017

Member

oh god Location is such a horrible, confusing name when we factor this in. I really bloody wish we had a different name for that column.

@zkat

zkat Mar 2, 2017

Member

oh god Location is such a horrible, confusing name when we factor this in. I really bloody wish we had a different name for that column.

This comment has been minimized.

@zkat

zkat Mar 2, 2017

Member

Additionally, I wish we had somewhere to put more information about the package getting installed, such as from or something, because the package name by itself doesn't give us much. But that's something for a separate RFC.

@zkat

zkat Mar 2, 2017

Member

Additionally, I wish we had somewhere to put more information about the package getting installed, such as from or something, because the package name by itself doesn't give us much. But that's something for a separate RFC.

This comment has been minimized.

@iarna

iarna Mar 2, 2017

Member

Yup, I'd love to change that too

@iarna

iarna Mar 2, 2017

Member

Yup, I'd love to change that too

This comment has been minimized.

@legodude17

legodude17 Mar 2, 2017

Contributor

Wait, I am confused. So package is the name, and Location is the path on disk? Or is it the path to it through the dependency chain? Or is it the package that depended on this?

@legodude17

legodude17 Mar 2, 2017

Contributor

Wait, I am confused. So package is the name, and Location is the path on disk? Or is it the path to it through the dependency chain? Or is it the package that depended on this?

This comment has been minimized.

@zkat

zkat Mar 2, 2017

Member

@legodude17 npm help outdated will answer your questions here.

@zkat

zkat Mar 2, 2017

Member

@legodude17 npm help outdated will answer your questions here.

This comment has been minimized.

@legodude17

legodude17 Mar 2, 2017

Contributor

@zkat thanks for the tip. You are right that Location is a very confusing name for that. Maybe Parent?

@legodude17

legodude17 Mar 2, 2017

Contributor

@zkat thanks for the tip. You are right that Location is a very confusing name for that. Maybe Parent?

This comment has been minimized.

@zkat

zkat Mar 2, 2017

Member

@legodude17 we'll talk about that once there's an RFC for it. Talking about it in here is bound to get lost in oblivion (and is offtopic for this PR)

@zkat

zkat Mar 2, 2017

Member

@legodude17 we'll talk about that once there's an RFC for it. Talking about it in here is bound to get lost in oblivion (and is offtopic for this PR)

This comment has been minimized.

@legodude17

legodude17 Mar 2, 2017

Contributor

Sounds like a good idea. Do you have a estimated time for this?

@legodude17

legodude17 Mar 2, 2017

Contributor

Sounds like a good idea. Do you have a estimated time for this?

This comment has been minimized.

@iarna

iarna Mar 4, 2017

Member

Nope, not part of the current schedule. I have on my todo-list writing up an up-to-date "state of NPM" blog post to bring everyone up-to-date on our goals and plans and expectations.

@iarna

iarna Mar 4, 2017

Member

Nope, not part of the current schedule. I have on my todo-list writing up an up-to-date "state of NPM" blog post to bring everyone up-to-date on our goals and plans and expectations.

@iarna iarna changed the title from RFC: file: specifier changes & link: specifier to RFC: file: specifier changes Mar 4, 2017

@mlucool

This comment has been minimized.

Show comment
Hide comment
@mlucool

mlucool Mar 8, 2017

Using symlinks with node, requires a flag set for it to work correctly: nodejs/node#8749 (review). Is the intent to change how node handles symlinks also?

One nice feature about how file works today is for peer dependencies. Due to the copy instead of a symlink, peers with global singletons work as expected. When this moves to symlinks only, the resolution will make certain cases impossible to resolve. Example:

  • Module A is a react component and has react as a peerDependency and devDependency
  • While working on A, I want to test this locally in Module B , so I use file:/path/to/a.
    • Today it will do the right thing and share the global version of react as it is copied. The dependency resolution won't include a second react.
    • If instead we did a symlink, this would not work because React is installed in in A's node_modules and would not be the same instance as in as B's node_modules.

Let me know if that example needs more clarification or if I have misunderstood RFC.

mlucool commented Mar 8, 2017

Using symlinks with node, requires a flag set for it to work correctly: nodejs/node#8749 (review). Is the intent to change how node handles symlinks also?

One nice feature about how file works today is for peer dependencies. Due to the copy instead of a symlink, peers with global singletons work as expected. When this moves to symlinks only, the resolution will make certain cases impossible to resolve. Example:

  • Module A is a react component and has react as a peerDependency and devDependency
  • While working on A, I want to test this locally in Module B , so I use file:/path/to/a.
    • Today it will do the right thing and share the global version of react as it is copied. The dependency resolution won't include a second react.
    • If instead we did a symlink, this would not work because React is installed in in A's node_modules and would not be the same instance as in as B's node_modules.

Let me know if that example needs more clarification or if I have misunderstood RFC.

@zkat

This comment has been minimized.

Show comment
Hide comment
@zkat

zkat Mar 8, 2017

Member

@mlucool my understanding of this case is that it would continue to work so long as Module A is anywhere inside the directory structure for B when you link it.

That is:

module-b
\
 | - module-a
 |   \
 |     node_modules
 \
  node_modules
  \
    react

module-a will have its react hoisted to the top. It's worth being more explicit about how we handle peerDeps in this situation. I hopefully didn't misunderstand the RFC in this case.

@iarna it might be worth specifying the logic for peerDependencies and devDependencies for these linked submodules in the RFC, cause I don't think I'm that clear on it rn.

Member

zkat commented Mar 8, 2017

@mlucool my understanding of this case is that it would continue to work so long as Module A is anywhere inside the directory structure for B when you link it.

That is:

module-b
\
 | - module-a
 |   \
 |     node_modules
 \
  node_modules
  \
    react

module-a will have its react hoisted to the top. It's worth being more explicit about how we handle peerDeps in this situation. I hopefully didn't misunderstand the RFC in this case.

@iarna it might be worth specifying the logic for peerDependencies and devDependencies for these linked submodules in the RFC, cause I don't think I'm that clear on it rn.

@mlucool

This comment has been minimized.

Show comment
Hide comment
@mlucool

mlucool Mar 8, 2017

@zkat In my case A is outside of B always. We do not have a monorepo. Still this seems to imply as long as it is note tar it'll use symlinks.

My current hack (which works quite well) is to create a symlink like util that does:

  1. Uninstall A
  2. Install via file:/path/to/a
  3. Start a file watcher for all files in A's package.json:files
  4. Start a file watcher for A's package.json to start from 1 again

Now for any change I make it appears nearly instantly. Clearly this is a hack, but it maintains that I can separate components across repos and not worry about dependency issues.

mlucool commented Mar 8, 2017

@zkat In my case A is outside of B always. We do not have a monorepo. Still this seems to imply as long as it is note tar it'll use symlinks.

My current hack (which works quite well) is to create a symlink like util that does:

  1. Uninstall A
  2. Install via file:/path/to/a
  3. Start a file watcher for all files in A's package.json:files
  4. Start a file watcher for A's package.json to start from 1 again

Now for any change I make it appears nearly instantly. Clearly this is a hack, but it maintains that I can separate components across repos and not worry about dependency issues.

@zkat

This comment has been minimized.

Show comment
Hide comment
@zkat

zkat Mar 8, 2017

Member

@mlucool if you're not going to use a monorepo structure, you could literally just run npm pack and then npm install file://path/to/module-a/module-a-1.2.3.tgz, which will have the same behavior as npm i file://path/to/module-a currently does. We're not changing local tarballs. It's a single extra step, but also something the CLI was already doing (it had to, in order to install the package)

Member

zkat commented Mar 8, 2017

@mlucool if you're not going to use a monorepo structure, you could literally just run npm pack and then npm install file://path/to/module-a/module-a-1.2.3.tgz, which will have the same behavior as npm i file://path/to/module-a currently does. We're not changing local tarballs. It's a single extra step, but also something the CLI was already doing (it had to, in order to install the package)

@zkat

This comment has been minimized.

Show comment
Hide comment
@zkat

zkat Mar 8, 2017

Member

and yes, we will always create symlinks for directory specs. The difference is that we hoist deps normally iff the target package is in a subdir of the current package.

Member

zkat commented Mar 8, 2017

and yes, we will always create symlinks for directory specs. The difference is that we hoist deps normally iff the target package is in a subdir of the current package.

@mlucool

This comment has been minimized.

Show comment
Hide comment
@mlucool

mlucool Mar 8, 2017

@zkat I could do that to keep my hack working. I am still a little unsure about how this hoisting will play out because you can imagine something like:

examples
\
  example1
    \ 
     package.json
components
\
  component1
   \ 
    package.json

If I did a file: in example1, this would not work as far as I can tell now (assuming this is still a React example). I believe this paradigm is relatively common also.

Any comments on the node flag required for this to work?

P.S. I am a BIG fan of better support for symlinks, per my commit to node.

mlucool commented Mar 8, 2017

@zkat I could do that to keep my hack working. I am still a little unsure about how this hoisting will play out because you can imagine something like:

examples
\
  example1
    \ 
     package.json
components
\
  component1
   \ 
    package.json

If I did a file: in example1, this would not work as far as I can tell now (assuming this is still a React example). I believe this paradigm is relatively common also.

Any comments on the node flag required for this to work?

P.S. I am a BIG fan of better support for symlinks, per my commit to node.

@zkat

This comment has been minimized.

Show comment
Hide comment
@zkat

zkat Mar 9, 2017

Member

@mlucool It's worth noting that it's tremendously unlikely for us at npm to try to change module resolution, including symlink stuff, ourselves. afaik, there's no intent for us to actually go and push for a change on that end in order to land this.

More likely, I think, is discussing the possibility of taking that flag into account when deciding how to build the hoisted tree -- some users will want that symlink preservation, some won't. I'll defer to @iarna about whether she thinks this is something worth exploring, with a note that any current hacks around directory dependencies can continue to exist with that npm pack thing I mentioned above.

@iarna does it make sense to you to read process.env.NODE_PRESERVE_SYMLINKS from the env and hoisting across symlink boundaries if we find that flag?

Member

zkat commented Mar 9, 2017

@mlucool It's worth noting that it's tremendously unlikely for us at npm to try to change module resolution, including symlink stuff, ourselves. afaik, there's no intent for us to actually go and push for a change on that end in order to land this.

More likely, I think, is discussing the possibility of taking that flag into account when deciding how to build the hoisted tree -- some users will want that symlink preservation, some won't. I'll defer to @iarna about whether she thinks this is something worth exploring, with a note that any current hacks around directory dependencies can continue to exist with that npm pack thing I mentioned above.

@iarna does it make sense to you to read process.env.NODE_PRESERVE_SYMLINKS from the env and hoisting across symlink boundaries if we find that flag?

@mlucool

This comment has been minimized.

Show comment
Hide comment
@mlucool

mlucool Mar 9, 2017

IMO if something is outside of a module you should never touch its contents.

mlucool commented Mar 9, 2017

IMO if something is outside of a module you should never touch its contents.

@iarna

This comment has been minimized.

Show comment
Hide comment
@iarna

iarna Mar 10, 2017

Member

@mlucool As @zkat says, changing the default Node.js module loader behavior isn't on the table, which is why 'preserve-symlinks' landed as an option.

Thank you for bringing this up! What you described isn't a use-case we had discussed previously.

I think having it change install behavior based on NODE_PRESERVE_SYMLINKS is actually mandatory. npm as part of installation has to model how Node.js would load the package to correctly determine where to put its dependencies. This is a detail we've thus far not had to deal with because npm install has treated symlinks as opaque things it can't know details of.

So yeah, if you have NODE_PRESERVE_SYMLINKS then npm will install transitive dependencies of linked dependencies in your local project, not in the symlinked module. That is, it will work the same way it works when the symlinked module is in a subdirectory of your project.

Member

iarna commented Mar 10, 2017

@mlucool As @zkat says, changing the default Node.js module loader behavior isn't on the table, which is why 'preserve-symlinks' landed as an option.

Thank you for bringing this up! What you described isn't a use-case we had discussed previously.

I think having it change install behavior based on NODE_PRESERVE_SYMLINKS is actually mandatory. npm as part of installation has to model how Node.js would load the package to correctly determine where to put its dependencies. This is a detail we've thus far not had to deal with because npm install has treated symlinks as opaque things it can't know details of.

So yeah, if you have NODE_PRESERVE_SYMLINKS then npm will install transitive dependencies of linked dependencies in your local project, not in the symlinked module. That is, it will work the same way it works when the symlinked module is in a subdirectory of your project.

@zkat zkat referenced this pull request Apr 4, 2017

Closed

release: npm@5.0.0 #16244

16 of 24 tasks complete

@iarna iarna changed the base branch from latest to release-next Apr 6, 2017

@benlangfeld

This comment has been minimized.

Show comment
Hide comment
@benlangfeld

benlangfeld Apr 11, 2017

@iarna This looks great, and would resolve a large number of headaches for my team using a monorepo. What can I do to help this make progress?

@iarna This looks great, and would resolve a large number of headaches for my team using a monorepo. What can I do to help this make progress?

@zkat

This comment has been minimized.

Show comment
Hide comment
@zkat

zkat Apr 11, 2017

Member

@benlangfeld not much right now. We're working towards an implementation now. You can track #16244, which is what this is going to be included in.

Member

zkat commented Apr 11, 2017

@benlangfeld not much right now. We're working towards an implementation now. You can track #16244, which is what this is going to be included in.

@benlangfeld

This comment has been minimized.

Show comment
Hide comment
@benlangfeld

benlangfeld Apr 11, 2017

Ok, thank you @zkat. If there's anything you can make use of warm bodies for to help with this, you know where we are :)

Ok, thank you @zkat. If there's anything you can make use of warm bodies for to help with this, you know where we are :)

@iarna

This comment has been minimized.

Show comment
Hide comment
@iarna

iarna Apr 12, 2017

Member

@benlangfeld Yeah, it's my current task. =) The actual implementation isn't too tricky, most of the work has been in tests.

Member

iarna commented Apr 12, 2017

@benlangfeld Yeah, it's my current task. =) The actual implementation isn't too tricky, most of the work has been in tests.

@pflannery pflannery referenced this pull request May 2, 2017

Closed

Support for Git #13

@iarna iarna changed the base branch from release-next to zkat/pacote May 4, 2017

iarna added a commit that referenced this pull request May 4, 2017

filespecs: Implment new file: specifier behavior
PR-URL: #15900
Credit: @iarna

actions: Allow actions to return promises

finalize: Rewrite finalize as promises

update-linked: Remove update-linked action

install: Remove obsolete invalid action filtering

fetch-package-metadata: Error on installing windows paths on non-windows systems

fetch-package-metadata: Read in modules installed inside of new links

finalize: Act on realpaths because we may be installing inside a link prior to the symlink being made

finalize: Create symlinks of directory deps

deps: Resolve ambiguity for file specifiers in the traditional way

deps: Set link and realpath properties for directory deps

inflate-bundled: Distinguish published bundles from lined modules

node: Add new fromLink attribute to track sourced-to-symlinks deps

diff-trees: Don't try to install deps that are already inside a link

fetch-package-metadata: Improve error messages for link failures

install: Run preinstall lifecycle after finalize

extract: Stop reading package.json files here

finalize: Start reading package.json files here

decompose-actions: Don't fetch or extract links

deps: Determine if a link matches spec based on where it points

deps: Compute correct save specifier for file spec

deps: Set the isInLink property on new children

deps: When finding the install location, don't walk up out of a symlink unless PRESERVE_SYMLINKS is on

diff-trees: Stop setting isInLink, this is now a first class property

diff-trees: Only exclude children of links from adding if they were already there behind the symlink

inflate-bundled: We are using isInLink now not fromLink

inflate-bundled: realpaths should be built on realpaths

node: Eliminate fromLink as a thing

node: Fill in values for inLink, isInLink & fromBundle

save: When updating a lock/shrinkwrap don't read the damn tree again

shrinkwrap: Fill in version per the new shrinkwrap spec

install: Make _inBundle purely a debugging artifact

inflate-shrinkwrap: Fix how bundle deps are inflated

iarna added a commit that referenced this pull request May 4, 2017

filespecs: Implment new file: specifier behavior
PR-URL: #15900
Credit: @iarna

actions: Allow actions to return promises

finalize: Rewrite finalize as promises

update-linked: Remove update-linked action

install: Remove obsolete invalid action filtering

fetch-package-metadata: Error on installing windows paths on non-windows systems

fetch-package-metadata: Read in modules installed inside of new links

finalize: Act on realpaths because we may be installing inside a link prior to the symlink being made

finalize: Create symlinks of directory deps

deps: Resolve ambiguity for file specifiers in the traditional way

deps: Set link and realpath properties for directory deps

inflate-bundled: Distinguish published bundles from lined modules

node: Add new fromLink attribute to track sourced-to-symlinks deps

diff-trees: Don't try to install deps that are already inside a link

fetch-package-metadata: Improve error messages for link failures

install: Run preinstall lifecycle after finalize

extract: Stop reading package.json files here

finalize: Start reading package.json files here

decompose-actions: Don't fetch or extract links

deps: Determine if a link matches spec based on where it points

deps: Compute correct save specifier for file spec

deps: Set the isInLink property on new children

deps: When finding the install location, don't walk up out of a symlink unless PRESERVE_SYMLINKS is on

diff-trees: Stop setting isInLink, this is now a first class property

diff-trees: Only exclude children of links from adding if they were already there behind the symlink

inflate-bundled: We are using isInLink now not fromLink

inflate-bundled: realpaths should be built on realpaths

node: Eliminate fromLink as a thing

node: Fill in values for inLink, isInLink & fromBundle

save: When updating a lock/shrinkwrap don't read the damn tree again

shrinkwrap: Fill in version per the new shrinkwrap spec

install: Make _inBundle purely a debugging artifact

inflate-shrinkwrap: Fix how bundle deps are inflated

iarna added a commit that referenced this pull request May 4, 2017

filespecs: Implment new file: specifier behavior
PR-URL: #15900
Credit: @iarna

actions: Allow actions to return promises

finalize: Rewrite finalize as promises

update-linked: Remove update-linked action

install: Remove obsolete invalid action filtering

fetch-package-metadata: Error on installing windows paths on non-windows systems

fetch-package-metadata: Read in modules installed inside of new links

finalize: Act on realpaths because we may be installing inside a link prior to the symlink being made

finalize: Create symlinks of directory deps

deps: Resolve ambiguity for file specifiers in the traditional way

deps: Set link and realpath properties for directory deps

inflate-bundled: Distinguish published bundles from lined modules

node: Add new fromLink attribute to track sourced-to-symlinks deps

diff-trees: Don't try to install deps that are already inside a link

fetch-package-metadata: Improve error messages for link failures

install: Run preinstall lifecycle after finalize

extract: Stop reading package.json files here

finalize: Start reading package.json files here

decompose-actions: Don't fetch or extract links

deps: Determine if a link matches spec based on where it points

deps: Compute correct save specifier for file spec

deps: Set the isInLink property on new children

deps: When finding the install location, don't walk up out of a symlink unless PRESERVE_SYMLINKS is on

diff-trees: Stop setting isInLink, this is now a first class property

diff-trees: Only exclude children of links from adding if they were already there behind the symlink

inflate-bundled: We are using isInLink now not fromLink

inflate-bundled: realpaths should be built on realpaths

node: Eliminate fromLink as a thing

node: Fill in values for inLink, isInLink & fromBundle

save: When updating a lock/shrinkwrap don't read the damn tree again

shrinkwrap: Fill in version per the new shrinkwrap spec

install: Make _inBundle purely a debugging artifact

inflate-shrinkwrap: Fix how bundle deps are inflated
@donaldpipowitch

This comment has been minimized.

Show comment
Hide comment
@donaldpipowitch

donaldpipowitch Jul 10, 2017

Okay, thank you both.

Okay, thank you both.

@shadowmint

This comment has been minimized.

Show comment
Hide comment
@shadowmint

shadowmint Jul 10, 2017

@iarna

This comment has been minimized.

Show comment
Hide comment
@iarna

iarna Jul 14, 2017

Member

@donaldpipowitch

Will npm support a feature similar to yarn's workspaces natively in the future? Is there an issue which I could track? I can't find one.

Yes, no issue as yet, I need to translate their RFC into something that makes sense more broadly (it was rather yarn specific, to the point that it was hard to follow as an implementor w/o deep yarn knowledge). But once I do, we'll track that in new RFC issue. Assuming they haven't strayed from the RFC they shared with us a few months back, I anticipate our implementation being compatible. The idea is that tools like lerna will get to defer more of the thinking about how things install to the package managers.

At its heart it's pretty simple: workspaces property of package.json, an array of globs. For all folders that match the globs, install their deps when running npm install at the top level and cross-link them when they request any version of a module w/ the same name.

That's currently scheduled for after fixing npm link's behavior as it relates to file dependencies.

Member

iarna commented Jul 14, 2017

@donaldpipowitch

Will npm support a feature similar to yarn's workspaces natively in the future? Is there an issue which I could track? I can't find one.

Yes, no issue as yet, I need to translate their RFC into something that makes sense more broadly (it was rather yarn specific, to the point that it was hard to follow as an implementor w/o deep yarn knowledge). But once I do, we'll track that in new RFC issue. Assuming they haven't strayed from the RFC they shared with us a few months back, I anticipate our implementation being compatible. The idea is that tools like lerna will get to defer more of the thinking about how things install to the package managers.

At its heart it's pretty simple: workspaces property of package.json, an array of globs. For all folders that match the globs, install their deps when running npm install at the top level and cross-link them when they request any version of a module w/ the same name.

That's currently scheduled for after fixing npm link's behavior as it relates to file dependencies.

@iarna

This comment has been minimized.

Show comment
Hide comment
@iarna

iarna Jul 14, 2017

Member

The spec here actually landed in e563d61, so I'm going to close this PR. I encourage folks looking for further discussion around this implementation to open a new issue or comment on an existing one.

Member

iarna commented Jul 14, 2017

The spec here actually landed in e563d61, so I'm going to close this PR. I encourage folks looking for further discussion around this implementation to open a new issue or comment on an existing one.

@iarna iarna closed this Jul 14, 2017

@donaldpipowitch

This comment has been minimized.

Show comment
Hide comment
@donaldpipowitch

donaldpipowitch Jul 14, 2017

Thank you for the explanation!

Thank you for the explanation!

hallettj added a commit to PoodleApp/poodle that referenced this pull request Jul 17, 2017

Remove lerna - we will use `link:` specifiers instead
I tried to get lerna working with hoisting, and un-published packages;
but I could not resolve all the issues. The only thing that we really
need mono-repo support for is hoisting certain dependencies - React in
particular fails if multiple copies are loaded. We can work around that
problem by installing common dependencies at the top-level.

For more information on the `link:` specifier see:

npm/npm#15900
@UnrememberMe

This comment has been minimized.

Show comment
Hide comment
@UnrememberMe

UnrememberMe Aug 24, 2017

FYI, the current implementation breaks one of the mono-repo scenario as discovered in the Pants issue linked above.

The scenario is:

  1. inside of a mono-repo, we created a wrapper for a commonly used tool, say webpack.
  2. include the wrapper module as a dependency on another module and trying to npm run the wrapper module to invoke webpack.

This scenario works in npm3 because webpack as a dependency for the wrapper module will be installed in node_module/.bin. For npm5, a symlink will be created for the wrapper module and when the wrapper trying to invoke webpack, webpack will not be on $PATH.

FYI, the current implementation breaks one of the mono-repo scenario as discovered in the Pants issue linked above.

The scenario is:

  1. inside of a mono-repo, we created a wrapper for a commonly used tool, say webpack.
  2. include the wrapper module as a dependency on another module and trying to npm run the wrapper module to invoke webpack.

This scenario works in npm3 because webpack as a dependency for the wrapper module will be installed in node_module/.bin. For npm5, a symlink will be created for the wrapper module and when the wrapper trying to invoke webpack, webpack will not be on $PATH.

@dotchev

This comment has been minimized.

Show comment
Hide comment
@dotchev

dotchev Aug 25, 2017

What is the reason for this? Who else could install those packages and why not reuse them?
For example we use a private npm registry and don't want to reveal its location in _resolved property in package.json. See #17934.

What is the reason for this? Who else could install those packages and why not reuse them?
For example we use a private npm registry and don't want to reveal its location in _resolved property in package.json. See #17934.

@viankakrisna viankakrisna referenced this pull request Aug 29, 2017

Merged

run npm 5.4.0 in CI #3026

brendankenny added a commit to GoogleChrome/lighthouse that referenced this pull request Sep 25, 2017

fix(npm): remove lighthouse-logger/ from npm package
since lighthouse-logger is now its own package. Also update CONTRIBUTING on local testing of the npm package due to `npm install <file>` now just creating a symlink (npm/npm#15900) instead of a real install
@Basel78

This comment has been minimized.

Show comment
Hide comment
@Basel78

Basel78 Sep 30, 2017

Hello,
I have LibA with ModuleA and LibB with ModuleB that use ModuleA from LibA
also I have Project that use Both LibA & LibB
how best to import LibA in LibB and both in Project if I am using the following versions
LibA & LibB both are angular4
Project is angular-cli
What I want to achieve here is to avoid the relative path of module in my code,
1- LibA as external to LibB so it wont be imported once once bundle the libraries,
2- LibA & LibB as externals and will only install them in Project.

BTW: I used symlink first to reference LibA in LibB but was only able to import ModuleA in relative path whivh what i want to avoid it, since that would mean n Project I would have LibA, and LibB with Liba under node_modules with relative path to ModuleA
and on the Project level will have libA & LibB in node_modules with Import to ModuleA & ModuleB in relative path.
now I am using npm pack and npm install file://path/to/module-a/moduleA-1.2.3.tgz inside LibB
but import { ModuleA } from "LibA"; is giving "error cannot find ModuleA"

also I have noticed that after using the npm pack that the referenced LibA appears in LibB-> node_modules folder but the "d.ts & .ts" files is dist folder does not appear anymore, even though it appear in folder in LibA ->dist.

Basel78 commented Sep 30, 2017

Hello,
I have LibA with ModuleA and LibB with ModuleB that use ModuleA from LibA
also I have Project that use Both LibA & LibB
how best to import LibA in LibB and both in Project if I am using the following versions
LibA & LibB both are angular4
Project is angular-cli
What I want to achieve here is to avoid the relative path of module in my code,
1- LibA as external to LibB so it wont be imported once once bundle the libraries,
2- LibA & LibB as externals and will only install them in Project.

BTW: I used symlink first to reference LibA in LibB but was only able to import ModuleA in relative path whivh what i want to avoid it, since that would mean n Project I would have LibA, and LibB with Liba under node_modules with relative path to ModuleA
and on the Project level will have libA & LibB in node_modules with Import to ModuleA & ModuleB in relative path.
now I am using npm pack and npm install file://path/to/module-a/moduleA-1.2.3.tgz inside LibB
but import { ModuleA } from "LibA"; is giving "error cannot find ModuleA"

also I have noticed that after using the npm pack that the referenced LibA appears in LibB-> node_modules folder but the "d.ts & .ts" files is dist folder does not appear anymore, even though it appear in folder in LibA ->dist.

@UnrememberMe

This comment has been minimized.

Show comment
Hide comment
@UnrememberMe

UnrememberMe Oct 3, 2017

to @dotchev, sorry I did not actively monitor this issue. To answer your question, in a mono-repo, a reasonable setup for a tool might be:

  1. create a wrapper (component A) for a commonly used tool, say eslint / webpack etc and supply some base/common configs, and expose a new command line
    a simple example can be
    mylint.sh =>
... overlay $1 on top of eslint.base.config if any and generate a new combined config
eslint --config=generated_config
  1. other projects (component B) within a mono-repo might reference the wrapper instead of the original tool via a file link and call mylint my-overlay-lint-config.

with npm5, eslint will no longer be in .bin for component B, and invoking mylint.sh directly will fail to execute if running under component B directory.
We can certainly package component A into a private npm registry. However, one advantage of mono-repo is codes can reference their dependencies directly without needing external artifact storages.

to @dotchev, sorry I did not actively monitor this issue. To answer your question, in a mono-repo, a reasonable setup for a tool might be:

  1. create a wrapper (component A) for a commonly used tool, say eslint / webpack etc and supply some base/common configs, and expose a new command line
    a simple example can be
    mylint.sh =>
... overlay $1 on top of eslint.base.config if any and generate a new combined config
eslint --config=generated_config
  1. other projects (component B) within a mono-repo might reference the wrapper instead of the original tool via a file link and call mylint my-overlay-lint-config.

with npm5, eslint will no longer be in .bin for component B, and invoking mylint.sh directly will fail to execute if running under component B directory.
We can certainly package component A into a private npm registry. However, one advantage of mono-repo is codes can reference their dependencies directly without needing external artifact storages.

stuhood added a commit to pantsbuild/pants that referenced this pull request Oct 16, 2017

Add node_module .bin path to node / npm / yarnpkg execution path. (#4932
)

### Problem
Due to change introduced in [npm/npm-15900](npm/npm#15900), source level dependencies are no longer installed, but symlinked directly.  This means we can no longer reference dependencies of dependencies directly any more since they will no longer be on execution path.
Additionally, due to an open npm issue npm/npm#18233, we need to add '--no-optional' in npm install command to workaround temporarily. 

### Solution
The fix is to supply all resolved node modules as a part of execution path. This can potentially have another side effect that is we might end up with two versions of same binaries on the execution path.  I am not sure if there is a good way to resolve this issue at current time, but we should keep an eye on the possibility.

### Result
Travis tests running with node 8.6.0 (npm 5.3.0) passes. 
https://travis-ci.org/pantsbuild/pants/builds/286743908 => Failed on shard 1 with unrelated issues.
@pgonzal

This comment has been minimized.

Show comment
Hide comment
@pgonzal

pgonzal Oct 28, 2017

Hi @iarna,

We're having an issue with the "file://" specifier not being handled correctly by NPM 5. We have an automated tool that generates a package.json with references like this:

"dependencies": {
    "yargs": "~4.6.0",
    "z-schema": "~3.18.3",
    "project1": "file:./projects/project1.tgz",
    "project2": "file:./projects/project2.tgz",

The "project1.tgz" archive contains a package.json file like this:

{
  "name": "project1",
  "version": "0.0.0",
  "private": true,

When NPM 4 installs it, the node_modules/project1/package.json looks like this, as expected:

  "version": "0.0.0"

However, NPM 5 for some reason writes it like this:

  "version": "file:projects/project1.tgz"

Was this an intentional design change? Is it an NPM 5 bug?

It's causing trouble because read-package-tree chokes on this JSON value. We get an error like this:

ERROR: Failed to parse package.json for project1: Invalid version: "file:projects/project1.tgz"

pgonzal commented Oct 28, 2017

Hi @iarna,

We're having an issue with the "file://" specifier not being handled correctly by NPM 5. We have an automated tool that generates a package.json with references like this:

"dependencies": {
    "yargs": "~4.6.0",
    "z-schema": "~3.18.3",
    "project1": "file:./projects/project1.tgz",
    "project2": "file:./projects/project2.tgz",

The "project1.tgz" archive contains a package.json file like this:

{
  "name": "project1",
  "version": "0.0.0",
  "private": true,

When NPM 4 installs it, the node_modules/project1/package.json looks like this, as expected:

  "version": "0.0.0"

However, NPM 5 for some reason writes it like this:

  "version": "file:projects/project1.tgz"

Was this an intentional design change? Is it an NPM 5 bug?

It's causing trouble because read-package-tree chokes on this JSON value. We get an error like this:

ERROR: Failed to parse package.json for project1: Invalid version: "file:projects/project1.tgz"
@pgonzal

This comment has been minimized.

Show comment
Hide comment
@pgonzal

pgonzal Oct 30, 2017

I'm pretty sure this is an NPM bug. I've opened #19006

pgonzal commented Oct 30, 2017

I'm pretty sure this is an NPM bug. I've opened #19006

@robogeek robogeek referenced this pull request Nov 28, 2017

Open

npm install should not create symlinks #18503

1 of 4 tasks complete

dhei added a commit to Microsoft/AppCenter-SDK-React-Native that referenced this pull request Nov 30, 2017

[TestApp] npm package up and install to avoid symlink behavior of `np…
…m install` on npm5

problem: npm4 install local dependencies by copying over the local package but npm5 install local dependencies as symlink. This cause issues for our TestApp that install local appcenter* packages because "Header Search Paths" can't find react native modules.

solution: `npm pack appcenter && npm install appcenter.tgz` will package up and then install the tgz files, which avoid the symlinks. (see npm/npm#15900 (comment))

lkraav added a commit to lkraav/pure that referenced this pull request Dec 15, 2017

hello build/ [1]
* we want to have assets available via simple `npm install`
* npm 5 changed `file:` semantics into `link:`, breaks local installs
* https://www.npmjs.com/package/install-local is cumbersome to use

[1]: npm/npm#15900

lkraav added a commit to lkraav/pure that referenced this pull request Dec 16, 2017

hello build/ [1]
* we want to have assets available via simple `npm install`
* npm 5 changed `file:` semantics into `link:`, breaks local installs
* https://www.npmjs.com/package/install-local is cumbersome to use

[1]: npm/npm#15900

@iarna iarna deleted the link-specifier branch Feb 1, 2018

evocateur added a commit to lerna/lerna that referenced this pull request Feb 13, 2018

Rewrite npm5 'file:' links during publish (#1262)
This helps avoid publishing broken packages when using npm5's new relative link specifiers

npm/npm#15900
@adrian-gierakowski

This comment has been minimized.

Show comment
Hide comment
@adrian-gierakowski

adrian-gierakowski Mar 6, 2018

@iarna what happened to this proposal? has it been rejected?

btw. looks like something similar has been implemented in yarn some time ago

@iarna what happened to this proposal? has it been rejected?

btw. looks like something similar has been implemented in yarn some time ago

@zkat

This comment has been minimized.

Show comment
Hide comment
@zkat

zkat Mar 6, 2018

Member

@adrian-gierakowski it was merged and implemented as part of npm@5.0.0

Member

zkat commented Mar 6, 2018

@adrian-gierakowski it was merged and implemented as part of npm@5.0.0

evocateur added a commit to lerna/lerna that referenced this pull request Mar 8, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.