fix(workspaces): Allow patterns with non-semver ranges to match workspace packages

[rally25rs]

Summary

Previously for a requested dependency pattern to match a package in a workspace, it's name had to
match and so did the semver range. If a request specified a non-semver range (like a github url)
then it would not match, causing the resolver to attempt to resolve the package on the registry
instead of use the workspace package. This change will match a workspace package just on it's name
if no valid semver range was found on the requested pattern.

This also fixes the command yarn workspace @scope/foo add @scope/bar where @scope/bar is a workspace package. The semver range of the request is empty (no version was specified in the command), so it used to not match the package and instead try to find it on the registry. Now it will properly match the workspace package name.

fixes #4878
fixes #3973
affects #5726

Test plan

Added new test file for workspace-layout.

[arcanis]

I don't know how I feel about this change. I have a few remarks:

• How often will someone add a non-npm dependency of a project and expect it to be fetched from the workspace? Why not use a version, then?

• The current algorithm is quite simple: if it matches, it's linked. If it doesn't, it will fetch it from wherever it is requested. This PR adds a significant edge case which I think might be quite confusing and unexpected (to me, at least).

• The yarn workspace @scope/a add @scope/b problem should be resolved by using the workspaces versions before resolving to the npm cache (so @scope/b would, by default, resolve to the version specified in the local workspace). It seems a different problem.

[rally25rs]

@arcanis

How often will someone add a non-npm dependency of a project

in #5726 (comment) @no-more mentions that he is using github refs because it is a set of private packages. That's the only case I've seen. I didn't think much about it at the time, but if they are private repos then they will probably never be published to the registry, so it might be fine to use a version or *. I just assumed that at some point one of the workspace packaged could be used outside the workspace and would need to fetch it's deps from a git url.

This PR adds a significant edge case which I think might be quite confusing and unexpected (to me, at least).

IMO it was confusing (and took me days of debugging through the code to even realize) that workspace packages aren't just matched on their name and even took version into account. It might be worth adding something about that limitation in the docs.

The yarn workspace @scope/a add @scope/b problem should be resolved by using the workspaces versions

If we ditch this PR then we could solve this a different way.

[eropple]

How often will someone add a non-npm dependency of a project and expect it to be fetched from the workspace? Why not use a version, then?

Hi. I just found this and it bit me really hard. Eventually packages are pushed to a private NPM registry, but during development I use a Yarn workspace to wire them up. My typical convention is to use versions with -wip at the end, i.e. 0.1.0-wip, for internal development; packages are then renamed over time, i.e. 0.1.0-alpha.1, 0.1.0-rc.1, etc. before being released as 0.1.0.

Yarn throwing a rod at this is a really big bummer. I'd hope that the PR under consideration gets some love.

[kjs3]

I think this is exactly what my team needs. We publish to some private github repos (never to any NPM registry) but want to have everything together during dev.

[bendemboski]

I'm in exactly the same situation as @kjs3 and the absence of this is currently blocking me from using workspaces, as I haven't found a workaround...

[bendemboski]

I didn't think much about it at the time, but if they are private repos then they will probably never be published to the registry, so it might be fine to use a version or *.

@rally25rs I'm not quite sure how this would work...say I have a workspace with project-a that depends on unpublished dep-b (also in the workspace), pulled from a git URL. If I modify project-a's package.json to specify "dep-b": "*" then it works fine in the workspace. But if I then try to use project-a not in a workspace (like in CI), it can't resolve dep-b with version * because it's un-published. I even tried adding "dep-b": "<git url>" to resolutions in project-a's package.json, but no dice -- outside the workspace I get

error An unexpected error occurred: "https://registry.yarnpkg.com/dep-b: Not found".

Am I missing something? I'm happy to specify * as the version, but need it to still work outside of workspaces (because I really don't want to have to modify my package.json when working in a workspace and always remember not to check it in).

[kjs3]

@bendemboski: Not exactly a completely open-ended "*" but I talk about that here in 6027.
A token in the GitHub URL allows it to work anywhere.

[rally25rs]

@bestander IIRC this issue is what I tried to PM you on Discord about.

[arcanis]

I still don't think it's the right solution, fwiw. Imo, a better fix might be to use a resolution field to force the dependency ranges to resolve to the workspace version rather than github.

[bestander]

I agree that git dependencies are a common thing and we should address this.

I think resolution field would work but it is too hacky and not obvious, matching for semver validity makes more sense to me

[arcanis]

Yep, but you have no way to know the version of the remote dependency until you bring it to the disk (at least not without querying the network, which is not ideal since - on top of the obvious problems with --offline - it would either require to be resolved each and every time, or to be persisted somewhere, like in the lockfile).

[bestander]

Well this PR does not specifically enable network access, it just relaxes semver validity if users did not specify a correct semver limit.
It does not seem like it is a risky change but would give some flexibility for the ecosystem.
So this one is +1 from me.
But I'll leave the decision to @arcanis

[batusai513]

@arcanis @rally25rs Any updates on this one?

[mdgbayly]

Not fully versed in all the subtleties of the issue and the proposed solution but it feels to me like there is an inconsistency in how yarn is handling non-standard semver ranges like github urls/references when it comes to workspace packages.

e.g. outside of workspaces, we can add versions that resolve to github urls like this to two different packages and yarn will resolve/dedupe this to most recent package.

component-a/package.json
"my-package": "myorg/my-package#^2.0.11"

component-b/package.json
"my-package": "myorg/my-package#^2.0.10"

So it was extremely confusing when we tried to use my-package as a workspace package but it never gets linked.

[toddbluhm]

Would really like to see this or a similar solution get merged. My company would like to move to yarn and workspaces is one of the main features we want to use, but unfortunately this issue is holding us up. All of our packages are privately connected via git urls + tokens and we do not publish to any private or public npm registry. So without this,workspaces will not work for us and is stopping us from migrating over.

[rally25rs]

I just ran smack into this issue on a new project. I thought it might be a decent idea to have a monorepo that consists of git submodules for each package.

Each package is maintained by a separate dev team, so in order to let them have their own Github PR and issues tracking, the workspace projects are separate github repos.

The problem becomes these teams obviously want to have their own CI and test process, but to do that the dependencies would need to be resolved from github URLs when it's run outside the monorepo (which doesn't work because of this issue with semver resolution).

Otherwise the CI server would have to checkout the entire monorepo and all subprojects, then flip one of them to the PR branch in order to be built/tested.

[kwelch]

I believe I ran into this by trying to have an alpha version as one of the workspace values and then in a different workspace linking it as *. I was unable to install the package.

However, when i switched * to the version in the other package it worked as expected. Hopefully that helps someone out there.

[ericwooley]

Just wanted to bump this.

#3973 has been an issue literally for years, and this PR has been open for (less) years.

However this is solved, it needs to be solved at some point.

We aren't using workspaces to publish packages, we are just using them to break up main app into modules, so we don't have to do a 20 minute rebuild every time.

Because of this issue, we have to manually add packages and versions to a workspace package.json, and it's super tedious, and lowers my teams confidence in yarn workspaces.

I'm currently considering publishing empty packages to work around this. Which is just silly.

[musicin3d]

Why hasn't this been reviewed yet?

It seems like there is no way to depend on another package in the same monorepo/workspace without publishing. Am I missing something?

Edit: I have discovered a solution for my case. I am including a private "common" package from another private package in the same monorepo. Specify "*" version in the dependent's dependency list, and add any version to the dependency itself.