Injected dependencies are not recopied after a workspace project is rebuilt

[octogonz]

pnpm version: 6.32.2

The real world scenario at my work involves internal company NPM packages with a relationship like this:

A --depends-on--> B --peer-depends-on--> C

...where A and C are in the monorepo, but B is developed in a separate repo and gets installed from an Artifactory private registry. This is a common situation at a big company if most teams work in a monorepo, but we share some packages with other teams who have their own repo.

Code to reproduce the issue:

This is an isolated minimal repro. I substituted @react-navigation/core to avoid involving private packages.

1. Clone this repository: https://github.com/octogonz/pnpm-issue-4407
2. pnpm install

In this repro:

• test-project depends on @react-navigation/core
• @react-navigation/core has a peer dependency on "react": "*"
• test-project also depends on "react": "workspace:*"
• The local workspace project react has version 16.0.0 which should satisfy the peer dependency

Actual behavior:

Progress: resolved 10, reused 10, downloaded 0, added 10, done
 WARN  Issues with peer dependencies found
test-project
└─┬ @react-navigation/core
  └── ✕ missing peer react@"*"
Peer dependencies that should be installed:
  react@"*"

Expected behavior:

The installation should succeed. PNPM should satisfy react@* by creating a symlink to the local workspace package.

Additional information:

• node -v prints: v14.17.4
• Windows, macOS, or Linux?: Windows

[octogonz]

More weirdness

After completing steps 1 and 2 above, try this:

3. rm pnpm-lock.yaml
4. rm -Rf node_modules (in the repository root, but not under the project folders)
5. pnpm install

Actual behavior:

This time, the installation completes without any errors/warnings:

Scope: all 2 workspace projects
Packages: +10
++++++++++
Packages are hard linked from the content-addressable store to the virtual store.
  Content-addressable store is at: D:\.pnpm-store\v3
  Virtual store is at:             node_modules/.pnpm
Progress: resolved 10, reused 10, downloaded 0, added 10, done

Expected behavior:

If step 2 printed an error, then so should step 5. I would expect that deleting pnpm-lock.yaml is sufficient to reset the installation. (?)

The error does not reappear unless you delete BOTH pnpm-lock.yaml AND test-project/node_modules/**

[octogonz]

We were able to work around this problem by using the dependenciesMeta.*.injected feature like this:

{
  "name": "test-project",
  "version": "0.0.0",
  "dependencies": {
    "@react-navigation/core": "^5.15.0",
    "react": "workspace:*"
  },
  "dependenciesMeta": {
    "react": { "inject": true }
  }
}

However this does not seem like a correct solution. The documentation for that feature sounds like it is written for a different problem and I didn't understand what is doing technically-- is it actually correctly satisfying the versions?

And practically speaking, it would be awkward to ask every developer to remember to do this for every peer dependency. If it's a correct solution, why can't PNPM do this automatically so that the installation behavior matches people's intuitive understanding of how you satisfy a peer dependency?

[zkochan]

Yes, it is correct to use the injected feature in this case.

[octogonz]

Hmmm... this worked in my test project (from the repro steps), but it's not working in our real repo.

The pnpm-issue-4407/test-project/node_modules/react folder is symlink pointing to the physical pnpm-issue-4407/react/ folder. 👍

Whereas when I use the real NPM packages, test-project\node_modules\@hbo\hbomax-core is a symlink pointing to C:\MyRepo\node_modules\.pnpm\local+C++MyRepo+hbomax-core\node_modules\@hbo\hbomax-core\ which contains a copy of the hbomax-core workspace project folder. Because it's a copy, it is missing the build outputs after the hbomax-core project has been compiled. 👎

I don't understand why "inject": true is producing different results in these two setups. Unfortunately I don't have time right now to narrow down the difference. (The packages are internal to our company, but if it helps I could privately share the generated pnpm-lock.yaml outputs.)

[zkochan]

Because it's a copy, it is missing the build outputs after the hbomax-core project has been compiled.

pnpm syncs the contents of injected dependencies after running their "prepare" or "postinstall" scripts.

[octogonz]

@zkochan I am confused.

Let's go back to my original example:

A --depends-on--> B --peer-depends-on--> C

A and C are workspace projects, whereas B is installed from the registry.

If A satisfies the peer by depending on "C": "workspace:*", then it seems that B's installation needs a direct symlink to the C:\MyRepo\C\ project folder. That way, whenever I recompile C, the latest C:\MyRepo\C\lib\*.js outputs should be available to B.

pnpm syncs the contents of injected dependencies after running their "prepare" or "postinstall" scripts.

The prepare and postinstall events happen during publishing/installation. Whereas the relevant event here is npm run build for project C. This seems to require a direct symlink. I don't see how it can work with an installation method that copies the C folder.

[octogonz]

I was able to sort of solve it by rewriting the peerDependencies to be regular dependencies:

.pnpmfile.cjs

function readPackage(pkg, context) {
  for (let workspaceProject of [
    "@hbo/hbomax-core",
    "@hbo/hbomax-localization-policy",
    "@hbo/hbomax-localization-policy-config",
  ]) {
    if (pkg.peerDependencies && pkg.peerDependencies[workspaceProject]) {
      console.log("Rewrite " + pkg.name + ": " + workspaceProject);

      delete pkg.peerDependencies[workspaceProject];
      if (!pkg.dependencies) {
        pkg.dependencies = {};
      }
      pkg.dependencies[workspaceProject] = "workspace:*";
    }
  }
  return pkg;
}

module.exports = { hooks: { readPackage } };

So whereas the old inject: true approach produced file: relationships:

  /@hbo/hurley-global-policies/4.2.0_76e775d50468c213fc9eed97bf9a4978:
    peerDependencies:
      '@hbo/hbomax-core': ^4.0.0
      '@hbo/hbomax-localization-policy': ^2.0.0
      '@hbo/hbomax-localization-policy-config': ^1.0.0
    dependencies:
      '@hbo/hbomax-core': file:hbomax-core
      '@hbo/hbomax-localization-policy': file:hbomax-localization-policy
      '@hbo/hbomax-localization-policy-config': file:hbomax-localization-policy-config
    transitivePeerDependencies:
      - supports-color
    dev: false

...the .pnpmfile.cjs hack instead produces link: relationships:

  /@hbo/hurley-global-policies/4.2.0:
    dependencies:
      '@hbo/hbomax-core': link:hbomax-core
      '@hbo/hbomax-localization-policy': link:hbomax-localization-policy
      '@hbo/hbomax-localization-policy-config': link:hbomax-localization-policy-config
    transitivePeerDependencies:
      - supports-color
    dev: false

Maybe what I'm proposing is this:

When a peer dependency is satisfied using a "workspace:*" dependency, it should be installed using a direct link: to the workspace folder.

[dmichon-msft]

I believe the best solution here would be to have some pnpm sync-injected or other command that, when executed, "publishes" the specified package (ideally mimicking exactly the output of a proper publish) into the node_modules folder at all locations where it is "injected". Then ensuring it is synced at the right stage of the build is simply the duty of the build orchestrator (Rush in this case) to run the command between building it and building projects that depend on it.

This would provide a general solution for multiple classes of problems:

• External dependencies that declare peer dependencies on packages in the workspace (this issue)
• Referring to a local workspace package with differing peer dependency configurations (what "injected: true" was made for, to the best of my knowledge)
• Ensuring that local workspace projects can't access files from dependencies that they wouldn't have if they lived in an external repository (this problem comes up a lot and is my interest in going through the publish machinery as opposed to just cloning everything in the folder)

This also makes monorepo builds easily distributable because building in the monorepo looks no different than building a single package in isolation with a bunch of tarballs for its dependencies.

[octogonz]

Update: We have put together a solution for a pnpm-sync command that can solve the problem of updating injected dependencies during the build. In summary, this command would be invoked whenever a workspace project is rebuilt, and it will copy the build outputs into the node_modules folder to sync up the injected installations.

@g-chao has implemented a proof-of-concept prototype here:

https://github.com/g-chao/pnpm-sync-prototype

The mechanics are a bit tricky to ensure the command is run at the right times (including the watch mode scenario). Our plan is to implement it first for Rush workspaces, since Rush imposes stricter constraints about how pnpm install is performed. Then if successful, we will propose to contribute this back as an integrated feature of PNPM.

Please share questions/suggestions about by commenting in this GitHub issue. Thanks!

@chengcyber @dmichon-msft @iclanton @zkochan