Support for Yarn workspaces/monorepos

[dinvlad]

First of all, thanks for the awesome work on this package!

I've started exploring the new "workspaces"/"monorepos" feature of Yarn (https://yarnpkg.com/blog/2017/08/02/introducing-workspaces/), which enables "hoisting" of all node_modules from any sub-projects to a single parent repository (so that if you have sub-folders with individual package.jsons, each one would install its deps in the "parent" node_modules and link them, unless there are version conflicts).

However, patch-package no longer automatically applies patches to sub-project packages in its default configuration (i.e. when I run yarn in the parent folder, it only applies patches/ from the "parent" folder). I had to move all patches from sub-projects into the "parent" patches/ to make it work, however that sort of "pollutes" the "global" patch space with patches from sub-projects.

Are you familiar with this feature, and are there any other (better) solutions to enable automatic patching of sub-projects (I know it's a lot to ask, was just hoping to hear your thoughts)?

Thanks again!

[ds300]

Hi! Sorry it took me so long to respond to this.

and are there any other (better) solutions to enable automatic patching of sub-projects (I know it's a lot to ask, was just hoping to hear your thoughts)?

Not that I know of.

I had to move all patches from sub-projects into the "parent" patches/ to make it work, however that sort of "pollutes" the "global" patch space with patches from sub-projects.

The alternative, however, would be prone to conflicts. imagine if you had sub-projects A and B, then A decides to patch left-pad@1.1.2 in one way, while B decides to patch left-pad@1.1.2 in a different way.

I think in order for this to work in a way that allows independent patches for the same package version, yarn would need to either be aware of the patch files, or make module resolution (or, specifically, the conflict detection part of module resolution) pluggable. Then it could install separate versions in sub-projects' own node_modules directory.

Otherwise, patch-package should be able to include support for workspaces where this kind of conflict is prohibited. I'm happy to provide guidance for anyone who is willing to work on it. I probably won't get around to it myself any time soon, although I certainly recognise the need for the feature.

Thanks for binging this up!

[dinvlad]

Ok, that makes sense, thanks for getting back on it! I appreciate the amount of effort it would take to solve (if at all possible), and that's not really needed because the workaround is more reliable and simple to set up at the moment (I have now been using it for a week without issues).

The one case this might be simple is when each sub-project depends on its own version of a package, in which case they'd be both installed in the sub-project directories instead of the root one (but then again, this likely works already, I just haven't tested it).

Feel free to keep this issue open for others to weigh in, or close it if you prefer. Thanks again!

[wardpeet]

@ds300 would it be an option if we add the version of the npm package to the patch file so if multiple versions are found it can still find the the correct package but only if there are multiple packages found. If patch-package can not resolve to a correct version it just throws an error?

Could this be a good approach?

[vieira]

I am having trouble to create patches when using yarn workspaces. From the "parent" folder I execute for instance yarn patch-package hoxy (which is a dependency of a child project and hoisted in the parent) and get:

yarn run v1.5.1
$ /Users/vieira/Code/platform/node_modules/.bin/patch-package hoxy
☑ Removing existing patches/hoxy+3.2.2.patch
☑ Creating temporary folder
☑ Building clean node_modules with yarn
☑ Diffing your files with clean files
{ Error: ENOENT: no such file or directory, unlink '/var/folders/wf/stx0qyms4319dzf06sggssjm0000gn/T/tmp-1868HXBrSWj4hTiA/node_modules/hoxy/package.json'
    at Object.fs.unlinkSync (fs.js:1085:18)
    at Object.makePatch [as default] (/Users/vieira/Code/platform/node_modules/patch-package/dist/makePatch.js:69:12)
    at /Users/vieira/Code/platform/node_modules/patch-package/dist/index.js:26:32
    at Array.forEach (<anonymous>)
    at Object.<anonymous> (/Users/vieira/Code/platform/node_modules/patch-package/dist/index.js:25:22)
    at Module._compile (module.js:662:30)
    at Object.Module._extensions..js (module.js:673:10)
    at Module.load (module.js:575:32)
    at tryModuleLoad (module.js:515:12)
    at Function.Module._load (module.js:507:3)
  errno: -2,
  code: 'ENOENT',
  syscall: 'unlink',
  path: '/var/folders/wf/stx0qyms4319dzf06sggssjm0000gn/T/tmp-1868HXBrSWj4hTiA/node_modules/hoxy/package.json' }
/Users/vieira/Code/platform/node_modules/patch-package/dist/makePatch.js:113
        throw e;
        ^

Error: ENOENT: no such file or directory, unlink '/var/folders/wf/stx0qyms4319dzf06sggssjm0000gn/T/tmp-1868HXBrSWj4hTiA/node_modules/hoxy/package.json'
    at Object.fs.unlinkSync (fs.js:1085:18)
    at Object.makePatch [as default] (/Users/vieira/Code/platform/node_modules/patch-package/dist/makePatch.js:69:12)
    at /Users/vieira/Code/platform/node_modules/patch-package/dist/index.js:26:32
    at Array.forEach (<anonymous>)
    at Object.<anonymous> (/Users/vieira/Code/platform/node_modules/patch-package/dist/index.js:25:22)
    at Module._compile (module.js:662:30)
    at Object.Module._extensions..js (module.js:673:10)
    at Module.load (module.js:575:32)
    at tryModuleLoad (module.js:515:12)
    at Function.Module._load (module.js:507:3)
error An unexpected error occurred: "Command failed.
Exit code: 1
Command: sh
Arguments: -c /Users/vieira/Code/platform/node_modules/.bin/patch-package hoxy
Directory: /Users/vieira/Code/platform
Output:
".
info If you think this is a bug, please open a bug report with the information provided in "/Users/vieira/Code/platform/yarn-error.log".
info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.

If I manually generate the patch and put it in the patches folder, it correctly applies later on yarn install, but the generation does not seem to work.

Is it working for you guys?

Thanks!

[dinvlad]

@vieira it cannot be applied from the parent folder if its node_modules is in a child folder. You'll need to apply it from a child's folder then.

[tonyxiao]

I'm getting a similar issue running from parent folder.

➜  beam-backend git:(plaid-to-synapse) ✗ yarn patch-package @types/luxon
yarn run v1.5.1
$ /Users/tony/dev/src/github.com/beamf/beam-backend/node_modules/.bin/patch-package @types/luxon
☑ Creating temporary folder
☑ Building clean node_modules with yarn
error An unexpected error occurred: "Couldn't find any versions for \"@beamf/db\" that matches \"^0.0.1\"".

It's not recognizing that @beamf/db is a local monorepo module.

[wardpeet]

it doesn't support it yet.

[tonyxiao]

@ds300 would you provide some pointers on how someone might be able to work on a patch for this particular issue?

[ds300]

Sure!

There are two workflows that need to be changed: Creating patches (makePatch.ts), and applying patches (applyPatches.ts).

• When creating the patch file, patch-package uses your project's lockfile to build an exact reproduction of your node_modules folder in a temporary location. With monorepos, it would be necessary to either:

  • Copy the entire monorepo to a tmp dir and reproduce the entire build. 😓
  • Use require.resolve to target the exact location where the files to be diffed are, and only reproduce that part of the tree somehow. It's potentially the case that patch-package wouldn't even need to know whether it's operating on a monorepo in this case

  I honestly don't know which of these options makes more sense going forward. Certainly the second one would provide better UX if it proves to be a robust method, and is therefore the one I'd spike first.

• Similarly, when patches are applied, patch-package currently assumes that the files to be patched are under <repoRoot>/<appRoot>/node_modules when in fact it could be just <repoRoot>/node_modules. I think detecting the appropriate target using require.resolve would work again here.

If you need any more detail on code specifics, let me know. I'd recommend branching off of #45 for now. I'm probably going to merge that into master soon and make smaller PRs for the bits that remain to get to a 6.0 release. Might take a few weeks, and would be super happy to help get this feature in too.

[ds300]

ooh, another thing is that patches should always be kept at the same level as the node_modules folder which contains the code that they'll be patching. So for a monorepo it might be like this:

<repoRoot>
  node_modules/
    react/
  patches/
    react+16.3.2.patch
  my-lovely-app/
    node_modules/
      left-pad/
    patches/
      left-pad+1.1.3.patch

I might even consider changing patches to node_patches to reinforce the relationship.

[dinvlad]

Currently, the only way to make patching work with Yarn seems to be to keep all patches under the root patches/ folder. Otherwise, patch-package complains about a missing yarn.lock in a daughter folder.

On top of that, I can only create patches with diff -u ... ... > <root>/patches/<package_name>+<package_version>.patch, otherwise there's the dreaded no such file or directory.

Finally, I install patch-package only in the root package.json, and postinstall-prepare at the root and in any of the daughter folders. In their respective package.jsons I add "prepare": "cd .. && yarn patch-package" under "scripts" (along with "prepare": "patch-package" in the root package.json)

With these workarounds, everything else works nicely on @beta: installing any package at the root or any of the daughter folders results in successful (re-)application of patches at the <root>/node_modules

[wardpeet]

@ds300 I'll take a stab at this. I can test this on a few repos

[ds300]

@dinvlad wow! I'm amazed you've managed to carve a workflow out of the darkness :-) Thanks for the details.

@wardpeet Sweet! 🙌 keep me posted