Allow passing a custom glob to find packages

[adambrgmn]

Affected Packages

• @changesets/cli

Problem

I'm trying to advocate at work to use changesets as a tool to automate the release of packages that we have. Today the release process is very manual and cumbersome. But I have an issue – we work in a monorepo but we don't follow the setup of either yarn, lerna, pnpm or bolt. Instead we use nx. It is a rather special setup, I realize that. But a tweak to the changesets cli would enable us to actually use it.

The thing with nx is that we have a single package.json in the root with all dependencies. Then we have placeholder package.json's in the libraries that we want to publish, with only metadata such as name and version. And when packaging the packages into publishable versions they end up in a dist-folder, with a package.json and everything.

That means we would like to run add and version agains one set of packages. And then publish against what's in dist.

This doesn't work with changesets for two reasons:

• @manypkg/get-packages can't determine that we use an nx repo setup and aren't able to find our packages
• The packages we want to publish end up in a different directory than the source code; libs/package-a -> dist/libs/package-a

Proposed solution

First I thought of making a PR against @manypkg/get-packages. But I realize that nx-repos function very differently than others which means it is not suitable for that project.

I have a working solution locally. The proposed solution is to be able to pass a --packages=<glob> flag to the add, version, publish and status cli commands.

Then the workflow for an nx monorepo would be something like this:

1. Changes done in package-a
2. Create changesets for package

   yarn changeset --packages="libs/*"

3. Create and merge PR into main
4. Changeset github action kicks in with this config:

   uses: changesets/action@master
   with:
     version: |
       yarn changeset version --packages="libs/*"
     publish: |
       yarn build
       yarn changeset publish --packages="dist/libs/*"

• yarn changeset version will check for the source libs so that the changelog and everything ends up where it should
• yarn build packages the module into the dist folder, with distributable files and a package.json and everything
• then yarn changeset publish will publish the packages in the dist folder

As I said I have a rough solution locally and I would be happy to polish it and supply a PR if this is a feature that you are interested in.

[adambrgmn]

After looking into the source code of the changeset github action I realize that the action needs some changes as well. It needs to accept a custom glob as well. But it is totally doable.

[Andarist]

First I thought of making a PR against @manypkg/get-packages. But I realize that nx-repos function very differently than others which means it is not suitable for that project.

Could you expand on that? What makes it difficult? If we could support Nx out-of-the-box that would be preferable.

The packages we want to publish end up in a different directory than the source code; libs/package-a -> dist/libs/package-a

Is this standard for Nx or is this your custom setup? In the past, I've suggested that one can temporarily apply changes to, let's say, package.json#workspaces so Changesets could pick a different config and then just revert the changes from the file system. I understand that this might look a little bit cumbersome but if your setup is custom then I think it would be an acceptable solution.

After looking into the source code of the changeset github action I realize that the action needs some changes as well. It needs to accept a custom glob as well.

Could you expand on this one? I guess if we get to it I would see what do you mean in the PR but if you already know what would have to be changed I could take a look sooner at the plan you have for those changes.

[adambrgmn]

Thanks for the quick feedback @Andarist!

We do use a standard nx setup. Nx has a concept of generators to generate new "projects". And you can generate "publishable" projects. That means that when "building" that project nx will pack everything up in a tidy package that can be published to a npm registry. But it will put it in the dist folder, with package.json and everything, while keeping the source folder.

I've actually written a minimal wrapper script around changesets that sets workspaces before running, and then reverts afterwards. And that is actually a viable solution. But as you say it would be great to support nx out of the box if possible.

The reason I decided not to push this to @manypkg/get-packages is because of what I mentioned above – sometimes it needs to look in libs/*, sometimes dist/libs/*. But there's no easy way for getPackages to know which one to look for. The data is available in workspace.json, but you need to know which keys to look for. Then I thought it would be easier for changesets to know this, or at least let the user pass a custom glob pattern.

I tidied up my fork and pushed a proposal to https://github.com/adambrgmn/changesets/tree/adambrgmn/packages-glob. If you feel it's easier I would happily provide a PR instead.

Another approach I though about was making a specific solution for nx repos which included looking into the workspace.json file and extracting libraries from there. But it is also a bit gnarly, because there's a lot of config.

Then, a third option which I thought of was adding new configuration options to .changeset/config.json:

{
  // Either as an array which applies to all commands
  "packages": ["libs/*"]
  // or as an object which would make it possible to configure on a per command basis
  "packages": {
    "default": ["libs/*"],
    "publish": ["dist/libs/*"],
  }
}

The github action uses @manypkg/get-packages internally. But the solution in my fork was to create a thin wrapper around it that takes a custom glob pattern as a parameter. So the github action would need to accept that same glob and pass it to the same function used by the cli.

[Andarist]

We could extend somehow manypkg - obviously, it would have to become aware of this distinction and it would either expose a single object with 2 properties (like source and publish) where they both would be the same for other tools or it would expose an additional function that one could choose to select "publishable" packages. This would require the user to select the appropriate function - which is probably not that great.

If this is the standard Nx setup - having sources and packed packages in completely separate directories then I think it's worth adding support for smth like this into both manypkg and Changesets. cc @mitchellhamilton , wdyt?

[adambrgmn]

I'm trying to think of ways that manypkg could support this with an as simple api as possible. But it is hard since there are so many ways in which you can configure nx. You could set up some restrictions though. The project needs to be tagged with e.g. publishable in nx.json and have both root and a build target with outputPath option configured in workspace.json. Then getPackages would need to know if it should look for the output paths or the roots.

It is hard 😅 And the more I think about it maybe this edge case shouldn't be a concern for neither changesets or @manypkg/get-packages? And as mentioned I can work around it by tweaking my package.json before/after running changesets.

[adambrgmn]

Another idea that I just got would be if we could expose a new config option in .changeset/config.json:

{
  "getPacakges": "../path/to/custom/package-resolver.js"
}

That file could export a getPackages function which I guess should take some kind of context as a parameter. That context object could tell which command is calling it, a cwd path and so on.

We could then create a new package called something like @changeset/get-packages which is just a thin wrapper around @manypkg/get-packages and be used as the default for this configuration.

It's an idea but would require some changes to the architecture since getPackages is used in many different places. Even the github action would need to be updated since it also uses getPackages directly.

[Andarist]

I'm trying to think of ways that manypkg could support this with an as simple api as possible. But it is hard since there are so many ways in which you can configure nx.

Could you at least showcase 2 so I could assess how different they are etc and what is involved in each one of those?

[adambrgmn]

I'm trying to think of ways that manypkg could support this with an as simple api as possible. But it is hard since there are so many ways in which you can configure nx.

Could you at least showcase 2 so I could assess how different they are etc and what is involved in each one of those?

I might have been a bit misleading in my comment. There aren't that many ways to configure NX libraries that are relevant to this discussion.

You have the source code living in libs/{library_name}. In there you have package.json with version and dependencies. Then when you build NX packages it all with correct main|module|types fields in package.json and places the "ready-to-publish" package in dist/libs/{library_name}.

As of now we only have one package, which we can handle with manual releases. But I see value in being able to use changesets in the future. The more I think about it, a way to define your own getPackages logic is probably the the solution that I would vote for. Maybe with an "official" NX-adapter published.

I built a POC a few weeks back (https://github.com/adambrgmn/changesets/tree/adambrgmn/get-packages). It needs some tweaking and should probably use built-in NX logic instead of my naive approach.

The way you would use it is like this:

.changesets/config.json

{
  "getPackages": "@changesets/get-packages/nx"
}

[james-nash]

I'm encountering a similar issue, albeit with Angular rather than Nx, and something like the proposed getPackages config would help me out too.

In my case I have a monorepo containing various packages for a design system. Today I was adding an Angular library for the first time (all the other existing stuff is non-Angular, btw). I created it using their ng CLI and it turns out, when you build the package it generates a fresh package.json in the build output dir. So, I end up with something like this:

packages/
  my-angular-lib/
    package.json  <-- My "original" package.json file
    ...
    dist/
      package.json <-- A generated one produced by ng build

As per Angular's docs, to publish your package you're supposed to do something like:

cd dist
npm publish

All well and good when you're manually doing that. But I'm using changesets and want it to update versions in my package.json. However, just as with the OP about NX, when it comes to actually publishing, I need it to use the generated one instead.