Organize monorepo better

[koddsson]

In an effort to lower the maintenance burden of the Modern Web projects we've decided to move certain packages out of the monorepo into their own repositories. This issue serves as a work-in-progress list of packages that are on the list to be migrated.

(Please edit this issue to add your name to a package if you are planning to create a repo for it to avoid double work)

• @web/dev-server-esbuild @koddsson
• @web/dev-server-hmr @koddsson
• @web/dev-server-polyfill @koddsson
• @web/dev-server-storybook @koddsson
• @web/mocks
• @web/rollup-plugin-copy @koddsson
• @web/rollup-plugin-html @koddsson
• @web/rollup-plugin-import-meta-assets @koddsson
• @web/rollup-plugin-polyfills-loader @koddsson
• @web/storybook-builder
• @web/storybook-framework-web-components @koddsson
• @web/storybook-utils @koddsson
• @web/test-runner-visual-regression @koddsson
• rollup-plugin-workbox @koddsson

Once we've created all the repositories we'll remove the packages from this repository and close this issue.

[koddsson]

I just realized that this nukes all of the git history which I hate. I'll have to look into fixing that.

[Westbrook]

Do we not want some of these to be together in some way?

Like doesn't putting all the Rollup plugins in one place make them easier to keep up to date?

@web/storybook-builder and @web/storybook-framework-web-components are partners anyways?

I can't speak to the Dev Server tools you're moving, but @web/test-runner-visual-regression is a pretty important piece of the testing ecosystem, doesn't it make better sense for it to continue living with the rest of the Test Runner tools?

[koddsson]

Like doesn't putting all the Rollup plugins in one place make them easier to keep up to date?

I've personally found it harder to maintain packages as they are colocated in a monorepo so this was supposed to help with that. The monorepo kind of mandates that there are specific TypeScript settings and we've got some custom scripts to keep that all in check. Having these just all in their own repos allows us to use off the shelve tooling wihtout all the script glue.

@web/storybook-builder and @web/storybook-framework-web-components are partners anyways?

Yeah I feel like these could then be merged into a single repo. I'm not too familiar so I'm not sure.

I can't speak to the Dev Server tools you're moving, but @web/test-runner-visual-regression is a pretty important piece of the testing ecosystem, doesn't it make better sense for it to continue living with the rest of the Test Runner tools?

My thinking is that modernweb-dev/web should pretty much just be test-runner and dev-server and everything else should live outside that repo. Not all projects need visual regression testing and so I think of it as a plugin rather than a core part of the test runner.

---

Other alternatives to doing all this would be to stop using npm workspaces and the custom scripts to also simplify the monorepo. Treating the monorepo as just a collection of independent packages but I worry that the API layer will be broken quite often as it's happened in the monorepo as it stands today.

[bashmish]

In an effort to lower the maintenance burden of the Modern Web projects we've decided to move certain packages out of the monorepo into their own repositories.

Can you please elaborate how splitting the monorepo would lower the maintenance?
Sure there are pros and cons of monorepos vs multiple repose, but curious about your personal experience here and what brought you to this decision?

[koddsson]

In an effort to lower the maintenance burden of the Modern Web projects we've decided to move certain packages out of the monorepo into their own repositories.

Can you please elaborate how splitting the monorepo would lower the maintenance? Sure there are pros and cons of monorepos vs multiple repose, but curious about your personal experience here and what brought you to this decision?

It's a lot easier to make changes in isolation. With the current setup you have to make huge sweeping changes like this: #2600.

The current setup also kind of assumes that every package is using the same technology, as different packages have grown at different paces we've had to add more and more custom scripts to try to mange these. TypeScript with it's different tsconfigs, some projects in ESM and some in CommonJS, etc..

[koddsson]

If we end up reverting this decision I'd like someone to review and approve #2600 in its place.

[bashmish]

Generally speaking, from my personal experience, having a monorepo makes sense if it's the same team maintaining it and everyone has knowledge about most of the packages within it. On the other hand, whether the packages are related to each other logically is not that important.
Also it should be limited in size, basically you can't just add new packages to it forever, so you need to draw a line. Maybe we reached that point.

In this specific case it might be hard to make changes like #2600, because it requires deep knowledge of some packages (e.g. storybook ones) where the logic and requirements for ESM might be different. On top of that the lack of tests makes it hard to verify if such changes are fine, and more packages without automated tests means more burden on the author of such PRs in case things go wrong. I guess this is what cause you a lot of challenges. Same for me as a reviewer: I honestly fear reviewing it as I don't have knoweldge about many of the packages and not sure how to reliably test such changes and sigh them off. Maybe in we can just release it as alpha/beta and ask the community to help with testing?

Having that said, I think it will be hard to split the repository. Because it's not just code, it's also the website with docs behind it, so you'll need to extract it into another repo too, but still central with the entire site content, right? Which will add more maintenance costs for it. It's not clear to me if such trade-off is worth it in the long run. We don't have changes like #2600 often, it's a bit of a special case.

[koddsson]

Because it's not just code, it's also the website with docs behind it, so you'll need to extract it into another repo too, but still central with the entire site content, right?

I don't know about this. As far as I can tell there's nothing that ties packages to the documentation so it shouldn't matter if the actual code is in the repo or not.