Monorepo

[subzero10]

I would like to know your thoughts on moving to a monorepo solution.
We could have a monorepo per language, starting with the Javascript packages.

I think it will be an improvement for contributors to the packages, but it may be confusing for our users (for those unfamiliar on what is a monorepo). Another major side-effect is that all issues will be under one repository.

If you have no major reason against it, we could explore pros and cons.

[joshuap]

I'm not against having a monorepo for JS (I hadn't thought of doing it for other languages). I did consider using a monorepo when I did the last honeybadger.js rewrite and moved from the old honeybadger-js package to @honeybadger-io/js. I opted to stay with the single universal package until if/when a more complicated multi-package design made sense. Here are a few things I considered at the time:

• A monorepo would make it possible to release @honeybadger-io/core, @honeybadger-io/browser, and @honeybadger-io/server packages (in addition to the universal @honeybadger-io/js), which could be useful for some people.
• We could potentially share other utilities across packages.
• Like you said, having everything in one repo could simplify support and maintenance.

There are probably tradeoffs; I prefer to keep things simple unless there is an obvious benefit from the added complexity. Feel free to explore!

[subzero10]

No no, workflows would still be separate for each project. So, change in react package => only CI for React (and maybe CI for core package).
But, if the core js package is changed, we could run CI for all packages.

[shalvah]

So, change in react package => only CI for React

Hmmm, how? Does GitHub allow you to nest workflows in different folders? Or do you mean, like, using a Bash script to detect what files changed and decide what workflows to run (but that would be fairly stressful)?

[subzero10]

Github Workflows are pretty advanced these days (though they still have limitations!):

• You can run a workflow based on the branch, path and more (no script, just yml)
• You can reuse workflows.
• You can listen for events on other workflows and trigger workflows.

Currently, on a different project, I'm working on complex workflows and the monorepo helps a lot.

[shalvah]

Oh, nice. That's beautiful.

[subzero10]

Indeed it is!

[shalvah]

My thoughts:

• As @joshuap points out, a monorepo would make separate server/browser packages easily doable, but I think we've passed that point (we had separate packages in the past, yes?).
• If we're doing monorepos, it would probably be JS alone. I don't think the other language packages need such (yet).
• It would be easier to work on, say, React and Webpack packages if they were in the same repo as the core package, but I think it would also add a lot of complexity for those working on a single package. How big of a pain point has it been switching between the different projects @subzero10 ?
• I think the core JS repo is complex enough that putting it into another repo with other packages might get more confusing.

[subzero10]

• It doesn't have to be about server vs browser. We could have separate packages for different backend/frontend frameworks if it makes sense. With a monorepo, it will be easier to take that decision.
• I also think JS monorepo is OK at the moment (at least for the beginning). I generalized the question, because we could have monorepo for PHP + Laravel.
• Can you elaborate about the complexity concern? I don't see it.
• One of the reasons of going to a monorepo would be to break/extract the core js repo into smaller packages (think that the AWS Lambda module could be a package on its own).

[shalvah]

Hmm, the monorepo for PHP isn't a good idea because the work involved will make for a worse DX than separate folders. If we have several PHP projects in the future, maybe. But for two or three, separate repos is fine, IMO.

One of the reasons of going to a monorepo would be to break/extract the core js repo into smaller packages

I don't mind this, but I think this in particular (splitting up current packages) is a big product decision that will directly affect users. We'd first need to justify that and get approval from Josh and others, so I don't really see it as part of the case for a monorepo yet.

We could have separate packages for different backend/frontend frameworks if it makes sense.

This would be cool too, especially with the recent changes. I think this is a BIG point. Some of the framework packages would be a single file or two. With a monorepo, we could easily tuck them into the folder. But I also think this is a product decision, albeit a smaller one.

Re complexity: I feel that's pretty obvious when you think about it.

• To manage the monorepo, you'll need to add a new tool. npm workspaces are still young and I don't think they support a lot, so you might need to use Lerna. Every new tool adds complexity. The JS world is already suffering from too much tooling; we should be cautious about adding more.
• There's also complexity of configuration. Many tools expect one main folder. Do we install TypeScript separately for each subfolder? I think Lerna somehow shares dependencies but this causes issues sometimes because it isn't the normal way.
• Versioning: how are releases done? Do we release a new version for every package when one package is released (so they have the same version number), or do we let each package keep its own?

These are a few things to think about. We could probably give it a shot, and there might be real benefits, but there'd also definitely be added complexity.

[subzero10]

Hmm, the monorepo for PHP isn't a good idea because the work involved will make for a worse DX than separate folders. If we have several PHP projects in the future, maybe. But for two or three, separate repos is fine, IMO.

I agree. Let's focus this discussion on JS packages for now.

I don't mind this, but I think this in particular (splitting up current packages) is a big product decision that will directly affect users. We'd first need to justify that and get approval from Josh and others, so I don't really see it as part of the case for a monorepo yet.

Correct. We can't take this decision on our own; plus it's not a small change. But I feel it makes sense and it is the right path to go. That's why I initiated the discussion.

Re complexity: I feel that's pretty obvious when you think about it.

Your points above are valid. There are definitely pros and cons. New tools means further complexity, but it shouldn't be an issue if properly documented, as most of the times you wouldn't have to worry about them. For example, at the moment, I don't have to think about our release process or tools, all I need to know is that I can comment on a PR "at_shipjs prepare".

Personally, I think the pros outweigh the cons by far (or at least this is what my experience tells me).

Other notes:

• Managing the monorepo: Yes, npm workspaces is not enough, and I recommend Lerna as well.
• Configuration complexity: You concern is about dependencies and the main feature of these tools is to manage them. All we have to do is npm install at the root. Typescript is just a devDependency of each package.
• Versioning: This is all configurable. We could do either, obviously the simplest would be to release new versions for all packages every time, but that wouldn't be ideal. Why release new version of vue package without any changes? Though, we could release new version if a core package was updated, and lerna can help with this.

[joshuap]

See my comment here with some additional thoughts. One of the reasons I started with the universal package was because people wanted to use honeybadger.js w/ frameworks like Next.js, which does SSR and wants a universal package (at least, it makes it easier to set up; you can detect the environment and load two different packages, which is what we had to do before when we had separate packages for browser and node).

The way I see it, the current honeybadger-js repo on GitHub would become the new canonical monorepo, and we'd start by refactoring the current core, browser, and server modules into their own NPM packages w/ Lerna (or whatever we choose). Once that's done, we can begin moving other projects into the monorepo (honeybadger-vue, honeybadger-react, honeybadger-webpack, etc.)

I want our automation (testing, linting, releasing, etc.) to be standardized across all of our JS projects. @subzero10 has been working on that for the honeybadger-js repo lately. If we don't do the big monorepo migration, we'll have to eventually add the same automations and workflows to each individual GitHub repo, which is also work/overhead.

I agree w/ @subzero10 that there seem to be clear benefits to moving to a monorepo in JavaScript. I also agree with @shalvah regarding our other ecosystems (like PHP)—monorepos are less understood in PHP, Ruby, etc., and in most cases we only have one or two packages for each language vs. the many packages that we maintain for JavaScript.

[joshuap]

Btw, I get what you're saying @shalvah, about the drawbacks of the universal package. Once we have the other packages, we could definitely discuss which package(s) we should recommend as the happy path in our docs.

[joshuap]

Here's a Lerna alternative that I had recommend to me recently. I'm not sure if it's better for a project like ours, but some of the testimonials mention Lerna.

https://turborepo.org

cc @subzero10

[subzero10]

Hello!

Here's how I think we can move all js libraries into a monorepo. The plan I suggest below allows us to execute it incrementally while not having to spend all efforts on this project.

Milestone 1 - Setup the monorepo and do minimal changes necessary to have two packages

• Use honeybadger-js (this repo) as the monorepo.
• Create a PR that installs Lerna.
• Configure shipjs to use the monorepo. This means that we will have the same version across all packages. This is temporary and will be dealt with in a later milestone.
• Create two initial packages: core and js. Why js? Because the npm published name is @honeybadger-io/js. I'm not super excited, but I think it's OK, considering that this will be the legacy package (that will support both server and browser).
• Move test/browser, test/server, test/integration, src/browser, src/server and examples into the js package.
• Modify CI to work with new folder structure.
• Assert that shipjs works as expected.

Milestone 2 - server and browser packages

• Create server and browser from js. Keep js for legacy support, it should import server and browser as dependencies.
• Any change in any package will result in a new release for all packages (js, browser and server).

Milestone 3 - Use Lerna versioning

• Unfortunately, shipjs does not support independent versioning.
• On the other hand, the selling point of Lerna over Turborepo is its versioning capabilities. Lerna can identify changes in each package and suggest the next version, assuming we have conventional commits in place (which we do). This is actually the reason I choose Lerna.
• Modify CI to read issue comments in order to replicate shipjs features such as shipjs prepare.

Milestone 4 and forward - Migrate vue, react, webpack, etc. into the monorepo

• I haven't planned for this yet. Though by this time, adding a new package to the monorepo should be straightforward, considering there will already be a few.

Note: We can also integrate both turborepo and lerna. The case for this is to use the benefits of both of these tools: speed of turborepo & versioning of lerna. However, our codebase is not large enough to be slow with Lerna. In any case, it's good to know that we have the option.

What do you think @shalvah @joshuap ?

[joshuap]

Hmmm. Do you have any idea where the larger JavaScript community is headed? I know a lot of projects still use Lerna. Are they switching to something else?

[subzero10]

Thanks @shalvah. Same as @joshuap, many of the monorepos I see are using Lerna. I may try nx + semver.

[subzero10]

https://monorepo.tools/

[subzero10]

Hello! I have some updates on this. For the last few days I have been trying to setup a monorepo with Nx. Here's what I can share:

• Disclaimer - First time trying nx.
• Nx is a far more feature-rich than Lerna. It has a lot of features and it is extensible. As a result, it has a bigger learning curve.
• The way the monorepo is setup greatly depends on how you want to use Nx and how many of its features you want to use:
  • If used with its plugins, there is a single root package.json that manages dependencies. There are pros and cons to this.
  • If used without its plugins, it becomes similar to maintaining a Lerna repository but it's easy to get confused between the two in the documentation pages.

My conclusions (please feel free to share your thoughts as well):

• Nx has a lot of features that we don't really need. I don't anticipate us needing them any time soon either (cloud, caching, distributed tasks, parallel tasks, complex graph analysis, plugins, etc.).
• Nx is built for large monorepos with hundreds/thousands of packages. Here is an answer from SO that is similar to my thoughts.
• A Lerna repository is not an irreversible route, since there are guides to migrate to Nx from Lerna (both are maintained by the same people).
• I will try out Lerna in the coming days.

[joshuap]

Thanks for the breakdown, @subzero10! Looking forward to hearing your thoughts on Lerna.

[shalvah]

Another thing I think I'd like to see @subzero10 if we go the Monorepo way is less transpiled code, especially for the server build. We're transpiling all the way down to ES2015, which makes the code more different from what we've written, and harder to follow and debug. On the server side, that level of transpiling isn't necessary.

[shalvah]

My bad; we're actually transpiling down to ES5 (aka ES2009). 😱 but I see you've raised the level to ES6 in #828. Great.

[subzero10]

True, but integration tests are failing 😅 . I haven't looked into it yet, but this may be related.