To be or not to be in Core

[danielleadams]

As a continuation of nodejs/TSC#1041, there was a meeting to discuss the following questions:

• Why should we include x in core?
• How should we include module x in core?

We are hoping to discuss these 2 questions here and land on ideas for the future of why and how libraries and dependencies should go into core. The goal of this discussion is to be able to write guidelines for currently discussed dependencies and future dependencies of interest.

[jasnell]

For the Web Platform related stuff, I've generally been following a simple rule: If the Web Platform API overlaps with existing functionality that is already in core, then it's a candidate for inclusion. If there's no overlap with existing functionality, then it is not an immediate candidate but could be if a strong enough use case is presented.

[mhdawson]

For me this would be a starting list of what motivates things to be in core (not necessarily in any order)

• They can only be implemented in core
• They can only be implemented in a "performant" way in core
• Developer experience is significantly improved if they are in core
• They will be widely used, have cross platform native code, and being in core will significantly improve consumption due to avoiding the need for the compilation chain and issues related to building native addons.
• Part or all of the feature will also be re-used or duplicated in core
• They are an expected part of a runtime (basic diagnostics, debug support etc.)
• They are a core part of the "value" that Node.js delivers (I agree in advance that this is pretty fuzzy).

[ronag]

I try not to repeat things already mentioned.

Pro Core:

• CITGM & CI workflow
• LTS workflow
• Unified documentation
• No dependency on npm

Con Core:

• Reduced development velocity
• Harder and slower to publish patches
• More difficult upgrade path for users (i.e. can't choose to just upgrade specific module).
• Harder for maintainers to get sponsors to fund development of specific modules.

[ronag]

I think the vendoring in packages that also live on npm (but are part of github nodejs org) is a good approach and would like to know what drawbacks people see with that?

[Trott]

I think the vendoring in packages that also live on npm (but are part of github nodejs org) is a good approach and would like to know what drawbacks people see with that?

That's what was done with node-inspect until relatively recently. There may have been other downsides so others might chime in, but the ones that stick out for me are:

• Vendored module's tests weren't run when a new version of Node.js was released. So, IIRC, significant parts of CLI debugging didn't work with Node.js 15 on Windows for the entire lifetime of that release.
• Confusion on the part of users about where to report bugs.
• When maintenance of node-inspect slowed and effectively stopped, people involved in core didn't know this was happening. That can happen with things in core too, but I think it's more likely to happen with vendored stuff.

[ronag]

Vendored module's tests weren't run when a new version of Node.js was released

But I guess that's just how it's vendored in. I don't see any technical reason why we couldn't vendor in the tests as well?

Confusion on the part of users about where to report bugs.

Was this a big problem? I would personally not have an issue with having issues in two places. We already partly have this situation e.g. with readable-streams.

When maintenance of node-inspect slowed and effectively stopped, people involved in core didn't know this was happening. That can happen with things in core too, but I think it's more likely to happen with vendored stuff.

Yea, I think a part of our discussion was that when it slows down (e.g. due to maturity or abandonment) then would be a good time to include it in core. Since the benefits of having it outside no longer remain.

[Trott]

Vendored module's tests weren't run when a new version of Node.js was released

But I guess that's just how it's vendored in. I don't see any technical reason why we couldn't vendor in the tests as well?

If the tests use a test framework, then we'd have to vendor that in or else skip the tests in certain situations. There are probably other complications, but probably nothing that can't be solved.

Confusion on the part of users about where to report bugs.

Was this a big problem? I would personally not have an issue with having issues in two places. We already partly have this situation e.g. with readable-streams.

Generally, no, it wasn't a big deal but it did mean that issues in the node-inspect repo, which were effectively issues in core, weren't seen by most people watching/maintaining/searching in the core repo. So there was a certain amount of bugs and technical debt that were kind of hidden away. I don't consider this a dealbreaker or anything, but it is a (probably small and manageable) drawback.

When maintenance of node-inspect slowed and effectively stopped, people involved in core didn't know this was happening. That can happen with things in core too, but I think it's more likely to happen with vendored stuff.

Yea, I think a part of our discussion was that when it slows down (e.g. due to maturity or abandonment) then would be a good time to include it in core. Since the benefits of having it outside no longer remain.

It was a not-small effort to get it into core. All the tests and code had to be re-worked to meet our linting and testing requirements and then all the tests that were failing that we didn't know about had to be fixed, and it often wasn't immediately clear (to me, at least) if the problem was the test or an actual bug. Primordials had to be used everywhere (which I know is a related discussion around to what extent will we use primordials going forward) and probably other things I'm forgetting. In fact, the node-inspect repository still exists and I have it on my to-do list to go through the pull requests and issues there before archiving it. So this process will probably take a year at least end-to-end, mostly because there's no one working on it much, but also because there are a lot of little pieces to deal with. Again, I'm not saying this means we shouldn't generally go this way. But a full list of drawbacks of the approach would include this. I think moving something into core right away might be easier in many ways than moving it in after it's been vendored for a while but has lived in a separate repository, and I don't think that's intuitively obvious to people.

[ronag]

I think moving something into core right away is easier than moving it in after it's been vendored for a while but has lived in a separate repository, and I don't think that's intuitively obvious to people.

I don't think "moving something into core right away" is realistic though. At least not in most cases. Then it should have been developed in core to begin with in which case there is no "to be or not to be in core" problem.

[Trott]

I don't think "moving something into core right away" is realistic though. At least not in most cases. Then it should have been developed in core to begin with in which case there is no "to be or not to be in core" problem.

Fair point. Guess I should keep the conversational context in mind. 😀