Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Strategic initiative to move to cmake for building Node.js? #648

Open
mhdawson opened this issue Jan 2, 2019 · 21 comments

Comments

Projects
None yet
@mhdawson
Copy link
Member

commented Jan 2, 2019

@bnoordhuis had done some experimentation to move Node.js to use cmake but I think we need more consensus/momentum to be able to make a move like that.

I'd like to restart discussion in the context of: nodejs/node-addon-api#422. Before we start to experiment with building addons with a different tool like cmake, it would be good to better understand if it might align with what we using in Node.js in the future.

@ofrobots

This comment has been minimized.

Copy link
Contributor

commented Jan 2, 2019

This seems related to #642.

For some background there is some related discussion in #464.

To keep the discussion clear and focussed, I think this is about how Node.js-core should be built rather than ecosystem native addons, right? The constraints are quite different between the two problem spaces, so let clarify that this is not about native addons.

Before we can have a strategic initiative for building with cmake, I think we will need to explicitly agree that cmake is the build system we are going to use.

At the core, the choice of build tool is about modularity. C++, unlike Go, rust, and JavaScript, does not have a standard "assemble-dependencies-and-build" system. Whatever choice of build system we have will have to deal with the fact that we depend on some fairly large projects (e.g. V8) and that we have to make those projects buildable with whatever build tool we chose for building Node.js itself. Most of our other C/C++ dependencies are simple enough that the main challenge we need to solve is how our build tool choice will affect how we incorporate V8. It is not a matter of migrating V8 to cmake once. We have to maintain this build system, and V8 notoriously moves very fast.

For diagnostics we (I?) want to pick up the Perfetto project as a dependency to replace the Trace Events infrastructure that we are maintaining in V8 and Node. This and other future dependencies, and how we consume them, will be affected by our choice of build tool.

I am not necessary against cmake, I just haven't seen consensus around it yet.

@gabrielschulhof

This comment has been minimized.

Copy link

commented Jan 2, 2019

FWIW there's some momentum in nodejs/node-addon-api#422 to move to cmake. Do we need to agree to move the whole project to CMake before we deploy cmake on our CI? I'd like to ask in @nodejs/build that CMake be added to the tools available on the CI (if not already there).

@mhdawson

This comment has been minimized.

Copy link
Member Author

commented Jan 2, 2019

@ofrobots that is exactly the discussion I wanted to get re-started "how Node.js-core should be built. @bnoordhuis for some reason I was thinking that you might have been building everything except v8 with cmake in your work, but I may not be remembering correctly.

@mhdawson

This comment has been minimized.

Copy link
Member Author

commented Jan 2, 2019

@gabrielschulhof I mentioned nodejs/node-addon-api#422 above as I wanted to see if there was any consensus on what we should do for Node.js core as that would make the choice for add-ons easier. I don't think it's a prereq but I think its worth exploring as it would be good to avoid having too many different build systems required.

@ofrobots

This comment has been minimized.

Copy link
Contributor

commented Jan 2, 2019

As a point of comparison, from another notable V8 embedder, deno seems to be using GN to build.

@mhdawson

This comment has been minimized.

Copy link
Member Author

commented Jan 2, 2019

I think when we asked the Google team last time they specifically said GN was not "supported" for use in a case like ours and recommended other options.

@ofrobots

This comment has been minimized.

Copy link
Contributor

commented Jan 2, 2019

Adding @nodejs/v8 for clarity about GN and level of support. It is ultimately the V8 team's choice to use GN that puts this cost on their embedders like us.

@mhdawson

This comment has been minimized.

Copy link
Member Author

commented Jan 2, 2019

@ofrobots thanks, its good to re-confirm on that front. Things may even have changed over time.

@mhdawson

This comment has been minimized.

Copy link
Member Author

commented Jan 2, 2019

While not the same topic I think it would be good to add to the discussion whether the build system for Node.js core and add-ons should be the same? I wanted to restart the discussion in core to have as good as data as possible for nodejs/node-addon-api#422 but if the concerns can be/should be separated it's good to figure that out as well.

@joyeecheung

This comment has been minimized.

Copy link
Member

commented Jan 3, 2019

It is possible to build v8 with gn but use cmake for the rest of the project? V8’s fork has been using the configure option —build-v8-with-gn to trigger an action and build v8 directly with gn, if we can somehow host gn in-tree and bootstrap by compiling it with cmake (if gn is not available in PATH) then we won’t take on the burden of porting v8 build files to cmake?

@Trott

This comment has been minimized.

Copy link
Member

commented Jan 3, 2019

As a point of comparison, from another notable V8 embedder, deno seems to be using GN to build.

Same for electron. (Not saying we should use GN. Just another data point.)

@ryzokuken

This comment has been minimized.

Copy link
Member

commented Jan 3, 2019

I had been compiling a list of pros/cons regarding build systems in our case, but had been reluctant to spark off this discussion myself since there's a lot of conflicting opinions.

Some objective truths that are important to consider IMO:

  1. If we want to build using GN, we will not only have make all contributors install depot_tools, we'd also have to use them for various other purposes including fetching dependencies (which is something electron does as well). Important things to consider: another step for new and returning contributors as well as atleast a minor loss in productivity while everyone gets used to the new workflow.

  2. Right now, @targos and I take turns at updating our gypfiles for building V8 whenever canary-base is broken, which on bad days is nearly daily. While CMake is great in the way that we do not have to maintain it by ourselves (Refael has taken up the burden to maintain gyp pretty much all by himself recently, but I wouldn't mention him since he's staying away from the project right now) we will have to do the same for CMake, porting the changes in the latest GN configs to the CMake format.

Pros of CMake/Cons of gyp:

  1. It's easily one of the most widely used build systems out there. Since there's a ton of resources, documentation and knowledge regarding CMake out there in the open, we'd get rid of the current situation where apart from a small group of collaborators, nobody really understands how gyp really works etc.

  2. It's well maintained, and not going anywhere anytime soon.

Pros of gyp/Cons of CMake:

  1. There's a small upfront cost in porting all our gyp configs to CMake, but that should probably be the least of our worries.

  2. CMake is completely unsupported by the V8 team, so it would be futile to expect for a lot.

  3. Nobody in the ecosystem (either upstream at V8, nor downstream at Electron, for example) really uses CMake. We make everyone else's lives hard (including people who help embed Node.js into Electron, /cc @codebytere).

Game changer: Over at Electron, the exact opposite is required to be done (porting our changes in gyp to GN), and they've hacked up a clever script that automatically does most of the work, if not all for them. I've wanted to write a similar script for us vs V8 for quite some time, which should take care of ~70-80% of minor changes to V8's build. If we do manage to cover that many changes, it would be a serious pro in favor of gyp.

(Note: A similar script could also be written down for CMake, tbh. The only difference is that we would not have much to refer to.)

If the official support (neither of the above two are, really) seems to be a major issue, we have two officially supported alternatives (these would cut down the amount of work we need to put in on a near daily basis): GN and Bazel.

Last I discussed this with @hashseed, the Chromium team was not very happy with any outside project using GN, but that might have changed: https://gn.googlesource.com/gn/.

They seemed to be (atleast at that time) more interested in the idea of us using Bazel since Google uses Blaze internally anyway and it was likely that they could share the currently internal build files with us, making our lives easier.

Ofcourse, the GN build files are already public so the sky's the limit for that one.

EDIT: Removed way too many occurrences of the phrase "unless I'm wrong". Please note that I'm by far not an authority in any of these matter and may be colossally wrong in multiple points.

@gengjiawen

This comment has been minimized.

Copy link
Member

commented Feb 3, 2019

I want list some pros of CMake:

CMake has better IDE support

I think CMake is very friendly to new cpp users. Both clion and visual studio support cmake.
With great IDE support we can attract more developers contribute to nodejs.

I have been use https://github.com/lev-kazakov/node-cmake-generator to generate CMake files for me.
Pretty pleasant developer exerience with clion.
If we can go to other build system in the end, maybe still provide an option to generate cmake.

CMake has better cpp tools integration

With CMake, it will be easier for us to integrate tools like clang-format, clang-tidy.
I recently see https://github.com/bugwelle/cpp-starter-project.

@refack

This comment has been minimized.

Copy link
Member

commented Feb 3, 2019

@gengjiawen in that respect GYP has two redeeming qualities. It can generate CMake files (specifically CMakeLists.txt), and it can generate compile_commands.json.

@gengjiawen

This comment has been minimized.

Copy link
Member

commented Feb 4, 2019

@refack Yeap, that's node-cmake-generator internally using. But it doesn't support the latest cmake.

@bnoordhuis

This comment has been minimized.

Copy link
Member

commented Jun 24, 2019

@bnoordhuis for some reason I was thinking that you might have been building everything except v8 with cmake in your work, but I may not be remembering correctly.

@mhdawson Belated reply but my cmake fork builds (well, built...) everything with cmake, including v8: https://github.com/bnoordhuis/io.js/tree/cmake/cmake/v8

@391861737

This comment has been minimized.

Copy link

commented Jun 27, 2019

This seems related to #642.

For some background there is some related discussion in #464.

To keep the discussion clear and focussed, I think this is about how Node.js-core should be built rather than ecosystem native addons, right? The constraints are quite different between the two problem spaces, so let clarify that this is not about native addons.

Before we can have a strategic initiative for building with cmake, I think we will need to explicitly agree that cmake is the build system we are going to use.

At the core, the choice of build tool is about modularity. C++, unlike Go, rust, and JavaScript, does not have a standard "assemble-dependencies-and-build" system. Whatever choice of build system we have will have to deal with the fact that we depend on some fairly large projects (e.g. V8) and that we have to make those projects buildable with whatever build tool we chose for building Node.js itself. Most of our other C/C++ dependencies are simple enough that the main challenge we need to solve is how our build tool choice will affect how we incorporate V8. It is not a matter of migrating V8 to cmake once. We have to maintain this build system, and V8 notoriously moves very fast.

For diagnostics we (I?) want to pick up the Perfetto project as a dependency to replace the Trace Events infrastructure that we are maintaining in V8 and Node. This and other future dependencies, and how we consume them, will be affected by our choice of build tool.

I am not necessary against cmake, I just haven't seen consensus around it yet.

What about CPPAN? It is really good at dealing with dependencies... and works great with CMake

@hashseed

This comment has been minimized.

Copy link
Member

commented Jun 27, 2019

@apaprocki mentioned that he has a cmake port that he could imagine upstreaming as well.

@apaprocki

This comment has been minimized.

Copy link

commented Jun 30, 2019

@hashseed We just now open-sourced our solution here https://github.com/frutiger/ncg#nodejs-cmake-generator. The team that uses it internally is more than happy to collaborate.

Only other comment re: Bazel & GN -- I don't believe either support AIX or Solaris-derivatives and Node builds on both. CMake does work on all supported platforms.

@mhdawson

This comment has been minimized.

Copy link
Member Author

commented Jul 10, 2019

@bnoordhuis could you look at the solution open sourced b y @apapirovski and comment as to whether you think this would be a good direction for node core?

@ryzokuken

This comment has been minimized.

Copy link
Member

commented Jul 11, 2019

@mhdawson I think you meant @apaprocki and not @apapirovski 😄

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.