-
Notifications
You must be signed in to change notification settings - Fork 29.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Replace gyp, node-gyp for addons by adding compiler abstraction #7440
Replace gyp, node-gyp for addons by adding compiler abstraction #7440
Conversation
This flag is useful, to make gyp not build and link node cores /src/*.cc files. However all dependecnies will be built. This can be used to build node proper with an arbitrary build runner and/ or with itself.
WIP, requiring from dir is not possible with NativeModule.
/cc @indutny |
I really like the idea, but it is hard for me to tell how useful it will be for the current users. We need to gather more information about current The problem with simplified system is that it is pretty hard to build things that needs lots of features, and considering that this eventually may become the only build system - this may limit addons. |
/cc @nodejs/addon-api I've been through a lot of ecosystem binding.gyp, and I think I can safely assert that they are almost all using only the basic declarative logic in there - if linux do this, if windows do this, if osx do this - they may as well just have a makefile and a .bat and most could probably be easily replaced with this kind of tooling. Being able to export some defaults from core like we do with common.gypi would be nice, although that's a bit of a dogs breakfast and it'd be nice to do something much cleaner than that since it's the cause of a lot of binding.gyp pain that exists out there. I'd like to see how much effort this takes to add v8 support for this, that seems like a pretty large job. Any idea on what amount of work that would be @eljefedelrodeodeljefe? |
c133999
to
83c7a88
Compare
@eljefedelrodeodeljefe ... still want to pursue this? The likely better course would be to engage with the new Node.js API (abstraction API) work |
@jasnell not sure what you mean. I would want to pursue this, but it's likely to be crunched by the related discussions about the future of gyp. I am a little bit wary arguing for a slim JS thingy. |
I think that this is worth reopening, here or elsewhere. |
@stevevachon why do you think so? This will very likely never go forward. |
If it were backwards compatible with gyp, it would provide time to deprecate. |
There is hope, |
Yep, and while that's great, I don't think that gyp should be used for anything more than backwards-compatibility moving forward. |
@stevenvachon need a viable working alternative. Are you willing to commit to |
I don't have any prior gyp experience, but it might not be very difficult if another library can handle the heavy gyp parsing. platform-tools#14 |
GYP does quite a lot though. "Another library" would basically be a reimplementation of GYP. You may be able to leave out some things but I think you'll end up being dismayed at the amount of functionality you will have to replicate. |
we do have |
@bnoordhuis I am not really convinced that gyp does a lot. Also I think this disregards a lot of things that are more crucial, like usability, which I hope we agree, is insufficient. |
Ohh yea, 👎 on usability. But if you have a way to eat |
Famous last words! All I can say is try it and see how far you get before you run out of steam. You could aim for a 80/20 solution but popular add-ons like node-sass and node-iconv are in that 20% category. |
See, I already did one year ago https://github.com/eljefedelrodeodeljefe/platform-tools. I mean, you know that I like you, but you are quite conservative about initiatives like that :) While that might good for personal mental hygiene, you could at least be supportive stating that you like the general idea or not, apart from the efforts (you don't have to take). I don't care too much about my hygiene for example :) |
The 80/20 things is of course correct. I would aim for Node addons and node proper first. That leaves complicated native addons and v8 / libuv. I think that already is great help for the ecosystem and node development (except those who integrate v8) |
Challenge accepted ⚔ eljefedelrodeodeljefe/platform-tools#15 |
@eljefedelrodeodeljefe I'm supportive, I'd like to get rid of GYP and node-gyp, but I'm also a realist: a great heard of yaks are going to get a new haircut before any successor is a drop-in replacement. There is a lot of institutional knowledge encoded in node-gyp and (particularly) GYP that you will have to figure out for yourself and replicate. I'll be happy to assist with moral support, though. :-) (In all seriousness, I know a great deal about both projects. If you have questions about why something works the way it does, don't hesitate to ask.) |
I am gonna take a shot at an implementation in the second half of September. Until then I will review related discussions. I am happy that @bnoordhuis is kinda in support. Sorry for being abscent. |
Hey! Welcome back! No apology necessary! |
Ping @eljefedelrodeodeljefe again |
Closing due to long inactivity. @eljefedelrodeodeljefe please feel free to reopen if you would like to pursue this further. |
EXPERIMENTAL, WIP
Checklist
make -j4 test
(UNIX), orvcbuild test nosign
(Windows) passesAdding
platform_tools
APIReducing Build Dependencies
In order to achieve nodejs/CTC#2 I am proposing an API for building node addons, C/C++ sources and Node itself.
This would not be a drop in replacement fro GYP, but rather follow the philosophy that a build system is not necessary for tasks such as ours. The only heavy lifting it does is escaping and putting flags in the right order so calling the respective platforms compilers has a normalized API.
At it's core it does exactly what build systems do, namely calling on a list
gcc [src] -c o [target].o
and link all of those togethergcc [1.o, 2.o, ... , n.o] [executable].o
. In GYP andmake
you'd do that by declaring file locations in a configuration file and then have control flow elements like target and platform conditions. When libraries grow those files pivot to being written with declarative programming.The dynamic nature of JS is seems to be more suited for that purpose and also does not require users to learn other (and declarative) languages, nor gluing this together with pyhton. Also it requires no dependency, except a node binary of the v4 lts line.
This is based on some experiments I conduct in a library here:
https://github.com/eljefedelrodeodeljefe/platform-tools
Discussion
Why this as API?
Does need to be one. But the intention behind is to curate building issues we have for a long time now and will, imo, not be able to fix them externally. Take
gccgo
/go build
as example.Native addons as feature should be promoted cross platform. Authoring native addons is really unpleasant.
Why not improve on Gyp?
gyp is a language, which is bad enough. Also its's really chromium specific and a more or less private Google repo. There won't be much improvements there. Also the python dependency is a big hurdle for any platform in order to allow for a nice build experience
Why no
make
?Make is also a declarative language one needs to learn and also it is very implicit about parameter passing. The biggest issue with it is that is not cross platform, which makes
vcbuild.bat
necessary.Why no
vcbuild.bat
?Same as
make
Why no IDE integration (for now and / or built-in)
Both Visual Studio and Xcode abstract away the tiniest dependencies, such as (VS) basic C headers like
#include <stdio.h>
, which requires a properly set up build environment. On windows that would mean callingvcvarsall.bat
, but all it does adding those include dir to your path. This lib has the default paths built-in, assuming users have the compilers in default locationsWhy no
ninja
Ninja is rather a meta build system, which is nice and fast, but won't do much alone. It then requires a set-up build environment. Also it's a binary dependency, users need to install.
Why no build system at all?
Build systems abstract the compiler, linker duality quite a lot and offer some additional features. Features such as timestamp comparison and dependency management are nice on paper, but probably can be done more easily in JS by native JS programmers. Dependency management is probably the prime domain of JS, where every build dependency is a
npm install
away. Take as opposing example the installation of gyp or runninggclient sync
. Linker and compiler abstraction might be nice but imo is also dangerous, since you can be easily trapped in a black bock likeVisual Studio
which suggests, that is doing the heavy lifting, but actually is just running the 500 loc logic presented here. Also runningCL.exe
directly is more portable approach, since you pretty much run the same logic withgcc
.Can this be used now?
Yes, by omitting the node core compilation step and then run this API. The makes it integratable now and can be improved to build all others C/C++ deps with the goal of getting rid of python and gyp in the long run. Those can be done one by one, with V8 being the biggest bite.
What about sugar features of the other build systems
Change detection and a dynamic dependency management could be implemented by userland. This should just serve as very thin layer for normalizing compiler calls and build stuff barebones.
Modular Core
This would be a first dependency that could be developed outside of core, but be an essential part to it, as discussed in #7098
Known Issues
NativeModule.require
, since I think it's not possible to require from deps dynamically. Then the consequence is probably compiling all sourcestar-stream
andmkdirp
. Those are necessary to build addons, since the node-gyp-like workflow needs headers that will be downloaded.API
Compiling a C Program
The below would be an example of emulating with Node.js
Compiling Node Addon
Compile Node
Obviously this is rather verbose, but a huge reduction to what we have in the repo and as dependencies.
Then run the custom node build logic.
Building V8
Building libuv
Building other C deps and js2c run