RFC: A transition to cmake as a build / configure system

[13steinj]

I know this is going to be a contentious issue, but after spending a decent amount of time digging into the weeds here I felt needed to brought up. I like make as much as the next guy, but with the number of dependencies that the Julia project has I feel as if it becomes difficult to integrate into larger systems. Especially in that julia is supposed to be cross platform, it was surprising to find that it builds with GNU make.

cmake, while I think everyone can agree is not ideal, fulfills the job of a cross-platform build system and has larger community buy-in at this point. It also provides simpler methodology for integration with either conan or vcpkg, or general patching by the end-user, who needs to say, not use a system lib but instead use zlib-ng rather than standard zlib where that's meant to be a drop-in replacement.

Added benefits also include tighter integration and parity with llvm's build system (as well as that of other dependencies), where make doesn't really have form of inspection / communication between projects, even on itself. Also of course, ability to use ninja generators, which is purported to be faster than make (of course, seeing an effect here depends on the project, but on larger ones I've used in the past I've legitimately seen improvements on the scale of 2-5 minutes (though, with Julia, I doubt improvements will be that significant).

[vchuravy]

While I sympathize with this, we tried going down the CMake route before and it didn't really work out.

Added benefits also include tighter integration and parity with llvm's build system (as well as that of other dependencies), where make doesn't really have form of inspection / communication between projects, even on itself.

I don't really see a benefit here. LLVM cmake can be really complicated and requires significant expertise to work with (at least in my experience...)

Now if you had said meson, I at least would be a bit more exited.

[13steinj]

I don't really know what you mean by this (but I also was unable to find this attempt).

I can understand LLVM cmake being very complicated; but trying to interact with it from make not only feels like a step backwards but just ends up being bizarre. There's definitely benefit to be had to being able to (within the same system) have knowledge about build artifacts / exported artifact names / groups.

I'm optimistic about meson, but I don't think it's the clear winner here, especially what with other projects are using. I understand that it has some level of introspection across build systems, so that's also definitely a valid choice. If it was my rodeo, I'd still pick cmake over meson (at least at this time), if only because of larger community buy-in and support (and yes, you could argue that make has even more...but it's not like you can easily find and dig through it ;)).

[phma]

What went wrong with CMake? And what is "LLVM cmake"? I've used CMake with gcc as well as with clang.

Are you trying to run CMake from the Makefile? That's not how it works. You run cmake, and it produces a Makefile (or a build.ninja file or other things), then you run make (or ninja), which usually just builds things, but could rerun cmake if something's changed.

[vchuravy]

"LLVM cmake" is the dialect of CMake that is being used in the LLVM project. CMake is lovely for small projects, but LLVM (a large/very-large project) needed to add a non-trivial amount of customization on top of that.

I find CMake harder to debug then raw Make...

I responded this morning here as well to the question "Why not Cmake" https://discourse.julialang.org/t/porting-julia-to-new-platforms/122698/6

I would ask instead the question "Why CMake", what does CMake get us the Julia developers that will make our lifes easier, our development workflows faster, that it will be worth the pain and suffering of porting over the build-system.

My understanding is that we would need to re-implement a significant set of capabilities that the current Make system has in CMake,
but what would the upside be?

[13steinj]

I would ask instead the question "Why CMake", what does CMake get us the Julia developers that will make our lifes easier, our development workflows faster, that it will be worth the pain and suffering of porting over the build-system.

The primary thing isn't how much it helps your lives, IMO-- it's incredibly debatable and can go ad infinitum. I know someone that hates modern build systems and would write manual recursive "meta-programmatic make" and everyone would think he's insane, but he'd argue it's easier than cmake.

To repeat from my original request: the point is moreso how much easier it becomes to integrate the julia build into a larger project, that is either using Julia in an embedded context, or in an extension-based context. A previous org of mine had various compiler flags / stdlib requirements that when not applying those to the Julia build, would cause a slew of ABI issues (might still be occurring, no guarantee that those were truly fixed).

A cmake-based build-system makes it easier to resolve these problems, because cmake is far more flexible in it's operation. People can supply a custom cache/option pre-population script docs, call it from any tool, or from within cmake have direct support and use something along the lines of add_subdirectory or FetchContent.... Meson and Bazel also have these benefits, but the key thing they are missing is buy-in.

Having been at CPPCon, however accurate this group of people are to the larger community, there was an overwhelming amount of support for CMake during relevant presentations and lightning talks, over the other two tools (IMO, because as I said, it provides the most interoperability and just "works", compared to bazel/meson that are somewhat infectious). I can say that for certain, one of the reasons my previous org is debating moving off of Julia in favor of something else like Python, is because of the difficulty in supporting the Julia stack every time there was an update.

One can argue that Julia (-lang) developers don't care for this upside-- and that's fine, but if you want more people to use Julia, better community interaction, including by using a build system that is more widely adopted, would go a long way.

[vchuravy]

#1832 has a lot of previous discussions on this.

Building the C/C++ parts of Julia is for me the smallest issue, it would probably only be a few hours for a experienced CMake user to port /src and /cli.

For me all the rest in particular dependency handling and bootstrapping Julia + stdlibs is a much bigger concern and engineering lift.

To repeat from my original request: the point is moreso how much easier it becomes to integrate the julia build into a larger project, that is either using Julia in an embedded context, or in an extension-based context. A previous org of mine had various compiler flags / stdlib requirements that when not applying those to the Julia build, would cause a slew of ABI issues (might still be occurring, no guarantee that those were truly fixed).

Now that's an actual reason, not just "make is crusty".

The problem then is more akin to you coming to my apartment and pointing out that if I had a better and newer clothes dryer, wouldn't life be much easier? I hesitantly agree and point out that I don't really think all that much about my current dryer and while it does an important job for me it works well as is, I don't think about it all that much, and it has a lot of extra features that I come to rely on.

Then you point out that a newer dryer had features that would make your life easier when you move into my apartment... (Or next to it). I and ally flatmates look confused since we don't know you and you hadn't made any indication of being interested in the apartment.

So yes it matters a lot how it impacts the quality of life of the Julia project, besides a technical argument you also need to make a social one. Who will do the change, who will maintain it, who will timely respond to issues, is there a long-term commitment?

There would be nothing worse than getting a new dryer, it breaking two months down the road and the whole house grinding to a halt, and the maintenance technician is on vacation and no-one else has the experience to fix things...

Now the argument for CMake is that it is likely that there will be someone with enough expertise around.

TLDR: there is no intention to switch from inside the house, but I won't stand in your way all that much, I only ask that you don't make my life worse, but rather easier.