Meson build system

[dontlaugh]

Step 1: Build embedded dependencies with Meson

For the embedded 3rdparty deps that are git submodules, I am creating forks into my own GitHub account and working on Meson builds for each. Non-git embedded sources are also listed here. Whether these need to be proper "subprojects" in Meson is TBD.

• cmp (Make) - shared library proof of concept here dontlaugh/cmp@2a03903
• dyncall (CMake) - compile with instructions in the root meson.build file by delegating to a CMake subproject dontlaugh@bbac955 ; Update: we might want to use Wrap for this and fetch the upstream (and way more up to date) Mercurial repo https://dyncall.org/pub/dyncall/dyncall/shortlog/tip
• libatomicops (autotools) - Maybe skip this if we can just declare that Meson only supports C11 and up.
• libtommath (Make) - dontlaugh/libtommath@f6307d8
• libuv (CMake) - dontlaugh@bb96fc6
• mimalloc (CMake) - dontlaugh@0507733
• ryu (Bazel CMake) - dontlaugh@7b6a68b
• dynasm - No build, but it is a git repo. Lua and .h files
• sha1 - No build, 1 .c and 1 .h file; not a git repo
• freebsd - No build. Single .c file, ~200 lines; not a git repo
• msinttypes - No build. two .h files; not a git repo

Meson docs on importing CMake projects and porting autotools projects

---

Why would we want a Meson build?

We maintain a rather heroic amount of Perl to build this C project. 3 thousand lines, maybe more.

 627 ./build/setup.pm
1144 ./build/probe.pm
1356 ./Configure.pl

This code detects platform stuff, finds libraries, finds toolchains, builds command line invocations for the compiler and the linker, etc. I think we should consider leaning on Meson to do all that stuff.

Meson is a Python 3 project with no dependencies besides the standard library. It generates Ninja build files (rather than make), and Ninja is also widely packaged.

Adopting Meson doesn't mean we'll never have to think about our build system again. But it will provide a framework for things like

• cross compilation
• generating source code
• ad hoc commands
• Tests
• Toggling between system libraries and embedded/submodule dependencies; See also the Wrap system docs
• Build options

Anyways, I'm willing to take a crack at this, because I like this sort of problem and I think it would be a win over the long term.

Does anyone have any thoughts on this?

Many C projects actually maintain multiple build systems. If Meson support was merged, I expect that it would live side-by-side with the bespoke Perl build system for a while.

---

Why not CMake

CMake is popular, but I don't have much else good to say about it. I've tried to learn it a couple of times, but it never sticks for me.

What about newer, space-age stuff like Buck2, Bazel?

Buck and Bazel could also do the job, but they are products designed for Google and Facebook's giant monorepos. I do have some experience with Bazel, and I think Meson is a bit friendlier to a project of this size.

Other build system discussions: #1154 #1789 #1563 #320 #1469 #1003 #999 #997 #372 #74 #597

[dontlaugh]

Hacking over here https://github.com/dontlaugh/MoarVM/tree/mesonbuild

[dontlaugh]

Attaching lists of files after clone, configure, make, and then console output. This will help me write out the Meson build definition files, which require specifying each object and executable.

01_Bare-clone.txt
02_Configure-no-args.txt
03_Make-no-args.txt
04_Console-output.txt

---

Update: after sorting/diffing 01,02,03 above, I believe I've found all the generated sources. At least for my Linux system. I need to know precisely when/how to do custom code generation so I can do that the "Meson way".

Configure.pl generates:

• Makefile
• build/mk-moar-pc.pl
• src/gen/config.c
• src/gen/config.h
• tools/check.mk

After that, running make generates:

• pkgconfig/moar.pc
• src/jit/core_templates.h (from src/jit/core_templates.expr?)
• src/jit/x64/tile_pattern.h (from src/jit/x64/tile_pattern.tile?)
• libmoar.so
• moar (executable)
• ... and an obj .o file for every .c file

Each submodule dependency does it's own thing. They will need tweaks to build with meson, as well. But it is possible to carry these patches out-of-tree so that we could - potentially - build against upstream libraries. We could even have Meson manage fetching them so that we don't need them to be git submodules, technically.

[patrickbkr]

The build system discussion cropped up one time or the other in the past. I do like the prospect of less maintenance work and more stability in our builds. (To be fair, the Perl based build system we have at the moment is rather stable. When issues with the build crop up they are usually caused by toolchain incompatibilities, not issues in the build system.) On the other hand there is the aspect of build dependencies. At the moment the only thing necessary besides a C toolchain is Perl. Adding Python and Meson to the set makes life harder for those trying their hands on a build. If we manage to move the entire thing (so the build system of NQP and Rakudo as well) over to Meson / Python, then there is not much of a difference.

vrurg did a huge revamp of the Perl based build back in 2017. @vrurg can you give your opinion on the idea to redo our buildsystem in Meson?

[patrickbkr]

@dontlaugh I really don't want to spoil your effort. Please do go ahead. As you already said, there is no reason not to keep both build systems for a while and see how things turn out.

[dontlaugh]

Yeah, I debated even mentioning my experiment (that's all it is right now), because it's a pretty big effort.

I'll make sure I can structure things such that Configure.pl and meson.build files can live side by side.

[dontlaugh]

Each dependency needs to be built in a Meson-aware way, as well. If these are a source build, Meson requires that they be checked in or cloned under a folder named subprojects. Meson can clone these with git without using submodules, so we do not necessarily need to move them out of the current 3rdparty folder.

Here is a list of projects under 3rdparty, and the type of their build.

cmp

Type: Make

dyncall

Type: CMake

libatomicops

Type: autotools

libtommath

Type: Make

libuv

Type: CMake

mimalloc

Type: CMake

ryu

Type: Bazel

Has it's own nested dependencies:
double-conversion gtest jaffer junit mersenne mersenne_java

dynasm

No build, but it is a git repo. Lua and .h files

sha1

No build, 1 .c and 1 .h file; not a git repo

freebsd

No build. Single .c file, ~200 lines; not a git repo

msinttypes

No build. two .h files; not a git repo

---

Each of these has to be built with meson, as well. This does not require any updates to our existing forks of (some of) there repositories, because Meson provides a way to use so-called diff_files that live outside of a subproject's source code, and these can be used to patch in the required build scripts.

The CMake builds might not even need that, since for simpler CMake builds Meson can interpret and convert these automatically.

[vrurg]

vrurg did a huge revamp of the Perl based build back in 2017. @vrurg can you give your opinion on the idea to redo our buildsystem in Meson?

As if I'm a big expert in build systems... ;)

Anyway, my only big objection would be if it would require Python code to be involved into a build directly. I mean, I don't care what lang a tool is written in, only don't want to see 3rd party language scripts in the repository. I was even considering cmake instead of Perl until somebody reasoned me out of it.

Also, I hope that the build environment (config), as MoarVM reports it, preserves its compatibility with NQP and Rakudo builds.

[dontlaugh]

The dinit project has examples of platform detection with Meson https://github.com/davmac314/dinit/blob/master/meson.build

[dontlaugh]

I have successfully compiled the easy deps: small Make projects and CMake projects. See the links in the description at the top of this issue.

ryu (Bazel) and libatomicops (autotools) might be harder. We'll see. Meson has advice for converting autotools projects

Update: Good news, ryu recently got CMake support. This means we might be able to import it easily. Furthermore, I've noticed that libatomicops is only required for compilers that don't support C11.

[topazus]

Hacking over here https://github.com/dontlaugh/MoarVM/tree/mesonbuild

Any ideas with using system dependencies in meson build system? Maybe add option to use system dependencies, or find to use system dependencies first if not found then use embed dependencies?

[dontlaugh]

Yes, Meson can be configured to prefer system dependencies and fall back to embedded.

Update: this talk from Meson's creator goes into some details about the approach

https://www.youtube.com/watch?v=B6LSdboN_wM