RFC: Heterogeneous architecture projects for meson

[lethalbit]

Problem

As it stands the current way meson supports cross builds for things like firmware only allow for the native architecture of the platform as well as the cross target itself.

This is fine for normal cross builds and firmware projects which are comprised of a homogeneous architecture, but for any sufficiently large embedded project, there might be two, or even more different none native architectures that need to be built.

When also factoring in the need to possibly build for the native system as well, such as firmware flashing utilities, the current method of using meson cross files prevents this from being a single meson project.

This means that one needs to use external wrapper scripts to orchestrate the project build and I feel that removes most of the incentive to use meson at all, when this could be done, admittedly less cleanly with makefiles.

This also prevents the building of heterogeneous firmware images, where for example, you have an arm native MCU, and FPGA bitstream, and a RISC-V firmware image for the soft-core on the FPGA which all need to be rolled into a single binary image.

Proposal

What I'm suggesting is a new platform abstract that users can leverage, which would be based around the same cross file mechanisms that exist already, but allow for multiple architectures in a unified meson project, but also not vastly alter how native or existing cross builds would work.

This is already how it's kinda done for cross builds with the host_machine, target_machine, build_machine objects, but rather than locking it to a single or dual architecture at most for the whole project, it would allow for large projects with many heterogeneous architectures to be composed all at once, allowing for the powerful dependency tracking that meson provides, and also removing the need for wrapper scripts and multiple meson projects per architectural component of the overall project.

Proposed Syntax

The general idea behind this is the introduction of a platform object, which would abstract the toolchain internals into something which could be passed around to things like library, executable, etc.

My initial thoughts are that it could potentially look something like this for basic usage:

platform = import('platform')

rv32 = platform.from_cross_file(meson.source_root() / 'cross_files' / 'rv32.ini')
arm9 = platform.from_cross_file(meson.source_root() / 'cross_files' / 'arm9.ini')

arm9_cc = arm9.get_compiler('cpp')

extended_warnings = [
	'-Wformat=2',
	'-Wformat-overflow=2',
	'-Wformat-signedness',
	'-Wformat-truncation',
	'-Wnull-dereference',
	'-Wreturn-type',
	'-Wunsafe-loop-optimizations',
	'-Wbad-function-cast',
	'-Wcast-qual',
	'-Wcast-align=strict',
	'-Wcast-function-type',
	'-Wconversion',
	'-Wdangling-else',
	'-Wsign-conversion',
	'-Wpacked',
	'-Wpadded',
]

add_project_arguments(
	arm9_cxx.get_supported_arguments(extended_warnings),
	language: 'cpp'
)

fw1 = executable(
    ...,
    platform: rv32
)

fw2 = executable(
	...,
	platform: arm9
)

native = executable(
	...
)

This would allow the project to build 3 independent components, each with their own architecture.

I would imagine that this change also wouldn't need anything new on old meson projects or cross builds, the same semantics could still be used, but leverage the new platform internals.

Prior Art

• Build multiple cross binaries #5125
• Assembling a multiarch build artifact #5157

[dcbaker]

There is another manifestation of this, and that's targeting virtual machines, what is the machine for java or .net? And also what do we do about so-called "fat" binaries, which both AIX and macOS use to put multiple architectures in the same binary?

This significantly changes the way cross compilation works in Meson, and also has an impact on the way our machine files are defined, and is a huge undertaking to re-architect things internally. I'm not saying no, but laying out that this isa huge amount of work.

[lethalbit]

Yeah I'm aware that a large amount of work would be needed to be done internally for this to be supported, and I'm more than willing to help if this is planned out and accepted. I feel that it would be a massive improvement.

I'm, not entirely sure what to do about fat binaries or something like Java or .NET personally.

I suppose the former would heavily depend on the platform and toolchain, most of the heavy lifting in that aspect is done by the linker afaik so figuring out how that would fit in is important, and I have no idea if looking into that is already being done.

As for Java/.NET, I'm not super familiar with either of those ecosystems, so I'm not entirely sure of what to do there myself unfortunately.

[jpakkane]

There is a mention of this in the dev docs: https://mesonbuild.com/Contributing.html#random-design-points-that-fit-nowhere-else

Specifically: "Any build directory will have at most two toolchains: one native and one cross."

Specifying multiple platforms like these inside the build files is not a good idea, because then you can't freely mix projects and subprojects. If you master project uses multiple things like this, then it can only use dependencies that have the same support. For example if you need zlib built for multiple CPU types, you can't use it as a subproject unless it, too, has that support. The end result of this is that every project must have build definition code for all targets and things just break apart and become unworkable monsters.

Having a combination part like this is something that can be provided, but it needs to be expressed outside the meson.build files somehow. This probably means some sort of an "aggregate project" thing on top for combining the results of multiple isolated build directories into one.

[lethalbit]

Hey, sorry for the delayed response, life is a pain.

Anyway, I can see where you're coming from with regards to that, but I think the only at most two toolchains limitation is arbitrarily limiting.

In the case where someone is using this, I don't think the limitation of being unable to freely mix subprojects and projects being a big deal honestly. And if it is such a large deal, then it could also be possible to pass down the platform into the subproject as it's "native" compiler then as well.

Maybe something like adding a platform: entry to dependency() and subproject()

I can see this feature being abused to build the same code for multiple architectures and why that's a problem, but for a sufficiently large project that has very limited shared code that spans over several architectures, but the end result is packed into a cohesive image (I.E firmware + FPGA gateware) having the ability to use Meson's powerful dependency tracking is a huge benefit.

The "aggregate project" solution feels like more of a hack than a proper fix to this problem, having multiple build directories and needing to invoke Meson multiple times sounds like more trouble than it's worth, and not being able to share a common build directory limits some things, like my previously mentioned firmware instance, where, lets say, RISC-V and ARMv7 firmware need to be merged into a single image.

I might be misunderstanding you points, but I don't feel like the initial suggestion would cause too much havoc if any.

[jpakkane]

having multiple build directories and needing to invoke Meson multiple times sounds like more trouble than it's worth

That is not how it would work, regardless of how it gets implemented. There would be only one main build dir and you'd run ninja only once. Anything else would be a poor UX. For example it could be implemented with a build tree layout like this:

build
  build-arm
  build-riscv
  build-whatever

Running Ninja at the top level would do all the requisite magic needed to build all subcomponents. The "top level" would then combine the results into a cohesive single image somehow.

[lethalbit]

Ah, okay, sorry I misunderstood. That makes sense, no complaints from me on that.

[lethalbit]

Hey, just a quick check in to see if any more thought has been put into the way that the aggregate projects should look, as I'm someone who is outside the project I don't quite know how the core maintainers would like it to look.

I'm more than happy to put in the legwork to actually work on and test the feature (as I have a few projects that desperately need it). But I'm a touch lost in the direction that people would like this to be taken.

What do you think an ideal aggregate project structure would look like?

[medhefgo]

The general idea behind this is the introduction of a platform object, which would abstract the toolchain internals into something which could be passed around to things like library, executable, etc.

In principle, meson already has everything needed to do this via the host/built/target machinery. All that's really needed is adding support for adding/defining more host machines with custom names with paths to cross-files for defaults and a means to override them via cross files by command line. Everything else is just a matter of passing the target name to native args.

This should also mean minimal work needed on the meson side.

declare_machine('rv32', default : meson.source_root() / 'cross_files' / 'rv32.ini')
declare_machine('arm9', default : meson.source_root() / 'cross_files' / 'arm9.ini')

extended_warnings = [
	'-Wformat=2',
	'-Wformat-overflow=2',
	'-Wformat-signedness',
	'-Wformat-truncation',
	'-Wnull-dereference',
	'-Wreturn-type',
	'-Wunsafe-loop-optimizations',
	'-Wbad-function-cast',
	'-Wcast-qual',
	'-Wcast-align=strict',
	'-Wcast-function-type',
	'-Wconversion',
	'-Wdangling-else',
	'-Wsign-conversion',
	'-Wpacked',
	'-Wpadded',
]

add_project_arguments(extended_warnings, language: 'cpp', native : 'arm9')

fw1 = executable(..., native : 'rv32')
fw2 = executable(..., native : 'arm9')

[lethalbit]

That would work fine, assuming you could have an arbitrary number of cross-platform targets and be able to properly specify the proper toolchains for them.

For instance having native amd64 software, rv32 firmware, arm firmware, and bare metal ppc code all as targetable would be perfect.

I've not recently looked at the codebase so i'm not 100% on how feasible it would be to make such changes, but it could be interesting to hack at to see if it's workable without too much effort.

[dcbaker]

This is.... extremely non-trivial to actually implement

Our build/host machinery is basically backed by a DataStructure that look llike:

class ForMachine(Enum):
   HOST = 0
   BUILD = 1

class PerMachine[Generic[T]]:
    host: T
    build: T

    def __getitem__(self, machine: ForMachine) -> T:
       return self.host if machine is ForMachine.HOST else self.build

We basically assume two machines through the depths of our code base. I'm not saying it's impossible, but I'm saying that it's going to be a ton of work to actually implement. I don't want to be a drag here, but we still have issues cross compiling correctly between two machines, and whoever undertakes this is going to have a lot of work ahead of them.