Better support for bare metal and low level OS targets

[dcbaker]

Lets have a discussion about what is needed to make meson better support bare metal targets and low level targets such as building a libc. I think it would be nice to have this all in one place so we can better keep track of it.

[dcbaker]

@marc-h38, I'm sure you have some ideas.

[xclaesse]

Besides the cyclic dependency between libc/python, what kind of issues do you have in mind?

(I would love to build my libc as meson subproject, to have 0 dependencies, lol)

[dcbaker]

@keith-packard is building a tiny libc with meson, so I figured he'd have some suggestions.

[keith-packard]

So, I've run into a couple of minor issues. The first one is that the compiler tests which check whether you can generate and run an executable just aren't going to do very well when building the toolchain.
I've hacked around that for now with two patches in my project:

1. Add '-nostdlib' to the compiler command. This avoids trouble when linking the test app as it doesn't look for crt0.o or libc
2. Kludge in my 'exe_wrapper' to 'succeed' when running the test app

The second issue was in how the exe_wrapper stuff works. I need to run a shell script distribued with my source code as an exe_wrapper, but meson makes that hard. I hacked around that by adding the meson.source_root() to the environment and making my exe_wrapper a bit of shell code that uses the environment variable.

I'm not worrying about bootstrapping a native build environment; that's an OS distribution problem, not a build system problem.

[jpakkane]

If you always need -nodstdlib then putting it in c_args in the cross file should use it also in the sanity test. If not, that is a bug. (This was fixed within the last year, before that it was probably broken.)

[keith-packard]

Yes, -nostdlib in the cross file worked fine. However, -nostdlib also skips libgcc.a, so I had to re-add that when building my test applications, so I'd prefer to not include that.

I think what I'd prefer is to have a configuration option that skipped the compilation tests entirely. If this seems like something you'd incorporate into meson, I can cook up a patch.

[jpakkane]

I'd prefer if we could make the sanity check more robust instead. Having cross files that point to the wrong things is not that uncommon.

[keith-packard]

Then the sanity check should only attempt to compile a trivial function with no headers and check that a .o file is produced. That would at least verify that the compiler path isn't completely broken.

[jpakkane]

That is what it already does if an exe wrapper is not defined.

[keith-packard]

The code I have attempts to build a complete executable, not just a single object file. By specifying -nostdlib on the command line, this happens to "work" (in that the compile succeeds with only warnings), but it probably shouldn't have as the resulting elf file is quite broken.

I also have an exe wrapper defined as I'm building test applications and running them under qemu; my magic exe wrapper hacks can tell when meson is running the sanity check and skips actually attempting to run that file.

I certainly appreciate that meson is testing the build environment so we reduce the errors seen only once you attempt to run ninja; this is a great design goal.

And, of course, thanks much for thinking about how this issue might inform future meson changes; I've managed to work around all of the meson limitations so far and am really happy using it instead of the alternatives :-)

[marc-h38]

Thanks @dcbaker for opening this.

I think it would be nice to have this all in one place so we can better keep track of it.

Yes, although I'm going to ask forgiveness not permission and use it as an "umbrella" issue pointing at other github issues without repeating too much of them.

Probably the main issue I met is the need for a custom_target to invoke the linker directly. I understand there's no objection to address this in the future, correct? Add a binutils module #6063

"Main issue" = the issue requiring the most lines of extra (meson.build) code. Probably not the most time-consuming one, especially not if you just re-use my static_library + custom_target tricks from #6063.

In an ideal world with a large choice of nicely designed, maintained and open-source toolchains no one should ever invoke anything but the compiler front-end which should always "know better". First, I'm afraid the world of toolchains is far from ideal. Even if it were, few toolchains seem to explicitly support "bare metal", they all seem to assume some "target" environment + an organically grown, cryptic and of course non-portable grab bag of -nostartfiles -nodefaultlibs -nolibc -nostdlib -isysroot -fno-canonical-system-headers -{n,o}magic options. As detailed a bit more in issue #6063, most toolchains seem to make assumptions about "the right way" to invoke the linker and the linker options they unconditionally pass by default (cause they know better) may get in the way when you want fine-grained linker control. Typically: with a linker script but not just.

BTW does meson assume that linkers on Unix build systems are always "dynamic"? https://mesonbuild.com/howtox.html#set-dynamic-linker
https://github.com/mesonbuild/meson/blob/32e0bcc516e2a6/mesonbuild/linkers.py#L403

Maybe not the best example but even with -fuse-ld=, XCode's clang keeps passing a relatively long list of ld64- or macOS-specific and non-portable options: #6063 (comment)

For other examples just add some -verbose flag to your ld_flags, you may be surprised. Then strace -e trace=%process the actual linker invocation from the front-end and you may be even more surprised! For instance combine with -[f][no-]{pie,PIE}.

Related to the above, the [host_machine].system setting could probably use a new, "bare metal" choice to tell meson to make intelligent decisions about neither native nor cross compilation. I guess the "safest" option for now is to pick a build system you don't support? Which implies choosing a build victim first :-)
Somewhat related: Cross compiler ignored when build and host architectures are identical #5102

This wouldn't hurt: Explicitly disable pie when pie is false #4651

(a couple more items coming)

@keith-packard is building a tiny libc with meson

Saving someone a search: https://github.com/keith-packard/picolibc

[marc-h38]

Usability issue. Meson assumes this workflow:

• run meson once
• run ninja many times

However toolchains suck (breaking news...) and suck even more for "bare metal", see above. So getting everything to work takes a significant amount of meson time and effort. This work requires re-running meson and ninja every time which is expected but somewhat tedious.

Starting from some version (0.52?) meson supports multiple, layered --cross-file= (except for #3878). This is great to support multiple toolchains and/or build operating systems. However it makes long command lines.

Long story short I ended up with non-portable wrapper script(s) on top of meson, which is itself... already a "layer" on top of ninja?

I think this user interface issue could be entirely solved with two new features and one small change:

• Some kind of new setup --config=mesonsetuprc option where users could dump too long command lines into mesonsetuprc files and easily share them as opposed to mandatory README reading or a collection of do-xxx-configure wrapper scripts achieving the same thing (free autotools scare at this URL!) Isn't there a github issue for such a mesonsetuprc file already?
• Some kind of new meson setup --AND_build=myexe command or option that would chain the ninja invocation immediately after meson setup.
• meson setup --wipe build_dir/ shouldn't require build_dir/ to already exist.

I can create some new github issue(s) out of this comment if none yet, just let me know.

See also cross build definition file not flexible enough to support cross compilation #4694

[marc-h38]

RFC: what happens when c_args/ld_args come from multiple locations? #6362
I just rephrased and reformatted the description

The main confusion is: Some c_args definitions append to each other, others override each other. Which/when/why/how?
While this is technically a PR, consider it more like a github issue with a lot of sample test code.

This is not specific to "bare metal". It just hurts more in cross-compilation and maybe even more in "bare metal"

[carlescufi]

This is a very interesting project using Meson and bare metal: https://github.com/pabigot/nrfcxx. @pabigot is the author.

[marc-h38]

While objcopy and objdump (#6063) are powerful, sometimes strip is useful:
custom_target: Ambiguous target when input and output is the same file #6070

[dcbaker]

Some kind of new setup --config=mesonsetuprc option where users could dump too long command lines into mesonsetuprc files and easily share them as opposed to mandatory README reading or a collection of do-xxx-configure wrapper scripts achieving the same thing (free autotools scare at this URL!) Isn't there a github issue for such a mesonsetuprc file already?

The plan has been to be able to set meson level settings and project level settings with cross/native files. Would that do what you need?

[marc-h38]

The plan has been...

Is the plan available somewhere? I don't mind searching.

... to be able to set meson level settings and project level settings with cross/native files. Would that do what you need?

Well, it wouldn't help with finding and typing the right combination of --cross-file= in the first place. For the rest I'm not sure: is the plan to support in cross or natives files all the flags listed in meson help setup? If not which sort of subset?

Long story short I ended up with non-portable wrapper script(s) on top of meson

Of course such wrapper(s) can be implemented in Python for portability, however subprocess is IMHO far from the most convenient shell script language. Maintaining Bourne and PowerShell scripts duplicating each other is probably less work.

PS: I just remembered argparse supports this "for free" as long as no future flag makes it ambiguous: meson setup --cr=cross/base.txt --cr=cross/variant.txt.

[dcbaker]

Is the plan available somewhere? I don't mind searching.

There is an issue about it, I can't seem to find it now (I thought I assigned it to myself...)

[marc-h38]

I think I found it, please confirm: Use native files for saving the command line #4637

(I thought I assigned it to myself...)

Indeed https://github.com/mesonbuild/meson/issues?utf8=%E2%9C%93&q=is%3Aopen+assignee%3Adcbaker

[dcbaker]

I think that's related, but there at least was an issue about being able to store project and meson options in a cross file (this issue predates the native file IIRC)

[marc-h38]

There is an issue about it, I can't seem to find it now (I thought I assigned it to myself...)

@dcbaker found it and it is Generic overrider functionality #3001

UPDATE: After a discussion on IRC, it's still not clear to me whether #3001 will avoid the need for wrapper scripts like the d-xxx-configure scripts found at https://github.com/keith-packard/picolibc/tree/80528c684
#4637 looks more like a match.

[dcbaker]

The idea of #3001 is to be able to take anything that could be passed on the command line and put it in a cross/native (I'm going to call them machine files now) file. #4637 is about using that ability to replace the opened coded format we currently use to serialize command line options to meson and instead produce a valid native file

@keith-packard, @marc-h38, you can already encode paths in the machine files, although I see there are more options that Keith is using than just extending paths:

[paths]
libdir = 'picolibc/xtensa-esp32-elf/lib'
includedir = 'picolibc/xtensa-esp32-elf/include'

[binaries]
c = 'xtensa-esp32-elf-gcc'
ar = 'xtensa-esp32-elf-ar'
as = 'xtensa-esp32-elf-as'
ld = 'xtensa-esp32-elf-ld'
strip = 'xtensa-esp32-elf-strip'

[host_machine]
system = 'unknown'
cpu_family = 'esp32'
cpu = 'esp32'
endian = 'little'

[WinterMute]

Yes, -nostdlib in the cross file worked fine. However, -nostdlib also skips libgcc.a, so I had to re-add that when building my test applications, so I'd prefer to not include that.

I think what I'd prefer is to have a configuration option that skipped the compilation tests entirely. If this seems like something you'd incorporate into meson, I can cook up a patch.

Interestingly I noticed a -nolibc which seems like a recent addition to gcc - see https://gcc.gnu.org/onlinedocs/gcc/Link-Options.html which doesn't skip libgcc

[denizzzka]

Maybe it will helps to someone. I build cross binaries for MCU with clang and ldc2. Clang also used as linker.
This is my cross .ini file for Cortex-M3:

[host_machine]
system = 'bare metal'
cpu_family = 'arm'
cpu = 'cortex-m3'
endian = 'little'

[binaries]
d = 'ldc2'
c = 'clang'
strip = 'llvm-strip'

[properties]
d_args = ['--mtriple=arm-none-eabi', '-mcpu=cortex-m3'] #ldc isn't understand LLVM "vendor" code
c_args = ['-target', 'arm-unknown-none-eabi', '-mcpu=cortex-m3']
c_link_args = [
        '-target', 'arm-unknown-none-eabi',
        '-mcpu=cortex-m3',
        '--no-standard-libraries',
        '-Xlinker', '-marmelf',
        #'-Xlinker', '--fatal-warnings',
        '-L../subprojects/libopencm3/lib/', #just to convient pick linker script below
        '-Xlinker', '--script=stm32/f1/stm32f103x8.ld',
    ]

[keith-packard]

Dave Murphy <notifications@github.com> writes:

Interestingly I noticed a -nolibc which seems like a recent addition to gcc - see https://gcc.gnu.org/onlinedocs/gcc/Link-Options.html which doesn't skip libgcc

Yup. It's a recent addition though, not available in the compilers I'm using (alas).

…

-- -keith

[marc-h38]

Thanks @denizzzka . Great reminder that what meson (and autoconf) call "host", clang calls it "target".

[host_machine]
cpu_family = 'arm'
cpu = 'cortex-m3'

[properties]
c_args = ['-target', 'arm-unknown-none-eabi', '-mcpu=cortex-m3']

Considering how popular clang is, now I feel like I should have mentioned this in my doc update #6301. @jpakkane , @dcbaker , should I?

I think this is because compiling a clang /LLVM toolchain relies on a different approach that doesn't require both --host and --target: clang is apparently ./configure --target=ALL always: https://clang.llvm.org/docs/CrossCompilation.html

[dcbaker]

Using --target with clang might not get you want you want, because I still haven't gotten around to fixing the linker detection to understand that... You can work around that by creating a symlink to clang called arm-unknown-none-eabi-clang, which will make the linker detection work

[marc-h38]

As described in one of my (too long) comments above, I'm linking with a linker script and a custom_target(command: ld.xx,...)

[keith-packard]

I haven't tried clang myself as it doesn't appear to ship on debian with support libraries compiled for every target architecture (I need to support 30 different RISC-V variants).

[luqasz]

A minor annoyance:

in meson.build:

objcopy = find_program('objcopy')
bin = custom_target(
        'bin',
        capture: true,
        command: [
            objcopy, [
                '-Obinary',
                exe.full_path(),
                exe.full_path() + '.bin',
                ]
            ],
        depends: exe,
        output: 'bin'
        )

in cross.file.txt:

objcopy = 'arm-none-eabi-objcopy'

What I'd like meson to have is instead of doing find_program() just use binnaries.objcopy

[jpakkane]

Errrr? Isn't that what it does already? If you have the binary in your cross file, then find_program will return that one. It is designed for exactly this use case.

You might also consider doing this instead:

bin = custom_target(
        'bin',
        capture: true,
        command: [
            objcopy, [
                '-Obinary',
                exe,
                exe.full_path() + '.bin',
                ]
            ],
        output: 'bin'
        )

If you put a build target in the command line argument array, Meson will automatically expand it and add the necessary dependencies.

[luqasz]

bin = custom_target(
        'bin',
        capture: true,
        command: [
            objcopy, [
                '-Obinary',
                exe,
                exe.full_path() + '.bin',
                ]
            ],
        output: 'bin'
        )

results in:

meson . build --cross-file cortex-m3.conf  --buildtype=minsize
The Meson build system
Version: 0.53.1
Source dir: /Users/lkostka/workspace/stm32f1
Build dir: /Users/lkostka/workspace/stm32f1/build
Build type: cross build
Project name: stm32f1
Project version: undefined
C compiler for the build machine: cc (clang 11.0.0 "Apple clang version 11.0.0 (clang-1100.0.33.17)")
C linker for the build machine: cc APPLE ld 530
C++ compiler for the build machine: c++ (clang 11.0.0 "Apple clang version 11.0.0 (clang-1100.0.33.17)")
C++ linker for the build machine: c++ APPLE ld 530
C compiler for the host machine: arm-none-eabi-gcc (gcc 9.2.1 "arm-none-eabi-gcc (GNU Tools for Arm Embedded Processors 9-2019-q4-major) 9.2.1 20191025 (release) [ARM/arm-9-branch revision 277599]")
C linker for the host machine: arm-none-eabi-gcc GNU ld.bfd 2.33.1.20191025
C++ compiler for the host machine: arm-none-eabi-g++ (gcc 9.2.1 "arm-none-eabi-g++ (GNU Tools for Arm Embedded Processors 9-2019-q4-major) 9.2.1 20191025 (release) [ARM/arm-9-branch revision 277599]")
C++ linker for the host machine: arm-none-eabi-g++ GNU ld.bfd 2.33.1.20191025
Build machine cpu family: x86_64
Build machine cpu: x86_64
Host machine cpu family: arm
Host machine cpu: cortex-m3
Target machine cpu family: arm
Target machine cpu: cortex-m3
Program st-info found: YES (/usr/local/bin/st-info)
Program st-flash found: YES (/usr/local/bin/st-flash)
Build targets in project: 4

Found ninja-1.10.0 at /usr/local/bin/ninja

ERROR:
Cannot use target stm32f1 as a generator because it is built for the
host machine and no exe wrapper is defined or needs_exe_wrapper is
true. You might want to set `native: true` instead to build it for
the build machine.

[luqasz]

Errrr? Isn't that what it does already? If you have the binary in your cross file, then find_program will return that one. It is designed for exactly this use case.

No. Now I need to call find_program() and use it in custom_target(). My point is to allow accessing each binnary defined in [binnaries] from meson.build e.g. if I specify size = 'arm-none-eabi-size' then I'd like to access it with e.g. binnaries.size

[gerion0]

When using executable() meson does a lot of stuff by itself. For some (low level) usecases, if can be extremly useful do be able to interrupt this process or to call the bunch of tools used in executable() manually. These usecases can be:

• Writing or extending the compiler (and test/use it with meson).
• Writing or extending the linker (and test/use it with meson).
• Writing or extending any other program involved in executable or library generating.
• Do any non standard thing with any of the intermediate targets involved in building the executable ("for this operating system image, I need to change bit 25 of this object file").

For example for C programs, I would like to call the preprocessor, compiler, linker, archiver, strip etc. manually but without specifying them as external programs. Correct me, if I'm wrong, but I think, meson must know them anyway (but hides this from the user). Maybe this can be done also a step further specifying a bunch of low level targets, that can be replace the executable() (or library() etc.) call, something like:

my_source = 'my_prog.cpp`

my_obj = object('objectify_source', my_source)
my_exe = link('my_exe_full', inputs: my_obj, output: 'my_program', type: 'executable')
my_exe_stripped = strip('my_exe_stripped', inputs: my_exe, output: 'my_program_stripped')

[marc-h38]

You can compile and link separately:
https://mesonbuild.com/Build-targets.html#object-files
https://mesonbuild.com/howtox.html#set-dynamic-linker

See also example in #6063

[gerion0]

You can compile and link separately:

Only partially, when I get it right. You can compile further from object files and you can extract object files from an already existing linked app. But you are right, these commands go in the direction I thought of.

[keith-packard]

I discovered another meson limitation when building components for an embedded toolchain. In that environment, the compiler is often built with 'sysroot' support. This allows specs files to include %R directives that substitute the path to the compiler bits instead of requiring absolute paths everywhere. This capability is used to allow the compiler to be installed in an arbitrary directory by the user, without requiring any reconfiguration or recompilation.

What this means for meson is that the --prefix value that should be used while building should be automatically discovered by invoking the compiler with -print-sysroot. However, meson does not allow the prefix value to be set from the build script. Right now, I'm checking to make sure the --prefix value supplied to meson matches the -print-sysroot value emitted by the compiler and failing the configuration process when they don't match, but it would be a lot more convenient to allow setting the prefix value at runtime.

[Ericson2314]

I just want to say a very good goal is allowing a libc and a regular program (zlib, hello) to both be subprojects. Once Meson can understand that regular projects have an implicit libc dependency which the subproject provides instead of the ambient systems, we will be in very good shape!

A lot of these questions of "when can the tests run? does linking work? etc. etc.) are best resolved on a fundamental level by materializing these implicit dependencies (crt0 for anything we can run, semi-normally; dynamic loader / runtime code for dynamic linking; etc.). By understanding these things directly, vs crossing fingers the black-box toolchains are "complete" enough, we gain a lot of power and flexibility!