Add support for tbf_protected_region_size ELF-file symbol

[lschuermann]

Pull Request Overview

This change causes elf2tab determine the TBF protected region size from a special tbf_protected_region_size symbol in the ELF file. This can be overriden through the --protected-region-size command line parameter. If neither the parameter nor the symbol are present, elf2tab falls back onto its heuristics for determining a reasonable protected region size.

Motivation

For non-PIC apps, it is very inconvenient to need to manually calculate the apps' actual text-section start address, and then choose a protected region parameter on elf2tab's command line which matches the pre-calculated flash address such that the TBF is filled up with headers and padding to match the expected app load address. While many times simply specifying an offset in the link address and allowing elf2tab to fall back onto a reasonable well-aligned address, this mechanism can break under some configurations. The following outlines how a PIC and non-PIC target is defined in libtock-c:

TOCK_TARGETS ?= cortex-m0\
                [...]
                rv32imac|rv32imac.0x20040060.0x80002800|0x20040060|0x80002800\

(Here, the intended load address for the application is 0x20040000. To generate correct binaries, 0x60 is added as an offset to the linker-provided flash address. Deverlopers either rely on elf2tab automatically falling back onto the next-lower 256-byte aligned address, or must pass --protected-region-size 0x60 to elf2tab.)

libtock-rs currently takes a different approach1: it reserves a section for the TBF headers at the start of its flash region. However, this does not seem to work reliably. For instance, the linker is free to re-arrange segments in the ELF file, and e.g., have the RAM segment located before the flash segment. While TBF is compatible with that (by setting the init_fn_offset field properly), having the TBF header reservation at the front of the FLASH segment (which is not the first segment in the ELF file) would break the assumption that the TBF header precedes the actual process binary. Furthermore, the current ELF parsing logic interprets this .tbf_header section as the start of application flash and prepends TBF headers before that section.

By introducing support for a tbf_protected_region_size symbol (which can be set to an arbitrary value outside of any ELF segment), applications can place their FLASH segment at an appropriate offset and hint this offset to elf2tab, without placing any constraints on the ELF segment layout itself.

Testing Strategy

These changes are tested with a preliminary integration into the linker-scripts of the libtock-c and libtock-rs toolchains.

[bradjc]

and must be also passed to elf2tab as --protected-region-size 0x60.

That's not right, primarily because elf2tab has to support both pic and non-pic elfs, so always setting the protected region size would generate undesirable pic tbfs.

I ran make V=1 RISCV=1 and here is the call to elf2tab:

elf2tab -n blink --stack 2048 --app-heap 1024 --kernel-heap 1024 --kernel-major 2 --kernel-minor 0 -v -o build/blink.tab build/cortex-m0/cortex-m0.elf build/cortex-m3/cortex-m3.elf build/cortex-m4/cortex-m4.elf build/cortex-m7/cortex-m7.elf build/rv32imac/rv32imac.0x20040060.0x80002800.elf build/rv32imac/rv32imac.0x403B0060.0x3FCC0000.elf build/rv32imc/rv32imc.0x41000060.0x42008000.elf build/rv32imc/rv32imc.0x00080060.0x40008000.elf build/rv32imc/rv32imc.0x20030080.0x10005000.elf build/rv32imc/rv32imc.0x20030880.0x10008000.elf build/rv32imc/rv32imc.0x20032080.0x10008000.elf build/rv32imc/rv32imc.0x20034080.0x10008000.elf build/rv32imac/rv32imac.0x40430060.0x80004000.elf build/rv32imac/rv32imac.0x40440060.0x80007000.elf

[lschuermann]

That's not right, primarily because elf2tab has to support both pic and non-pic elfs, so always setting the protected region size would generate undesirable pic tbfs.

Sorry, I should've been more clear: this is for the current state-of-practice for non-PIC apps only. We'd eliminate the need for passing this option with this PR, assuming libtock-c's toolchain will set the tbf_header symbol correctly.

You're right -- this is not needed for PIC apps, and in practice you can even get non-PIC apps to work, as the 256 byte elf2tab alignment magic has a good chance of working:

elf2tab/src/convert.rs

Lines 476 to 491 in 9e1f707

	// Non-PIC case. As a reasonable guess we try to get our TBF
	// start address to be at a 256 byte alignment.
	let app_binary_address = fixed_address_flash.unwrap_or(0); // Already checked for `None`.
	let tbf_start_address = util::align_down(app_binary_address, 256);
	let protected_region_size = app_binary_address - tbf_start_address;
	if protected_region_size > header_length as u32 {
	// We do want to set the protected size past the header to
	// something nonzero.
	if verbose {
	println!(
	"Inserting nonzero protected region of length: {} bytes",
	protected_region_size - header_length as u32
	);
	}
	tbfheader.set_protected_size(protected_region_size - header_length as u32);
	protected_region_size

However, this is brittle and broke for me in the past. To reliably generate working apps at the intended offsets for non-PIC platforms with the current(!) libtock-c and elf2tab, keeping the linker address offset and the --protected-region-size argument in sync is required.

[bradjc]

The underlying argument in this PR is that aligning the start of a tbf to a multiple of 256 bytes (our current practice) is insufficient. Why is that?

[bradjc]

Sorry, I should've been more clear: this is for the current state-of-practice for non-PIC apps only.

libtock-c and libtock-rs are quite different, and we've (I've) worked extensively on supporting both PIC and non-PIC architectures, and creating TABs which are board independent for libtock-c. This is not the case for libtock-rs, so I think it is worth discussing why libtock-rs cannot just copy libtock-c's lead.

libtock-c has never used the --protected-region-size flag:

❯ pwd
/Users/bradjc/git/libtock-c
❯ git grep protected-region-size | wc -l
       0

That flag was added to get support for non-pic apps up and working quickly in libtock-rs.

We don't need to use 256 for the alignment. We could choose something based on the size of the app binary.

Since we add a header in the TBF, once the tbf is generated we know exactly where it needs to be placed to make the app binary start at the correct address. So one question is: why is the app being loaded at the wrong location, and how does adding a start symbol help with that?

[lschuermann]

The underlying argument in this PR is that aligning the start of a tbf to a multiple of 256 bytes (our current practice) is insufficient. Why is that?

This is a good question. In general, aligning "downward" onto 256 bytes should cover most cases. I don't know with certainty what went wrong in my case, but here's some cases where the above code would break (if I understand it correctly):

• if the app is linked so some address 0x140 and the app is intended to be loaded at address 0x100, but the generated header is larger than 64 bytes, the above magic will reserve 64 bytes but the header size will exceed that and prepend more bytes, breaking the intended app load address
• the above mechanism doesn't support headers larger than 256 byte (unlikely?)

[lschuermann]

Since we add a header in the TBF, once the tbf is generated we know exactly where it needs to be placed to make the app binary start at the correct address. So one question is: why is the app being loaded at the wrong location, and how does adding a start symbol help with that?

I think the core of this issue is that applications are linked with the intention of having them be loaded at a certain address. This address is specified in the linker script's MEMORY section. Given the way Tock traverses through the list of TBF apps, it's important to have an app built for the exact specified load address (at least in the non-PIC case).

The unintuitive / "tricky" part is that the an app must not actually be linked to this exact address. In the non-PIC case, the app must be linked to an address which leaves enough headroom for elf2tab to prepend the header, such that the app's linker-supplied address minus the prepended header size falls onto the intended app load address. This worked in the past through two mechanisms:

• assume that the app is linked for an address which has sufficient headroom to fix the generated TBF headers. Further, assume that the app load address is 256-byte aligned. Also (at least in the current implementation) the headers shouldn't exceed 256 bytes. Then, elf2tab chooses a protected region size appropriate for these assumptions using the above mechanism.
• manually supply the difference between the intended load address and the app's link address, and pass that into --protected-region-size

libtock-rs' linker script is, in theory, much more intuitive: the developer provides the intended app load address (and optionally a custom TBF header size), and libtock-rs will try to reserve a section usable for elf2tab to insert its header. This PR tries to take the intuition of this mechanism (have the developer only supply the intended address once, and then let the toolchain figure out placement from there), but circumvent the issues that libtock-rs has. Its is also compatible with libtock-c, and does not change the PIC case at all.

One could see this change as a UX adjustment: instead of guessing an offset from within elf2tab or requiring the user to manually pass an appropriate linker address and protected region size, have the user provide just the intended address to the linker once.

Here's how this could integrate with libtock-rs: tock/libtock-rs@1b69eb0 Notice how the protected size no longer needs to be kept in sync with the linker script.

And this shows how it could be integrated into libtock-c: lschuermann/libtock-c@3bfee54 Developers would now actually specify the intended load address for non-PIC apps, and can optionally specify a custom TBF header size. elf2tab can reliably reconstruct this intended load address and the protected region size from the tbf_header symbol.

Most importantly, PIC apps are entirely unaffected by these changes.

[bradjc]

I think the core of this issue is that applications are linked with the intention of having them be loaded at a certain address. This address is specified in the linker script's MEMORY section. Given the way Tock traverses through the list of TBF apps, it's important to have an app built for the exact specified load address (at least in the non-PIC case).

The unintuitive / "tricky" part is that the an app must not actually be linked to this exact address.

But this is only true for libtock-rs. This is not true for libtock-c. For libtock-c, it is exactly as expected: we specify the address the app binary should be compiled for and the linker makes it so.

onto the intended app load address

I think this might another assumption we are making differently. While yes libtock-c sort of has the expected nice round number where it implicitly expects apps to be placed, we don't explicitly have an intended load address. We link the binary several times with different fixed addresses in flash, let elf2tab add a reasonable header, and then expect the loader tool (in my case tockloader) to put the tbf in flash in a location where the kernel will run it.

One could see this change as a UX adjustment

I think the way forward is to actually forget about libtock-c and libtock-rs and instead think about what we want the interface to elf2tab to be. I generally don't like adding features to elf2tab to support non-pic apps which are not a good fit for the tock linked-list-of-apps model. I also would like to see libtock-rs implement full TAB support (as libtock-c has where one compilation supports [nearly] every board, with multiple apps) before using it as a model for designing this interface.

One of the aspects that I do like about --protected-region-size is it is not non-pic specific. In fact, one could argue the feature has nothing to do with pic/non-pic, and that it is only there so that elf2tab can help create tbfs with additional space in the protected region for the kernel to use. This particularly relevant if also using elf2tab to sign tbfs.

The drawback to all command line flags in elf2tab is that they apply to all elfs, --protected-region-size included. What about allowing the protected region size to be set via an elf symbol? That is a feature that all elfs/tbfs could use if they were so inclined. In contrast, something like a tbf_start_address symbol has no meaning for a pic app.

How a userland compilation framework generates these symbols is up to it (and outside the scope of this PR).

Also (at least in the current implementation) the headers shouldn't exceed 256 bytes.

We should fix this.

[lschuermann]

What about allowing the protected region size to be set via an elf symbol?

Ah! I mean, this is almost exactly what this change does (it just sets an absolute address instead, and calls it tbf_header). I'll happily refactor this to work with a protected_region_size symbol! And I do agree that this is more general and has better semantics over the tbf_header symbol.

[bradjc]

What about allowing the protected region size to be set via an elf symbol?

Ah! I mean, this is almost exactly what this change does (it just sets an absolute address instead, and calls it tbf_header). I'll happily refactor this to work with a protected_region_size symbol! And I do agree that this is more general and has better semantics over the tbf_header symbol.

We would need to establish what happens if both are set. Probably the elf overrides the command line.

[lschuermann]

We would need to establish what happens if both are set. Probably the elf overrides the command line.

I would've vouched for the other way around. My guess is that build infrastructures will adopt this option (at least libtock-rs in the short term, to actually build working binaries with recent elf2tab versions). The command line would give users a convenient way to override the toolchain's default value. I would print out a warning in case one overrides the other (the current patch would do so already).

But I don't have strong feelings either way. If you'd prefer the ELF symbol to take precedence, I'd be fine with that too.

[lschuermann]

I have updated the code, commit message and pull request description according to this discussion. I'm happy to change the ELF-symbol to take precedence over the command line, if desired. Tested this to work with libtock-rs and libtock-c apps. Let me know what you think.

[bradjc]

But I don't have strong feelings either way. If you'd prefer the ELF symbol to take precedence, I'd be fine with that too.

My thinking being that if an elf goes to the trouble of specifying the protected size then it's probably important to use that. A user might want to set a default for all other TBFs via the command line.

[bradjc]

Looks good. Need to update the readme.

I think we shouldn't override the elf setting with the command line. It might not make sense to set all tbfs to the same protected size, but it might be useful to be able to set a default.

[lschuermann]

Updated the readme and the ELF symbol / command line parameter precedence.

I do now realize why it would make sense for the symbol to take precedence: I missed the fact that elf2tab can consider multiple ELFs simultaneously and pack them into a TAB archive. I agree with the argument that it makes sense to add different protected region sizes for different TBFs in a single TAB.

[bradjc]

This works with the current libtock-c, correct?

[lschuermann]

Tested that libtock-c binaries work both with the existing build infrastructure, and with a patched build infrastructure which sets this symbol (and thus does not require pre-calculated addresses with headroom for the header). Also tested that this does not affect current libtock-rs apps, and that we can successfully use this mechanism to build working libtock-rs apps with a patched build infrastructure.

Produced TBFs are bit-by-bit identical with these changes, assuming the symbol not set (the below example is for c_hello on the current libtock-c unmodified master):

eafe8bdc07a823728fd4320bc4a30442d5ce9caf3946e5fd7ad4a38f8d50858c  ./build/cortex-m4/cortex-m4.tbf
15d34bb06f9aca8dbdf241565ad791e3da1d5444b9bf4f2900ca1a10246cae25  ./build/cortex-m0/cortex-m0.tbf
cba44b42c222de8bcc00289b063f7562841bba9c1a33bbcd19cff7721ba6b084  ./build/cortex-m3/cortex-m3.tbf
4e74a050eb27204ea27e525428a24ee15194e72f8886e4fa9fd5879dbb923efa  ./build/cortex-m7/cortex-m7.tbf
110e3fbc8c6aeb7ca3cb6f83afb35a05a24846d4fd6b47785d081c9c64e0a050  ./build/rv32imac/rv32imac.0x40430060.0x80004000.tbf
90de9206c15f10fff8e2e2d8f081aa9e4329388af62bad088fa4e1ff7f8d0d50  ./build/rv32imac/rv32imac.0x403B0060.0x3FCC0000.tbf
2fc6d13552ec58c57f7ccf03ff7004743cc197893acdeaadda4522b7015e4f53  ./build/rv32imac/rv32imac.0x20040060.0x80002800.tbf
baaca598d06805a06a45ce58e5ae241f88f7c04f4e40a34b37b883d1ac9cc91c  ./build/rv32imac/rv32imac.0x40440060.0x80007000.tbf
3496393994d9be83a9f5fca52f90abdc40d631edb0a05c6bdb333c22940f8ad6  ./build/rv32imc/rv32imc.0x20030880.0x10008000.tbf
8d91ab24952467a558ac73b01b818827c7de0f3879cabfcbf0a14def122fc9e0  ./build/rv32imc/rv32imc.0x20032080.0x10008000.tbf
8d2159f9b1f4db772f82e0448d70e977790abdf993b0d25cc9e8a6c35beb0e27  ./build/rv32imc/rv32imc.0x00080060.0x40008000.tbf
562496c1111199c359e7bafd089c0c33398a9a0841266418582a77002c7de250  ./build/rv32imc/rv32imc.0x20030080.0x10005000.tbf
e5e96f264bc42a781077639b6fb86a9ea0696389c148a51a3367f2c0b6e00fe7  ./build/rv32imc/rv32imc.0x41000060.0x42008000.tbf
7d95e3c203af5a61cb0c9e08c39014638e40b68e5525d5a599ad476d4849360f  ./build/rv32imc/rv32imc.0x20034080.0x10008000.tbf
--------------------------------------------------------------------------------
eafe8bdc07a823728fd4320bc4a30442d5ce9caf3946e5fd7ad4a38f8d50858c  ./build_new/cortex-m4/cortex-m4.tbf
15d34bb06f9aca8dbdf241565ad791e3da1d5444b9bf4f2900ca1a10246cae25  ./build_new/cortex-m0/cortex-m0.tbf
cba44b42c222de8bcc00289b063f7562841bba9c1a33bbcd19cff7721ba6b084  ./build_new/cortex-m3/cortex-m3.tbf
4e74a050eb27204ea27e525428a24ee15194e72f8886e4fa9fd5879dbb923efa  ./build_new/cortex-m7/cortex-m7.tbf
110e3fbc8c6aeb7ca3cb6f83afb35a05a24846d4fd6b47785d081c9c64e0a050  ./build_new/rv32imac/rv32imac.0x40430060.0x80004000.tbf
90de9206c15f10fff8e2e2d8f081aa9e4329388af62bad088fa4e1ff7f8d0d50  ./build_new/rv32imac/rv32imac.0x403B0060.0x3FCC0000.tbf
2fc6d13552ec58c57f7ccf03ff7004743cc197893acdeaadda4522b7015e4f53  ./build_new/rv32imac/rv32imac.0x20040060.0x80002800.tbf
baaca598d06805a06a45ce58e5ae241f88f7c04f4e40a34b37b883d1ac9cc91c  ./build_new/rv32imac/rv32imac.0x40440060.0x80007000.tbf
3496393994d9be83a9f5fca52f90abdc40d631edb0a05c6bdb333c22940f8ad6  ./build_new/rv32imc/rv32imc.0x20030880.0x10008000.tbf
8d91ab24952467a558ac73b01b818827c7de0f3879cabfcbf0a14def122fc9e0  ./build_new/rv32imc/rv32imc.0x20032080.0x10008000.tbf
8d2159f9b1f4db772f82e0448d70e977790abdf993b0d25cc9e8a6c35beb0e27  ./build_new/rv32imc/rv32imc.0x00080060.0x40008000.tbf
562496c1111199c359e7bafd089c0c33398a9a0841266418582a77002c7de250  ./build_new/rv32imc/rv32imc.0x20030080.0x10005000.tbf
e5e96f264bc42a781077639b6fb86a9ea0696389c148a51a3367f2c0b6e00fe7  ./build_new/rv32imc/rv32imc.0x41000060.0x42008000.tbf
7d95e3c203af5a61cb0c9e08c39014638e40b68e5525d5a599ad476d4849360f  ./build_new/rv32imc/rv32imc.0x20034080.0x10008000.tbf

[bradjc]

Ok great let's merge this. We can always change it before a release if needed.