-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add PIE support for Xen and KVM #772
Conversation
078632c
to
dd94e5d
Compare
v2:
|
dd94e5d
to
2abdc9d
Compare
v3:
|
2abdc9d
to
de9e2c2
Compare
de9e2c2
to
206ca83
Compare
❗ Checkpatch failed Beep boop! I ran Unikraft's
Truncated logs starting from first error 8d6919c:
View complete logs | Learn more about Unikraft's coding style and contribution guidelines. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
After offline discussion with @mogasergiu, @mschlumpp, @razvand there are still open questions, yet due to: 1. the extensive length of this PR, 2. the amount of testing required with every update, and 3. the time pressure to the upcoming release the consensus was to merge as is and provide a follow-up update after the maintainers team concludes on the remaining topics.
CC: @skuenzer
Reviewed-by: Michalis Pappas michalis@unikraft.io
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewed-by: Razvan Deaconescu razvand@unikraft.io
Approved-by: Razvan Deaconescu razvand@unikraft.io
Due to the placements of the `tls` and `tls_load` Program Headers, The last two memory regions inserted in bootinfo structures will always end up unordered. Thus, make sure they are ordered by their address first, before building the memory regions and inserting them. Signed-off-by: Sergiu Moga <sergiu.moga@protonmail.com> Reviewed-by: Dragos Petre <dragos.petre27@gmail.com> Reviewed-by: Michalis Pappas <michalis@unikraft.io> Reviewed-by: Razvan Deaconescu <razvand@unikraft.io> Approved-by: Razvan Deaconescu <razvand@unikraft.io> Tested-by: Unikraft CI <monkey@unikraft.io> GitHub-Closes: #772
Since it is the final binary that is loaded and executed, make sure that `build_bootinfo` is called on the final stripped binary, instead of the debug image, whose segments' sizes may vary after stripping. Signed-off-by: Sergiu Moga <sergiu.moga@protonmail.com> Reviewed-by: Dragos Petre <dragos.petre27@gmail.com> Reviewed-by: Michalis Pappas <michalis@unikraft.io> Reviewed-by: Razvan Deaconescu <razvand@unikraft.io> Approved-by: Razvan Deaconescu <razvand@unikraft.io> Tested-by: Unikraft CI <monkey@unikraft.io> GitHub-Closes: #772
Export a linker script symbol called `_base_addr` to represent the exact reference value used by the linker to resolve absolute references to symbols. Also, similarly to the other exported symbols, create a corresponding macro, to avoid having to explicitly cast it for every reference. Signed-off-by: Sergiu Moga <sergiu.moga@protonmail.com> Reviewed-by: Dragos Petre <dragos.petre27@gmail.com> Reviewed-by: Michalis Pappas <michalis@unikraft.io> Reviewed-by: Razvan Deaconescu <razvand@unikraft.io> Approved-by: Razvan Deaconescu <razvand@unikraft.io> Tested-by: Unikraft CI <monkey@unikraft.io> GitHub-Closes: #772
Remove the `elf32-i386` linker flag that forces the final Unikernel binary output as a 32-bit kernel. Since `qemu` currently boots us through `multiboot`, a boot protocol that does not support 64-bit kernels, add a new support script to be run on the final image. This `python` script will convert the Unikernel's `ELF64` headers to their `ELF32` equivalents, thus fooling the loader into thinking it's a 32-bit kernel. Since most of the addresses contained into the fields of the headers are assumed to fit into at most 4 bytes, the script simply takes those values and directly places them in their corresponding `ELF32` header fields without truncating. The actual `ELF32` header will be prepended to the final binary together with the equivalent `ELF32` Program Headers and the Section Headers will be appended to the end of the binary so that we can keep the `Multiboot` header in the first 8192, as the specification requires. Although the specification does not require the Program Headers be in the first 8192 bytes, GRUB seems to want it to be like that. The script will be called using a newly added `Makefile.rules` file meant to contain definitions common to all platforms. Since, at this time, the only supported boot protocol on x86 KVM QEMU is `multiboot`, enable this script for x86 KVM QEMU Multiboot builds. Signed-off-by: Sergiu Moga <sergiu.moga@protonmail.com> Reviewed-by: Dragos Petre <dragos.petre27@gmail.com> Reviewed-by: Michalis Pappas <michalis@unikraft.io> Reviewed-by: Razvan Deaconescu <razvand@unikraft.io> Approved-by: Razvan Deaconescu <razvand@unikraft.io> Tested-by: Unikraft CI <monkey@unikraft.io> GitHub-Closes: #772
Add compiler and linker options to properly build the Unikernel as a static PIE. For now, only KVM and Xen are supported. Strip `*dyn*` related ELF sections from the final binary. This will cause tools such as `readelf` or `objdump` to issue some warnings or errors when using on the final image due to the stripped unnecessary sections, but they will still work and, as a trade off, the image will be as little as it can be. Signed-off-by: Sergiu Moga <sergiu.moga@protonmail.com> Reviewed-by: Dragos Petre <dragos.petre27@gmail.com> Reviewed-by: Michalis Pappas <michalis@unikraft.io> Reviewed-by: Razvan Deaconescu <razvand@unikraft.io> Approved-by: Razvan Deaconescu <razvand@unikraft.io> Tested-by: Unikraft CI <monkey@unikraft.io> GitHub-Closes: #772
In order to start supporting positional independence we need to be able to self relocate. Thus, define a new custom relocation related structure `struct uk_reloc` that is meant to hold the offset in memory where the relocation is to be applied, the offset of the value from the original symbol value computed by the linker through the linker script's base reference address, the size of the relocation and a flags field respectively (for now we only have the `UKRELOC_FLAGS_PHYS_REL` flag meant to indicate whether the relocation is to be applied based on a physical address or not, which comes in handy in bootstrap code). The final binary blob that the `mkukreloc.py` script builds is made of a signature (UKRELOC_SIGNATURE), `struct uk_reloc` entries appended to each other and ends with a zeroed out `struct uk_reloc` to act as a sentinel. This binary blob will be placed in the binary through the `.uk_reloc` ELF section, updated through `objcopy`. Therefore, in order for this forced section update to not mess up linker script exported symbols, this section must be placed towards the end of the binary. The binary blob is obtained by parsing all of the `.rela.dyn` entries, converting them into `struct uk_reloc` entries and by searching through the debug image's symbols for `_uk_reloc_` symbols created with the help of the architecture independent helper assembly macro: `ur_data`. This macro has been added to aid in replacing the references to absolute symbol values with position independent equivalents that, instead of referencing this very symbol, it places a placeholder value (UKRELOC_PLACEHOLDER) and generates a symbol to be parsed by the script to build a uk_reloc entry. Furthermore, it increases the size of the `.uk_reloc` section by one entry through the `ur_sec_updt` macro. Note that `ur_sec_updt` also inserts a dummy relocation entry in `.uk_reloc` in order to force the linker to not optimize away the symbols. A unique `uk_reloc` symbol will be generated through the `ur_sym` macro so that an `ur_*` macro can be used on the same symbol more than once. Consider the following usage example: `ur_data quad, symbol, 8, _phys` This will result in the `nm` tool, whose output the script will parse, generating such entry: `0000000000100106 T symbol_uk_reloc_data8_phys` For `mkukreloc.py` this means, with a linker script reference address of 0x100000: ``` struct uk_reloc { __u64 r_mem_off = 0x106; // 0x100106 - 0x100000 __u64 r_addr; __u32 r_sz = 8; // symbol_uk_reloc_data[8]_phys __u32 flags = UKRELOC_FLAGS_PHYS_REL; // symbol_uk_reloc_data8[_phys] } __packed; ``` In order for actual value of `symbol` to be found, since its name is at the beginning of the generated symbol, the script will now look for such entry: `0000000000100078 T symbol` // from [symbol]_uk_reloc_data8_phys and thus, `0x78` becomes the value of `r_addr` and the `struct uk_reloc` is completed and appended to the binary blob with the others. `mkukreloc.py` will be invoked through `build_uk_reloc` Makefile definition and the "section.*lma.*adjusted to.*" type of `objcopy` warnings will be ignored since they are intended and harmless. The script will also check that the distance between `.uk_reloc` and `.bss` is enough to not cause an adjustment or spill. In case it is not enough, then the script will raise an Exception. All of this functionality shall be contained within the `libukreloc` library. Signed-off-by: Sergiu Moga <sergiu.moga@protonmail.com> Reviewed-by: Dragos Petre <dragos.petre27@gmail.com> Reviewed-by: Michalis Pappas <michalis@unikraft.io> Reviewed-by: Razvan Deaconescu <razvand@unikraft.io> Approved-by: Razvan Deaconescu <razvand@unikraft.io> Tested-by: Unikraft CI <monkey@unikraft.io> GitHub-Closes: #772
Add a separate Linker Script containing all the required sections to achieve positional independence and the ability to self relocate. The `.uk_reloc`, `.dynsym` and `.rela.dyn` sections will be part of the main data segment and the last two will be stripped in the end, without causing a hole in memory, due to the fact that `.uk_reloc` already contains the forced `struct uk_reloc` relocations and, after being updated, will also contain the `struct uk_reloc` equivalents of `.rela.dyn`'s entries, which will be of the same size, so the `.bss` section will stay in place. The `.dynamic` and `.dynstr` sections become irrelevant in the final image so we will strip them as well. In order not to cause a memory hole, place them after `.comment` section. Signed-off-by: Sergiu Moga <sergiu.moga@protonmail.com> Reviewed-by: Dragos Petre <dragos.petre27@gmail.com> Reviewed-by: Michalis Pappas <michalis@unikraft.io> Reviewed-by: Razvan Deaconescu <razvand@unikraft.io> Approved-by: Razvan Deaconescu <razvand@unikraft.io> Tested-by: Unikraft CI <monkey@unikraft.io> GitHub-Closes: #772
Implement a simple inline function to properly apply, depending on the size of the relocation, a `uk_reloc` entry. `apply_uk_reloc` may cause `UBSAN` to issue false positives, in the case of the `x86` architecture which leads to the Unikernel crashing since the serial console is not initialized yet, as it is mostly intended to be called within the early self relocator. So make sure that `UBSAN` does not touch `apply_uk_reloc` because it makes sense, for `x86`, to make unaligned accesses (especially in Kernel Space) in order to resolve non-compliant relocations. Signed-off-by: Sergiu Moga <sergiu.moga@protonmail.com> Reviewed-by: Dragos Petre <dragos.petre27@gmail.com> Reviewed-by: Michalis Pappas <michalis@unikraft.io> Reviewed-by: Razvan Deaconescu <razvand@unikraft.io> Approved-by: Razvan Deaconescu <razvand@unikraft.io> Tested-by: Unikraft CI <monkey@unikraft.io> GitHub-Closes: #772
Add `do_uk_reloc` a method that parses the `.uk_reloc` section and applies the relocations accordingly. As arguments, this method receives a base physical address `r_paddr` for relocations that employ the `UKRELOC_FLAGS_PHYS_REL` flag and a base virtual address `r_vaddr` for those that do not want to be relocated against a physical address, but rather by a given virtual address (e.g. can be used with KASLR). In ARM's case, the `-fPIC`/`-fPIE`, with the help of `adrp`, makes the code reference the page-aligned address of the section and the offset this symbol has in this section where the relocations are to be applied to and thus the desired relocated value is obtained. Thus, unlike `x86` that simply does a `%rip` relative access, ARM ends up dereferencing that in-section offset when trying to get the runtime value of `__BASE_ADDR` which creates a chicken-egg problem: we apply the relocations based on the current base address, but in order to find the current base address we need to relocate the value that is placed at the address of where the value of the base address is supposed to be. We solve this with the help of the `get_rt_addr()` function, which forces an `adrp`, `add :lo12:` assembly sequence, which leads to achieving something similar to x86's `%rip` relative addressing. A case that may explain the need for `r_paddr` and `r_vaddr` would be a situation where the previous program loader would place us at a random physical address (r_paddr) and we then want to apply our own virtual mappings for KASLR (r_vaddr), over the already applied virtual mappings, if any. Furthermore, right before applying relocations, the initial, added through `mkbootinfo.py`, memory region descriptors are also relocated. This is done by subtracting the initial link time base address, that has been statically resolved by the linker in `lt_baddr`, and adding the runtime base address. Note: `lt_baddr` will generate a relocation so it will be overwritten by the relocation loop. We want to keep it in case someone may want to run the relocator multiple times for the same image. So, instead of slowing down the relocator by checking each relocation against `lt_baddr`'s address so that we do not overwrite it, simply back it up in a temporary variable and restore it after relocations are done. Signed-off-by: Sergiu Moga <sergiu.moga@protonmail.com> Reviewed-by: Dragos Petre <dragos.petre27@gmail.com> Reviewed-by: Michalis Pappas <michalis@unikraft.io> Reviewed-by: Razvan Deaconescu <razvand@unikraft.io> Approved-by: Razvan Deaconescu <razvand@unikraft.io> Tested-by: Unikraft CI <monkey@unikraft.io> GitHub-Closes: #772
Remove the in-file definition of the `ENTRY` macro and use the already existing macro from `uk/asm.h`. Signed-off-by: Sergiu Moga <sergiu.moga@protonmail.com> Reviewed-by: Dragos Petre <dragos.petre27@gmail.com> Reviewed-by: Michalis Pappas <michalis@unikraft.io> Reviewed-by: Razvan Deaconescu <razvand@unikraft.io> Approved-by: Razvan Deaconescu <razvand@unikraft.io> Tested-by: Unikraft CI <monkey@unikraft.io> GitHub-Closes: #772
In order to be position independent, `struct uk_reloc` and the self relocator based off it are used. Thus, the bootstrapping code calls `do_uk_reloc` as early as possible, by using the page-sized default stack that the Hypervisor passes onto us, because the actual stack that we use, is also used by the Hypervisor and any early world switch may cause the guest to crash. Furthermore, the obvious absolute symbol references have been replaced with their instruction pointer relative, position independent, equivalents. The `r9` register has been used as a scratch register in this case since, after a brief code analysis, it's been found to not be used anywhere else during bootstrap. Signed-off-by: Sergiu Moga <sergiu.moga@protonmail.com> Reviewed-by: Dragos Petre <dragos.petre27@gmail.com> Reviewed-by: Michalis Pappas <michalis@unikraft.io> Reviewed-by: Razvan Deaconescu <razvand@unikraft.io> Approved-by: Razvan Deaconescu <razvand@unikraft.io> Tested-by: Unikraft CI <monkey@unikraft.io> GitHub-Closes: #772
In order to cope with our static page tables used by the bootstrap code we need to also be able to relocate our page table entries if we want to be position independent on platforms that makes use of such page tables. This macro makes use of the already existing `ur_data` macro to make the script create the usual `struct uk_reloc`. However a small modification to the script makes the script do an additional lookup for corresponding `pte_attr` symbols that `ur_pte` creates, so that it knows to also add the page table entry attributes to the final value that the self relocator uses to properly resolve a relocation. Signed-off-by: Sergiu Moga <sergiu.moga@protonmail.com> Reviewed-by: Dragos Petre <dragos.petre27@gmail.com> Reviewed-by: Michalis Pappas <michalis@unikraft.io> Reviewed-by: Razvan Deaconescu <razvand@unikraft.io> Approved-by: Razvan Deaconescu <razvand@unikraft.io> Tested-by: Unikraft CI <monkey@unikraft.io> GitHub-Closes: #772
Create a custom `ur_ldr` macro to replace the `ldr` instruction in places where references to absolute symbol values are made. Signed-off-by: Sergiu Moga <sergiu.moga@protonmail.com> Reviewed-by: Dragos Petre <dragos.petre27@gmail.com> Reviewed-by: Michalis Pappas <michalis@unikraft.io> Reviewed-by: Razvan Deaconescu <razvand@unikraft.io> Approved-by: Razvan Deaconescu <razvand@unikraft.io> Tested-by: Unikraft CI <monkey@unikraft.io> GitHub-Closes: #772
Replace the static page table entries with relocatable ones through the `ur_pte` macro. Thus, we make sure that relocations take place before enabling the MMU by making a call to `do_uk_reloc` self relocator method right before jumping to `start_mmu`. Furthermore, make `Config.uk` select `LIBUKRELOC`. Signed-off-by: Sergiu Moga <sergiu.moga@protonmail.com> Reviewed-by: Dragos Petre <dragos.petre27@gmail.com> Reviewed-by: Michalis Pappas <michalis@unikraft.io> Reviewed-by: Razvan Deaconescu <razvand@unikraft.io> Approved-by: Razvan Deaconescu <razvand@unikraft.io> Tested-by: Unikraft CI <monkey@unikraft.io> GitHub-Closes: #772
Due to the high number of references to absolute symbol values in the 16-bit and 32-bit bootstrap code, a new macro, `ur_mov`, was introduced. The new macro creates a new type of `_uk_reloc_` symbol, with an `imm` suffix, to show that it is placed right after a `mov` instruction whose immediate value must be relocated/patched. Thus, update `mkukreloc.py` accordingly. Furthermore, although useless, for completeness's sake add a 64-bit variant. This variant does a `movabs` with the placeholder to guarantee that the size of the immediate is 8 bytes. Signed-off-by: Sergiu Moga <sergiu.moga@protonmail.com> Reviewed-by: Dragos Petre <dragos.petre27@gmail.com> Reviewed-by: Michalis Pappas <michalis@unikraft.io> Reviewed-by: Razvan Deaconescu <razvand@unikraft.io> Approved-by: Razvan Deaconescu <razvand@unikraft.io> Tested-by: Unikraft CI <monkey@unikraft.io> GitHub-Closes: #772
Similarly to the 64-bit C code `do_uk_reloc` self relocator, `do_uk_reloc32` is a macro that takes as argument a value equal or different from 0 to specify whether a stack is supplied or not. If the stack is not supplied, the macro can generate its own minimal scratch stack. Furthermore, the base virtual address is expected in %esi:%edi (lower 32 bits in %edi and higher 32 bits in %esi), as well as the physical base address in the %edx register. Since we are in Protected Mode, we can assume a 32-bit register is enough to hold this address. Signed-off-by: Sergiu Moga <sergiu.moga@protonmail.com> Reviewed-by: Dragos Petre <dragos.petre27@gmail.com> Reviewed-by: Michalis Pappas <michalis@unikraft.io> Reviewed-by: Razvan Deaconescu <razvand@unikraft.io> Approved-by: Razvan Deaconescu <razvand@unikraft.io> Tested-by: Unikraft CI <monkey@unikraft.io> GitHub-Closes: #772
Since `.data.boot` is meant mainly for boot specific data such as a Multiboot header, move this unrelated `x86_start16_addr` out of this section. The reason we do this is because this might cause conflict with actual boot related data (e.g. `x86_start16_addr` be placed at the beginning of the binary instead of the actual boot related header data). Signed-off-by: Sergiu Moga <sergiu.moga@protonmail.com> Reviewed-by: Dragos Petre <dragos.petre27@gmail.com> Reviewed-by: Michalis Pappas <michalis@unikraft.io> Reviewed-by: Razvan Deaconescu <razvand@unikraft.io> Approved-by: Razvan Deaconescu <razvand@unikraft.io> Tested-by: Unikraft CI <monkey@unikraft.io> GitHub-Closes: #772
Pull this functionality out of `do_uk_reloc` so that other code has access to the ability to relocate the Kernel ELF segments. Since this must be used with caution and in very restrictive conditions, a corresponding warning has been added to this function. Signed-off-by: Sergiu Moga <sergiu@unikraft.io> Reviewed-by: Dragos Petre <dragos.petre27@gmail.com> Reviewed-by: Michalis Pappas <michalis@unikraft.io> Reviewed-by: Razvan Deaconescu <razvand@unikraft.io> Approved-by: Razvan Deaconescu <razvand@unikraft.io> Tested-by: Unikraft CI <monkey@unikraft.io> GitHub-Closes: #772
Replace all `mov`'s and data declarations (`.long`, `.quad` etc.) that use absolute symbol references with the equivalent `ur_*` macro's. One exception to this type of relocation is going to be the 16-bit code, as there is no point in relocating it before SMP initialization without even knowing the physical address in the first 1MiB where the secondary cores are to start execution. So, implement dedicated `ur_*` macro's whose only use-case is the 16-bit code to cope with its existence. Furthermore, add an exception to these `start16` relocation symbols in the script. Since, on this platform, we make use of static page tables, these were also made relocatable through the `ur_pte` macro. Signed-off-by: Sergiu Moga <sergiu.moga@protonmail.com> Reviewed-by: Dragos Petre <dragos.petre27@gmail.com> Reviewed-by: Michalis Pappas <michalis@unikraft.io> Reviewed-by: Razvan Deaconescu <razvand@unikraft.io> Approved-by: Razvan Deaconescu <razvand@unikraft.io> Tested-by: Unikraft CI <monkey@unikraft.io> GitHub-Closes: #772
Before waking up the secondary cores, the SMP initialization code copies the, now position independent through `uk_reloc`, 16-bit and 32-bit bootstrapping code to a physical page, in the lower 1 Mib of physical memory. Since, at this point of execution, the immediate values used by this bootstrapping code take the form of `UK_RELOC_PLACEHOLDER`, they will need to be properly resolved after being moved into lower memory. Therefore, add a new locally defined `struct uk_reloc` array holding the hardcoded corresponding entries of these relocations, adapted to reference the desired relocation address. Use this array after memory copying the bootstrapping code to lower memory to resolve its corresponding `start16` relocations. Encode these entries with the help of `start16` related macro's and since these definitions are starting to visually occupy a lot of space, move everything to `start16_helpers.h`, a separate header file. Signed-off-by: Sergiu Moga <sergiu.moga@protonmail.com> Reviewed-by: Dragos Petre <dragos.petre27@gmail.com> Reviewed-by: Michalis Pappas <michalis@unikraft.io> Reviewed-by: Razvan Deaconescu <razvand@unikraft.io> Approved-by: Razvan Deaconescu <razvand@unikraft.io> Tested-by: Unikraft CI <monkey@unikraft.io> GitHub-Closes: #772
Building `linuxu` with the `PIE` build options results in errors being issued as the way we currently implement PIE is not suited for this platform yet. Therefore, make sure that it is impossible to build `linuxu` as long as `PIE` is enabled. Signed-off-by: Sergiu Moga <sergiu@unikraft.io> Reviewed-by: Dragos Petre <dragos.petre27@gmail.com> Reviewed-by: Michalis Pappas <michalis@unikraft.io> Reviewed-by: Razvan Deaconescu <razvand@unikraft.io> Approved-by: Razvan Deaconescu <razvand@unikraft.io> Tested-by: Unikraft CI <monkey@unikraft.io> GitHub-Closes: #772
Prerequisite checklist
checkpatch.pl
on your commit series before opening this PR;Base target
x86_64
,arm64
kvm
,xen
Additional configuration
CONFIG_OPTIMIZE_PIE
CONFIG_UKPLAT_MEMREGION_MAX_COUNT
Description of changes
Drop the
elf32-i386
linker flag through a python script that converts an ELF64's Header, Section Headers and Program Headers to their 32-bit equivalents.Make the Unikernel position independent with the help of
struct uk_reloc
, a new structure accompanied by helper symbol-generating macros and a script that analyzes the final Unikernel binary such symbols as well as the.rela.dyn
section's entries to build a binary blob inside a new custom ELF section.uk_reloc
made up ofstruct uk_reloc
entries that an early self relocator can make use of.NOTE
This is part of a larger set of Pull Requests. To make things easier and to ensure that things build as expected, it is recommended that testing is done based on #912, which includes everything. The splitting has been done to ease the review process.