plat/kvm/arm: Rework early cache clean & invalidate

[michpappas]

Prerequisite checklist

• Read the contribution guidelines regarding submitting new changes to the project;
• Tested your changes against relevant architectures and platforms;
• Ran the checkpatch.uk on your commit series before opening this PR;
• Updated relevant documentation.

Base target

• Architecture(s): [arm64]
• Platform(s): [kvm]
• Application(s): [e.g. app-python3 or N/A]

Additional configuration

Description of changes

During boot the following cache operations are required:

• Clean the cache to the PoC to avoid evicted dirty cache lines to overwrite subsequent writes.
• With the MMU / caches off, perform any accesses necessary
• Invalidate any memory accessed to avoid clean cache lines to shadow what we previously wrote into memory.

The arm64 linux boot protocol provides the requirements for the system's state before jumping into the kernel [1[. Among these it is required that upon entry:

• The MMU is off
• The D-cache for the region corresponging to the loaded image must be cleaned to the point of coherence.

These conditions provide the opportunity of optimization the existing boot code. Specifically we can skip the extremely expensive clean & invalidating of the entire image on LXBOOT and QEMU virt, and futhermore limit further cache invalidation to the regions accessed before enabling the MMU.

Additionally to the above, optimize the existing cache and invalidate functionality by using a single barrier at the end of the operation.

[michpappas]

Marking as Draft to discuss PIE with @mogasergiu and get feedback from @kubanrob.

[kubanrob]

Does OPTIMIZE_PIE imply that the Linux Boot Protocol is used? At a first glance this seems unrelated.

[michpappas]

It's unrelated indeed (also notice this is an ifndef). I do not make an explicit check for CONFIG_KVM_BOOT_PROTO_LXBOOT but rely on the MMU heuristic to also cover for QEMU virt, and potentially other platforms that don't use the linux boot protocol but exhibit the same behavior. Tbh I'm between two minds about this. On the one hand implicit detection looks like a bad idea, on the other hand it allows more flexibility. If we want to be on the safe side I can make this conditional to BOOT_PROTO_LXBOOT OR BOOT_PROTO_QEMU_VIRT.
What do you think?

[michpappas]

After discussing with @mogasergiu I am updating this to also have PIE rely on the bootloader to have the cache clean, but still unconditionally invalidate the entire image region as libukreloc won't invalidate individual cache lines right now.

[mogasergiu]

Awesome optimizations! 💯
A few minor comments.

[michpappas]

Updated commit message, fixed conditional includes according to coding style based on comments from @mogasergiu. Replaced CONFIG_OPTIMIZE_PIE with CONFIG_LIBUKRELOC based on internal discussion with @skuenzer.

[michpappas]

This has been successfully tested on QEMU both natively and emulated, and on Firecracker. @kubanrob besides the above open question, please let me know if you need to do any testing on your side, otherwise we're close to merge.

[mogasergiu]

Updated commit message, fixed conditional includes according to coding style based on comments from @mogasergiu. Replaced CONFIG_OPTIMIZE_PIE with CONFIG_LIBUKRELOC based on internal discussion with @skuenzer.

Right, that too, forgot about it, thanks for keeping it in mind! 🙏

[mogasergiu]

It is very much appreciated that you took PIE into consideration. Thank you! 🛩️

Reviewed-by: Sergiu Moga sergiu@unikraft.io

[razvand]

Approved-by: Razvan Deaconescu razvand@unikraft.io

[unikraft-bot]

✅ Checkpatch passed

Beep boop! I ran Unikraft's checkpatch.pl support script on your pull request and it all looks good!

SHA	commit	checkpatch
cc88672	plat/kvm: Do not unconditionally clean and invalidate the cache	✅
caba225	plat/kvm/arm: Do not clean & invalidate the cache after enabling the MMU	✅
74ce86b	plat/common/arm: Do not use a barrier on every invalidation	✅