kernel filename for AArch64 mode

[swarren]

The VC FW already supports different filenames for ARMv6/ARMv7 (kernel.img/kernel7.img). I'd like to see something equivalent for ARMv8. This would allow construction of a single SD card that auto-selected the best possible kernel*.img that the HW the card is plugged into can support.

Since ARMv8 supports both 32- and 64-bit boot, I'd suggest filenames kern8-32.img and kern8-64.img. Those filenames contain "kern" not "kernel" so that they fit into 8.3 format.

With the existing filenames, I believe the FW tries kernel7.img first if it's relevant to the current HW, then falls back to kernel.img if kernel7.img wasn't found or wasn't attempted. I assume doing something similar for the new filenames would make sense.

I'm not sure if config.txt option arm_control should determine whether the FW looks for kern8-32.img vs. kern8-64.img, or if the presence of one or the other filename should override the value of arm_control? The latter would allow more automatic support of all possible HW with a single SD card.

In other words, I'd like one of the following algorithms:

if running_on_armv8_hw:
if exists("kern8-64.img"):
boot that in AArch64 mode
if failed, error
if exists("kern8-32.img"):
boot that in AArch32 mode
if failed, error
if running_on_armv7_or_armv8_hw:
# that if might just be running_on_armv7_hw, but I assume it has to check both
# for backwards compatibility with current FW behaviour on ARMv8
if exists("kernel7.img"):
boot that (in AArch32 mode if HW distinguishes)
if failed, error
if exists("kernel.img"):
boot that (in AArch32 mode if HW distinguishes)
if failed, error
error

or:

if running_on_armv8_hw:
if arm_control & 0x200:
if exists("kern8-64.img"):
boot that in AArch64 mode
if failed, error
else:
if exists("kern8-32.img):
boot that in AArch32 mode
if failed, error
if running_on_armv7_or_armv8_hw:
# that if might just be running_on_armv7_hw, but I assume it has to check both
# for backwards compatibility with current FW behaviour on ARMv8
if exists("kernel7.img"):
boot that (not sure what to do if arm_control & 0x200)
if failed, error
if exists("kernel.img"):
boot that (not sure what to do if arm_control & 0x200)
if failed, error
error

Does this all seem reasonable?

[capnm]

Does this all seem reasonable?

I believe that just

if (rpi3 && kernel8.img) {
        initialize and boot the ARM core in Aarch64 mode according
        https://www.kernel.org/doc/Documentation/arm64/booting.txt
}

would be a considerably better approach.

Why?

• kernel7.img works fine in AArch32 mode
• "Sorry, I think the issues ... were because I'd forgotten to disable the config.txt options I used to make AArch64 booting work on the RPi3."
  Sounds familiar? ;)
• without a proper SoC documentation it is just guess work (or maybe even not possible) to set-up the system correctly
  E.g. the VC currently enables in the AArch32 mode the hw data coherency. It may be too late to set it yourself, see the suspicious note in the manual:
  "Set the SMPEN bit before enabling the caches, ..." (ARM DDI 0500F page 4-253).

[popcornmix]

I think I agree with @capnm
As kern8-32.img and kernel7.img will be so similar I wonder if how valuable it is to support kern8-32.img?

I completely agree that if kernel8.img exists and we're running on a 64-bit capable arm that we should use kernel8.img and enable 64-bit mode automatically. Otherwise we fall back to booting kernel7.img in 32-bit mode.

[pelwell]

That's also what I thought - kernel8 for 64-bit, kernel7 for 32-bit. I think the interest in a dedicated ARMv8-32 kernel is going to be very limited.

[swarren]

I definitely see a need for separate 32- and 64-bit images for the RPi3, or put another way for separate 32-bit images for the RPi2 and RPi3. The RPi2 32-bit image would default to using the PL01X UART but the RPi3 32-bit image would default to using the mini UART.

[pelwell]

Which element of the kernel would access a UART without explicitly being told to, either through cmdline.txt or through DT?

[swarren]

Remember that kernel*.IMG is not always the Linux kernel; it's whatever bootloader or operating system the user has put onto the SD card. Not all of them user DT. At least U-Boot doesn't use the DT or command-line in any way at all at present, since a major part of the choice to use U-Boot is a desire to load kernel and DT from other places (e.g. network boot).

[swarren]

Oh and there's also earlyprintk/DEBUG_LL in Linux, although admittedly if you're using that you'd probably just have one kernel*.img present. 32-bit kernels hard-code the DEBUG_LL UART at build time, although I believe 64-bit kernels take a command-line parameter to configure this at run-time.

[pelwell]

So are you saying that U-Boot expects a UART to be configured by something, but has no way of receiving information from that something?

[MilhouseVH]

I did build OpenELEC kernel and userland for ARMv8-32 (targeting cortex-a53 CPU with armv8-a+crc arch, and neon-fp-armv8 fpu) and there was no discernible performance difference over a regular Cortex-A7 build running on an RPi3. This was with gcc 5.3.0.

ARMv8-32 really doesn't seem to provide any benefit over ARMv7-32.

[susman]

@MilhouseVH I just finished Gentoo build with the same optimization, can you share performance tests you've been running to determine the difference?

[MilhouseVH]

I ran a number of ffmpeg benchmarks, transcoding various video and audio formats, and the results (time to complete the tests) were identical on both ARMv8-32 and ARMv7, so concluded there was no benefit from the NEON or CPU specific updates.

Note: It wasn't exhaustive testing, but certainly there was zero indication of any difference at the level I was interested in (OpenELEC being a media center-focused distribution).

[susman]

Understood. I guess multimedia won't really benefit from v8 upgrades, encryption is the interesting part now.

[swarren]

So are you saying that U-Boot expects a UART to be configured by something,
but has no way of receiving information from that something?

Yes, but I don't think the two parts of that statement are related, or that the statement is a bad thing.

Re: U-Boot not receiving DT from the FW:

The original reason I ported U-Boot was to allow network booting of a mainline kernel, to speed up mainline kernel development. In this scenario, both the kernel and DT passed to the kernel come from the mainline kernel source tree, not the Foundation source tree. So, there was no need for U-Boot to receive and pass on the FW-provided DT.

Also, U-Boot has historically been able to determine all information about the hardware from either (a) firmware calls (e.g. RAM size, board ID) or (b) standard enumeration protocols (e.g. USB). There's been no ARM-visible configurability in HW use that's affected the peripherals U-Boot controls (UART, MMC, HDMI). So, there's been no need for U-Boot to receive the FW-provided DT. Consequently, code to do so doesn't currently exist in U-Boot.

Finally, at least early on (I haven't checked very recently), the Foundation kernel at least partially used/s DT bindings that had not been reviewed by the DT binding maintainers. I felt it would be odd to have mainline U-Boot process bindings that were not part of the mainline Linux kernel (mainline U-Boot support was aimed primarily at use with the mainline Linux kernel). Doing so would cause issues if/when the bindings were reviewed and incorporated into mainline Linux, since there are invariably some fixes/clean-ups during that process. I didn't want to have to support two sets of bindings in U-Boot (and do so forever since it's an ABI); it was easier and completely functional just to ignore DT initially.

So, that explains why U-Boot doesn't use DT to configure itself; there's been no need so far. This may change going forward; the UART "switching" on the RPi 3 may need to be determined from the FW-provided DT (although I might be tempted to lean toward looking at the command-line or AUX MU enable bit instead for simplicity), and some people have expressed an interest in using U-Boot to pass the FW-provided DT to the OS kernel. However, none of that is in place yet.

Re: FW UART initialization

On RPi 0/1/2, IIUC the FW always initializes the pinmux to route the UART controller signals to the GPIO header pins and programs the UART baud rate divider according to config.txt settings (or defaults if not specified). I'd like the FW to do this in all situations, irrespective of which UART is in use, so that any code run after the FW can rely on this and not duplicate the functionality. I interpret (or hope that!) the original FW behaviour is categorized as "the in-use UART is initialized" rather than "the PL01x UART is initialized".

In my opinion, this is entirely orthogonal to how that code determines which UART to use (hard-coded, by parsing DT, ...). Put another way, even if U-Boot grows the ability to parse the FW-provided DT to determine which UART to use at run-time, I don't want to duplicate the UART, AUX enable, and pinmux setup for it.

Why?

kernel7.img works fine in AArch32 mode

It depends on how kernel7.img works and what it does. U-Boot (and I'd expect a lot of bare metal and perhaps non-Linux OS code) simply hard-codes use of a PL01x at present, since that's all that's been used before. It's much quicker and easier to port to the new HW by creating a separate build for it that hard-codes use of the mini UART instead. Even if this solution isn't the final long-term plan, it's still a reasonable thing to do in the interim. Since there are therefore differences between 32-bit builds, I'd argue there are cases where your statement isn't true.

"Sorry, I think the issues ... were because I'd forgotten to disable the config.txt
options I used to make AArch64 booting work on the RPi3."
Sounds familiar? ;)

Yes, but whether to have a kernel8-32.img or not doesn't affect this. There are two sets of flags needed to use AArch64 right now: arm_control to actually boot the CPU in AArch64, and kernel_old/disable_commandline_tags to work around the lack of AArch64 "boot stub" code embedded in the VC FW at present. The arm_control value (or at least the AArch64 bit in it) can be derived from the set of filenames present on the SD card. However, kernel_old/disable_commandline_tags probably shouldn't be, since the user should always have the flexibility to enable those options even if/when the VC FW does get AArch64 "boot stub" code added to it. Thus, I'll always be able to shoot myself in the foot by screwing up config.txt:-( (I'm going to go write some scripts now to set up config.txt for all the cases to avoid doing it manually and making mistakes).

without a proper SoC documentation it is just guess work (or maybe even
not possible) to set-up the system correctly
E.g. the VC currently enables in the AArch32 mode the hw data coherency.
It may be too late to set it yourself, see the suspicious note in the manual:
"Set the SMPEN bit before enabling the caches, ..." (ARM DDI 0500F page 4-253).

I don't understand this issue; could you elaborate some more? I've been able to boot U-Boot on the RPi3 in both AArch32 and AArch64 mode without issue, although admittedly I haven't done anything with CPUs 1..3 other than put them into a spin-loop, since U-Boot is single-threaded.

ARMv8-32 really doesn't seem to provide any benefit over ARMv7-32.

I can believe this in terms of raw performance of C-compiled code with current compilers. However, IIRC there are some new instructions in ARMv8 AArch32 mode. It might be nice to differentiate between cases where they're available vs. not based on the kernel filename. I don't recall what those instructions are right now, so I don't know if this is simply a matter of compilers not yet taking advantage of them, or the new instructions having targetted use-cases that make it difficult for compilers to ever use them automatically and thus relegating their use to hand-written/-optimized assembly.

[capnm]

The original reason I ported U-Boot was to allow network booting of a mainline kernel, to speed up mainline kernel development.

(OT) I think having a working U-Boot/Grub is a great idea, but particularly for the kernel development? You can easily create more boot partitions, scp the kernel.img, config.txt, etc. there, echo "part number" > /sys/module/bcm27*/parameters/reboot_part and reboot to the test kernel without any additional layer.

It depends on how kernel7.img works and what it does. U-Boot (and I'd expect a lot of bare metal and perhaps non-Linux OS code) simply hard-codes use of a PL01x at present, since that's all that's been used before ...

Personally I also think if linux can switch the implementation, non OS code can do it too. Simply parse the VC-given command line (or other data structures). A little more work and you don't need to worry about a zillion confusing, error-prone and hard to maintain image versions.

Yes, but whether to have a kernel8-32.img or not doesn't affect this. There are two sets of flags needed to use AArch64 right now: arm_control to actually boot the CPU in AArch64, and kernel_old/disable_commandline_tags to work around the lack of AArch64 "boot stub" code embedded in the VC FW at present.

My point was, the arm_control hack or other more clever config options shouldn't be there. Again, confusing, error-prone and hard to maintain, you already burnt yourself.
One what-ever-kernel8 image (similar to the kernel7) that get booted with a proper initialized AArch64 core. Please :-)

I don't understand this issue; could you elaborate some more? I've been able to boot U-Boot on the RPi3 in both AArch32 and AArch64 mode without issue, although admittedly I haven't done anything with CPUs 1..3 other than put them into a spin-loop, since U-Boot is single-threaded.

Neither do I. You may know, the internal SoC's structure is zero documented. But if you put the dom's comments spilled in the baremetal thread and the A53 manual together, one may conclude that without a similar VC SMP initialization code is the ARM core hardly usable...

Generally I really can't understand why by asking for the AArch64 support, everybody immediately starts talking about a speed. Wasn't Raspberry Pi made for education purposes in the first place? If I needed a fast device, I certainly wouldn't use this one.

[swarren]

@capnm the fact that I made a mistake while tired and being confused by a FW bug shouldn't disallow FW from implementing a complete set of kernel image filenames.

The /sys/.../reboot_part trick doesn't work for me since that means (a) I need to always be able to boot all the way into a fully working (downstream) kernel in order to trigger that. Also, I don't believe my WiFi SD card will allow me to access anything other than the first FAT partition, so I can't use that WAR to write files to where I'd need.

I'll note again: My preferred set of kernel image filenames and FW algorithm for handling those eliminates the need for the arm_control config.txt option. My proposal makes it simpler to get things right, not harder. The mistake I made wouldn't have been possible if my preferred algorithm above was implemented.

Re: SMPEN: This appears to be a bit in the CPUECTLR register. The FW-provided "ARM boot stub" for 32-bit booting does sets this bit during the early boot process. Any equivalent boot stub for 64-bit booting will need to do the same. I don't believe any of this is relevant to this bug though; the 64-bit boot stub will need to do this no matter what the kernel image's filename is, and no matter how it determines whether to boot into 32- or 64-bit mode.

[capnm]

WiFi SD card ...

(OT) Interesting idea. IIRC I had one (unfortunately SD sized) where it was even possible to replace the firmware with your own.

I'll note again: My preferred set of kernel image filenames ...

I don't understand the need for that complexity myself, but - as long as my image simply boots in the AArch64 mode, I am restless happy.

Re: SMPEN: ... I don't believe any of this is relevant to this bug though

Yes, sorry, thanks for creating #579 👍

[swarren]

Closing. This is implemented as per #579 (comment), and I validated that it worked at least with the pre-release FW.