Code generation changes cause CPU aborts with use of format_args!()

[berkus]

I am using nightly compiler.

It worked well until a point, and now started generating code that GPFs.

The project and branch for testing this is here: https://github.com/metta-systems/vesper/tree/investigate/invalid-code-gen

In the rust-toolchain file I have found the exact version which breaks: nightly-2022-08-12 works, while nightly-2022-08-13 started generating something incorrectly.
I have looked though the generated machine code in the ELF binary and it seems to be exactly the same.

You can test yourself by running just qemu with one or another version of nightly.

NB: UPDATED the code below seems to be misleading, there's other stuff breaking before that, see my comments below.

With 08-13 this happens:

----------------
IN:
0x00092a38:  f9400268  ldr      x8, [x19]
0x00092a3c:  a94227e0  ldp      x0, x9, [sp, #0x20]
0x00092a40:  8b141108  add      x8, x8, x20, lsl #4
0x00092a44:  a9400901  ldp      x1, x2, [x8]
0x00092a48:  f9400d28  ldr      x8, [x9, #0x18]
0x00092a4c:  d63f0100  blr      x8                                  ; this is a virtual call in function Console::replace_with()

Taking exception 3 [Prefetch Abort] on CPU 0
...from EL1 to EL1
...with ESR 0x21/0x86000000
...with FAR 0x9599000000000
...with ELR 0x9599000000000
...to EL1 PC 0x96200 PSTATE 0x3c5
----------------

The console resides in the BSS section, as it is fully zero-initialised at the start. Since I was working on BSS initialization for my OS I at first suspected something is wrong with my code, so I replaced it with a simple plain-asm version: see this commit, however, the behavior persisted and triggered a search for exact rustc version that breaks.

Version compiled with 08-12 works well with both versions of BSS init code (inline rust that calls to memset and an asm version), see this commit.

Generated code to compare should be in target/nucleus.bin

Version it worked on

nightly-2022-08-12

Version with regression

Broke on nightly-2022-08-13 and remains so.

[nagisa]

Is this generally reproducible or with QEMU only?

As a more general note, it may be necessary for you to narrow down the problem further (minimal reproducible code sample would be great), more precisely bisect the offending commit ranges (cargo-bisect-rustc can help somewhat), etc.

[berkus]

I've only noticed and tested it on QEMU so far, could try on real hw as well, but the fact that it works on the same QEMU with 08-12 means something.

I will try to bisect rustc.

[nagisa]

Reproducing this on real hardware would beyond reasonable doubt rule out any possible bugs in QEMU.

[berkus]

Reproducing this on real hardware would beyond reasonable doubt rule out any possible bugs in QEMU.

True. I'll need to restore my hw setup for that though.

[berkus]

I've validated this on real Raspberry Pi 4.

Kernel compiled with 08-12 boots fine.
Kernel compiled with 08-13 doesn't.

It will take some time to investigate the details, will need to jump through numerous gdb hoops.

[berkus]

Weird note that this triggers only in one specific kernel - the second kernel I build from same sources but that uses slightly different boot process (it relocates itself to higher memory addresses) seems to boot fine even with 08-13.

This smells of an UB.

[berkus]

I made something of a reproduction reduced to a simple use of fmt::Write (with a {} in the format string it simply stops working in 08-13 nightly but works in 08-12), will upload it soon.

[berkus]

The minimal repro code is here:

• https://github.com/metta-systems/vesper/tree/wip/rustc-102849-repro (note the branch wip/rustc-102849-repro)
• git checkout c06d89a23c5c2b0cd3ed5d270aec10054216ea5a this is a QEMU build, tip of the branch is for actual RasPi4 hardware.
• just qemu should trigger the happy case
• modify rust-toolchain file to use nightly-2022-08-13 instead of 08-12 and re-run just qemu to see it failing (not printing anything in semihosting qemu output)
• comment out a marked line in src/main.rs and just qemu to see it working again

Totally could be something I missed (an esoteric section rustc generates for format_args?), but this is the smallest I've got.

[berkus]

@nagisa could you take a look if this repro is small enough?

Just to reiterate, this happens also on a RasPi4 hardware, not only qemu.

For testing on real hardware, use tip of the branch and just build to generate nucleus.bin which can then be added onto the SD card as kernel8.img. Binary built with 08-12 toolchain outputs via Mini UART serial, binary built with 08-13 doesn't.