RISCV kernel fails to get to userspace

[nathanchance]

After working around a few issues with the following patch, we get a kernel that builds! :)

• arch/riscv/Makefile works around '-save-restore' is not a recognized feature for RISC-V #804.
• arch/riscv/configs/defconfig works around Issues with -fPIC on RISCV #865.
• The rest of the register shuffles work around -Wuninitialized warnings like -Wuninitialized in arch/mips/lib/uncached.c #606 and -Wuninitialized in arch/riscv/include/asm/current.h #801.

However, it does not boot in QEMU with this rootfs.

$ curl -LSs https://gist.github.com/nathanchance/58381497a3d351551d829e11bc1d58f3/raw/7189b2e2d6ecf29bc7f279af472271bbf130d338/gistfile1.txt | git apply -3v
Checking patch arch/riscv/Makefile...
Checking patch arch/riscv/configs/defconfig...
Checking patch arch/riscv/include/asm/current.h...
Checking patch arch/riscv/kernel/process.c...
Checking patch arch/riscv/kernel/stacktrace.c...
Applied patch arch/riscv/Makefile cleanly.
Applied patch arch/riscv/configs/defconfig cleanly.
Applied patch arch/riscv/include/asm/current.h cleanly.
Applied patch arch/riscv/kernel/process.c cleanly.
Applied patch arch/riscv/kernel/stacktrace.c cleanly.

$ make -j$(nproc) -s ARCH=riscv CC=clang CROSS_COMPILE=riscv64-linux-gnu- O=out.riscv distclean defconfig all
...

$ readelf -p.comment out.riscv/vmlinux

String dump of section '.comment':
  [     0]  ClangBuiltLinux clang version 11.0.0 (git://github.com/llvm/llvm-project 512c03bac449d8d40c5fc8d0ff1719f887c7fdc3)

$ qemu-system-riscv64 -M virt -m 512M -no-reboot -bios default -kernel ${HOME}/src/linux/out.riscv/arch/riscv/boot/Image -display none -serial mon:stdio -initrd ${HOME}/cbl/git/ci/images/riscv/rootfs.cpio
...
[    0.000000] OF: fdt: Ignoring memory range 0x80000000 - 0x80200000
[    0.000000] Linux version 5.6.0-rc1-00001-g7859ecfd0918 (nathan@ubuntu-x2-xlarge-x86) (ClangBuiltLinux clang version 11.0.0 (git://github.com/llvm/llvm-project 512c03bac449d8d40c5fc8d0ff1719f887c7fdc3)) #1 SMP Mon Feb 10 00:17:30 MST 2020
...
[    0.788671] Run /init as init process
/init: exec: line 7: /sbin/init: Text file busy
[    0.903957] Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000200
[    0.904631] CPU: 0 PID: 1 Comm: init Not tainted 5.6.0-rc1-00001-g7859ecfd0918 #1
[    0.905137] Call Trace:
[    0.905445] [<ffffffe00004a3e6>] walk_stackframe+0x0/0xc6
[    0.905807] [<ffffffe0007c01e6>] dump_stack+0x9e/0xd6
[    0.906076] [<ffffffe00004f250>] panic+0x112/0x2dc
[    0.906337] [<ffffffe000051888>] exit_mm+0x0/0x12a
[    0.906640] [<ffffffe000051a82>] sys_exit_group+0x0/0xe
[    0.906926] [<ffffffe000051aa6>] __wake_up_parent+0x0/0x24
[    0.907230] [<ffffffe000051a90>] __do_sys_exit_group+0x0/0x16
[    0.907568] [<ffffffe000048e3e>] ret_from_syscall+0x0/0x2
[    0.908461] ---[ end Kernel panic - not syncing: Attempted to kill init! exitcode=0x00000200 ]---

A recent version of QEMU is needed for this to work, I am using 4.2.0.

GCC works fine with the same diff.

$ make -j$(nproc) -s ARCH=riscv CC=clang CROSS_COMPILE=riscv64-linux- O=out.riscv distclean defconfig all

$ readelf -p.comment out.riscv/vmlinux

String dump of section '.comment':
  [     0]  GCC: (GNU) 9.2.0

$ qemu-system-riscv64 -M virt -m 512M -no-reboot -bios default -kernel ${HOME}/src/linux/out.riscv/arch/riscv/boot/Image -display none -serial mon:stdio -initrd ${HOME}/cbl/git/ci/images/riscv/rootfs.cpio
...
[    0.000000] OF: fdt: Ignoring memory range 0x80000000 - 0x80200000
[    0.000000] Linux version 5.6.0-rc1-00001-g7859ecfd0918 (nathan@ubuntu-x2-xlarge-x86) (gcc version 9.2.0 (GCC)) #1 SMP Mon Feb 10 00:28:21 MST 2020
...
[    0.699403] Run /init as init process
Starting syslogd: OK
Starting klogd: OK
Initializing random number generator... [    1.406427] random: dd: uninitialized urandom read (512 bytes read)
done.
Starting network: OK
Linux version 5.6.0-rc1-00001-g7859ecfd0918 (nathan@ubuntu-x2-xlarge-x86) (gcc version 9.2.0 (GCC)) #1 SMP Mon Feb 10 00:28:21 MST 2020
Linux version 5.6.0-rc1-00001-g7859ecfd0918 (nathan@ubuntu-x2-xlarge-x86) (gcc version 9.2.0 (GCC)) #1 SMP Mon Feb 10 00:28:21 MST 2020
Stopping network: OK
Saving random seed... [    2.362898] random: dd: uninitialized urandom read (512 bytes read)
done.
Stopping klogd: OK
Stopping syslogd: OK
umount: devtmpfs busy - remounted read-only
umount: can't unmount /: Invalid argument
The system is going down NOW!
Sent SIGTERM to all processes
Sent SIGKILL to all processes
Requesting system poweroff
[    4.596314] reboot: Power down

Something tells me is it is probably related to the diff in arch/riscv/include/asm/current.h but I have not looked into it in gdb close enough yet. Hopefully it isn't too complicated but I will take all the eyes I can get on this :)

[nickdesaulniers]

on Linux next we're now able to build. When I try the above qemu command, I observe:

$ qemu-system-riscv64 -M virt -m 512M -no-reboot -bios default -kernel arch/riscv/boot/Image -display none
 -serial mon:stdio -initrd /android1/boot-utils/images/riscv/rootfs.cpio 
qemu-system-riscv64: Unable to load the RISC-V firmware "opensbi-riscv64-virt-fw_jump.bin"

I had previously installed qemu-system-riscv64 via sudo apt install qemu-system-riscv64 which instead installed qemu-system-misc. This PR seems to indicate that the bios image is included in the qemu sources.

@nathanchance were you using qemu built from source? cc @palmer-dabbelt

[nickdesaulniers]

ah https://wiki.debian.org/RISC-V#Setting_up_a_riscv64_virtual_machine has more info.

[nathanchance]

I probably was using a source built QEMU but I could have sworn that it reproduced in our Docker image but I might have installed opensbi at the same time.

[nickdesaulniers]

Doesn't look like opensbi contains that fw file: https://packages.debian.org/sid/all/opensbi/filelist
see also ClangBuiltLinux/dockerimage#30

[nhukc]

I found at least the first issue stopping boot. Exec /sbin/init fails because the file has a positive number of writers and deny_write_access must be called for exec to successfully complete. The reason for this is that atomic_dec_unless_positive is not decrementing when the value passed is negative and eventually leads to positive writer counts.

After removing some inlining the problem shows up in atomic_try_cmpxchg.

The disassembly for clang:

Dump of assembler code for function atomic_try_cmpxchg:
   0xffffffe000304d02 <+0>:     addi    sp,sp,-16
   0xffffffe000304d04 <+2>:     sd      ra,8(sp)
   0xffffffe000304d06 <+4>:     sd      s0,0(sp)
   0xffffffe000304d08 <+6>:     addi    s0,sp,16
*  0xffffffe000304d0a <+8>:     lw      a5,0(a1)
*  0xffffffe000304d0c <+10>:    slli    a3,a5,0x20
*  0xffffffe000304d10 <+14>:    srli    a3,a3,0x20
*  0xffffffe000304d12 <+16>:    lr.w    a6,(a0)
*  0xffffffe000304d16 <+20>:    bne     a6,a3,0xffffffe000304d24 <atomic_try_cmpxchg+34>
   0xffffffe000304d1a <+24>:    sc.w.rl a4,a2,(a0)
   0xffffffe000304d1e <+28>:    bnez    a4,0xffffffe000304d12 <atomic_try_cmpxchg+16>
   0xffffffe000304d20 <+30>:    fence   rw,rw
   0xffffffe000304d24 <+34>:    sext.w  a0,a6
   0xffffffe000304d28 <+38>:    bne     a0,a5,0xffffffe000304d3e <atomic_try_cmpxchg+60>
   0xffffffe000304d2c <+42>:    sext.w  a1,a3
   0xffffffe000304d30 <+46>:    xor     a0,a0,a1
   0xffffffe000304d32 <+48>:    seqz    a0,a0
   0xffffffe000304d36 <+52>:    ld      s0,0(sp)
   0xffffffe000304d38 <+54>:    ld      ra,8(sp)
   0xffffffe000304d3a <+56>:    addi    sp,sp,16
   0xffffffe000304d3c <+58>:    ret
   0xffffffe000304d3e <+60>:    sw      a6,0(a1)
   0xffffffe000304d42 <+64>:    j       0xffffffe000304d2c <atomic_try_cmpxchg+42>

The disassembly for gcc:

Dump of assembler code for function atomic_try_cmpxchg:
   0xffffffe000361ab6 <+0>:     addi    sp,sp,-16
   0xffffffe000361ab8 <+2>:     sd      s0,8(sp)
   0xffffffe000361aba <+4>:     addi    s0,sp,16
*  0xffffffe000361abc <+6>:     lw      a5,0(a1)
*  0xffffffe000361abe <+8>:     lr.w    a4,(a0)
*  0xffffffe000361ac2 <+12>:    bne     a4,a5,0xffffffe000361ad0 <atomic_try_cmpxchg+26>
   0xffffffe000361ac6 <+16>:    sc.w.rl a3,a2,(a0)
   0xffffffe000361aca <+20>:    bnez    a3,0xffffffe000361abe <atomic_try_cmpxchg+8>
   0xffffffe000361acc <+22>:    fence   rw,rw
   0xffffffe000361ad0 <+26>:    sext.w  a0,a4
   0xffffffe000361ad4 <+30>:    bne     a5,a0,0xffffffe000361ae6 <atomic_try_cmpxchg+48>
   0xffffffe000361ad8 <+34>:    ld      s0,8(sp)
   0xffffffe000361ada <+36>:    sub     a0,a5,a0
   0xffffffe000361ade <+40>:    seqz    a0,a0
   0xffffffe000361ae2 <+44>:    addi    sp,sp,16
   0xffffffe000361ae4 <+46>:    ret
   0xffffffe000361ae6 <+48>:    ld      s0,8(sp)
   0xffffffe000361ae8 <+50>:    sw      a4,0(a1)
   0xffffffe000361aea <+52>:    sub     a0,a5,a0
   0xffffffe000361aee <+56>:    seqz    a0,a0
   0xffffffe000361af2 <+60>:    addi    sp,sp,16
   0xffffffe000361af4 <+62>:    ret
End of assembler dump.

I starred the lines causing the issue. The top 32 bits of the load before the critical section are truncated on the clang version. For negative numbers we end up comparing -x (32-bit) and -x (64-bit). Then we don't call the atomic store for -(x-1). I'm now looking into why the truncation happens.

[nhukc]

Patch submitted: https://lore.kernel.org/lkml/20200611183235.37508-1-nhuck@google.com/T/#u

[nathanchance]

Accepted: https://git.kernel.org/riscv/c/6c58f25e6938c073198af8b1e1832f83f8f0df33

[nathanchance]

This was released in Linux 5.8-rc2:

https://git.kernel.org/linus/6c58f25e6938c073198af8b1e1832f83f8f0df33

https://lore.kernel.org/lkml/CAHk-=whXm00G4DJ-L2XSPS+TFU_-mcL0qgcMVq62mCzuUZsCwQ@mail.gmail.com/