litex_sim fails loading opensbi into rom, doesn't even reach boot stage, hangs on liftoff

[OrkunAliOzkan]

Bug Description
My aim was to replicate the steps taken on the following issues tab litex-hub/linux-on-litex-rocket#29 to boot linux, more specifically busybox, onto a RISCV Rocket cored SoC, but rather on the litex_sim simulator than on the diligent nexy video FPGA. However, I found that an unexpected error which prevents me from progressing further in simulating a Linux-hosting Rocket-cored SoC, and I simply don't know what to do.

Method to replicate steps
Before starting, it should be mentioned that I am building all dependencies from source, and shall be listing out how one can do that, should they be keen enough to also try. It may also be helpful in finding what mistake I made may have resorted in this issue even arising in the first place. (I am also a beginner in using this technology, and open source projects for that matter, and would've find providing as much detail here quite helpful so I am just trying to help any other fellow beginners in using Litex sim! :> If I have made any mistakes while using the issues tab please let me know, I am always happy to learn and improve!)

Dependencies:

• Litex. (duh)
• Busybox. I shall be using busybox 1.36.0 (https://busybox.net/downloads/busybox-1.36.0.tar.bz2)
• Linux kernel tree. (https://github.com/litex-hub/linux/tree/litex-rebase)
• OpenSBI. (https://github.com/riscv-software-src/opensbi)
• dtc (device-tree-compiler)
• AMD Vivado
• cpio package

Steps:

1. Build RISCV Kernel Image file:

make ARCH=riscv CROSS_COMPILE=riscv64-linux-gnu-    litex_rocket_defconfig litex_rocket_initramfs.config
make ARCH=riscv CROSS_COMPILE=riscv64-linux-gnu-

This will have produced an Image file.

2. Write SoCs mapping into .json file

./litex-boards/litex_boards/targets/digilent_nexys_video.py --build --cpu-type rocket --cpu-variant full --cpu-num-cores 1 --cpu-mem-width 2 --sys-clk-freq 50e6     --with-ethernet --with-sdcard     --with-sata --sata-gen 1 --with-sata-pll-refclk     --csr-json ./csr.json

This'll produce the csr as a json file.

3. Convert csr into a dts, achievable through json2dts. Ensure interrupt control labels are the same (ref: json2dts issue for Linux on Litex #1615).

4. Compile dts into dtb.

dtc -I dts -O dtb -f mydts.dts -o mydtb.dtb

5. Build opensbi bootloader

make CROSS_COMPILE=riscv64-linux-gnu- PLATFORM=generic 
    FW_FDT_PATH=~/...path/to/.../hopefullycorrect.dtb FW_JUMP_FDT_ADDR=0x82000000

This'll make fw_jump.bin

6. Build the root file system.
   Make sure you've made busybox_1.36.0/busybox!

rm -rf initramfs
rm -rf initramfs.cpio
mkdir initramfs
pushd initramfs
mkdir -p bin sbin lib etc dev home proc sys tmp mnt nfs root \
          usr/bin usr/sbin usr/lib
sudo mknod -m 622 dev/console c 5 1
sudo mknod -m 622 dev/tty0 c 4 0
cp ../busybox_1.36.0/busybox bin/
ln -s bin/busybox ./init
cat > etc/inittab <<- "EOT"
::sysinit:/bin/busybox mount -t proc proc /proc
::sysinit:/bin/busybox mount -t devtmpfs devtmpfs /dev
::sysinit:/bin/busybox mount -t tmpfs tmpfs /tmp
::sysinit:/bin/busybox mount -t sysfs sysfs /sys
::sysinit:/bin/busybox --install -s
/dev/console::sysinit:-/bin/ash
EOT
fakeroot <<- "EOT"
find . | cpio -H newc -o > ../initramfs.cpio
EOT
popd

And rename initramfs.cpio to initrd_bb

7. Chuck all dependencies into a directory, for cleanliness sake.

8. Make a boot.json file. I used:

{
	"initrd_bb":   "0x82000000",
	"Image":       "0x80200000",
	"fw_jump.bin": "0x80000000"
}

9. Run the sim!

litex_sim --cpu-type rocket --cpu-variant full --with-ethernet --ethernet-phy-model=sim --with-sdcard --ram-init ./boot.json --rom-init ./fw_jump.bin

Expected behaviour
At least successfully reach boot stage...

Found behaviour
Prior to even reaching the Litex logo (right before listing the SoC or booting), the script would fail and terminate. The error handle it would produce is the following:

[jtagremote] loaded (0x55c47474fef0)
[spdeeprom] loaded (addr = 0x0)
[serial2console] loaded (0x55c47474fef0)
[clocker] loaded
[ethernet] loaded (0x55c47474fef0)
[gmii_ethernet] loaded (0x55c47474fef0)
[serial2tcp] loaded (0x55c47474fef0)
[xgmii_ethernet] loaded (0x55c47474fef0)
[clocker] sys_clk: freq_hz=1000000, phase_deg=0
Error opening device /dev/net/tun: No such file or directory
Check that you are running the program with root privileges
%Error: sim_rom.init:32768: $readmem file address beyond bounds of array
Aborting...
run_sim.sh: line 1: 40525 Aborted                 sudo obj_dir/Vsim

Edit no. 1
/build/sim/gateware/sim.v on line 12306 is the source of my errors. The ROM is not large enough to contain sim_rom.init, I think. If this is true, can the simulator even host an SoC running a linux distro?

I am unsure on how to proceed. Any advice would be greatly appreciated.

Edit no. 2
By any chance does anyone know what script ./build/sim/gateware/sim.v is generated as a result of? I was thinking of deep diving to try and see where exactly this rom size is getting set from...

Edit no. 3
Set --initial-rom-size to 1e6, hangs right before displaying Litex logo however no more error regarding ROM size, reattempting with 2e5, does the same thing...

[jtagremote] loaded (0x55969acccef0)
[spdeeprom] loaded (addr = 0x0)
[serial2console] loaded (0x55969acccef0)
[clocker] loaded
[ethernet] loaded (0x55969acccef0)
[gmii_ethernet] loaded (0x55969acccef0)
[serial2tcp] loaded (0x55969acccef0)
[xgmii_ethernet] loaded (0x55969acccef0)
[clocker] sys_clk: freq_hz=1000000, phase_deg=0
Error opening device /dev/net/tun: No such file or directory
Check that you are running the program with root privileges

Edit no. 4
I do not observe this hanging issue when I do not specify --rom-init to be fw_jump.bin.

Edit no. 5
I am using the following tutorial https://github.com/litex-hub/linux-on-litex-rocket/blob/master/README.md, to figure out how to boot linux onto a Rocket-cored SoC, and have reached the bios stage. I did this by removing --rom-init and instead placing fw_jump.bin as the --ram-init content instead of boot.json. However I am unsure from here on out how exactly to boot busybox.

Edit no. 6
Hangs on liftoff:

--============= Console ================--

litex> mem_list
Available memory regions:
OPENSBI  0x80000000 0x200000 
PLIC     0x0c000000 0x400000 
CLINT    0x02000000 0x10000 
ROM      0x10000000 0x20000 
SRAM     0x11000000 0x2000 
ETHMAC   0x30000000 0x2000 
CSR      0x12000000 0x10000 

litex> boot 0x80000000
Executing booted program at 0x80000000

--============= Liftoff! ===============--

Unrelated
What exactly are the boot methods available on litex_sim (which support opensbi as the bootloader)? And would you mind providing template script commands to invoke these unique ways of booting the simulator? Any help would be greatly appreciated! ^.^

[OrkunAliOzkan]

THIS CONVERSATION HAS BEEN MOVED TO: litex-hub/linux-on-litex-rocket#30