Linux Kernel Module Cheat

The perfect emulation setup to study and develop the Linux kernel v5.9.2, kernel modules, QEMU, gem5 and x86_64, ARMv7 and ARMv8 userland and baremetal assembly, ANSI C, C++ and POSIX. GDB step debug and KGDB just work. Powered by Buildroot and crosstool-NG. Highly automated. Thoroughly documented. Automated tests. "Tested" in an Ubuntu 20.04 host.

TL;DR: Section 1.2.1, “QEMU Buildroot setup getting started”

The source code for this page is located at: https://github.com/cirosantilli/linux-kernel-module-cheat. Due to a GitHub limitation, this README is too long and not fully rendered on github.com, so either use: https://cirosantilli.com/linux-kernel-module-cheat or build the docs yourself.

• 1. Getting started
  • 1.1. Should you waste your life with systems programming?
  • 1.2. QEMU Buildroot setup
    • 1.2.1. QEMU Buildroot setup getting started
    • 1.2.2. How to hack stuff
      • 1.2.2.1. Your first Linux kernel hack
      • 1.2.2.2. Your first kernel module hack
      • 1.2.2.3. Your first glibc hack
      • 1.2.2.4. Your first Binutils hack
      • 1.2.2.5. Your first GCC hack
    • 1.2.3. About the QEMU Buildroot setup
  • 1.3. Dry run to get commands for your project
  • 1.4. gem5 Buildroot setup
    • 1.4.1. About the gem5 Buildroot setup
    • 1.4.2. gem5 Buildroot setup getting started
  • 1.5. Docker host setup
  • 1.6. Prebuilt setup
    • 1.6.1. About the prebuilt setup
    • 1.6.2. Prebuilt setup getting started
  • 1.7. Host kernel module setup
    • 1.7.1. Hello host
  • 1.8. Userland setup
    • 1.8.1. About the userland setup
    • 1.8.2. Userland setup getting started
      • 1.8.2.1. Userland setup getting started natively
      • 1.8.2.2. Userland setup getting started with prebuilt toolchain and QEMU user mode
      • 1.8.2.3. Userland setup getting started full system
  • 1.9. Baremetal setup
    • 1.9.1. About the baremetal setup
    • 1.9.2. Baremetal setup getting started
  • 1.10. Build the documentation
• 2. GDB step debug
  • 2.1. GDB step debug kernel boot
    • 2.1.1. GDB step debug kernel boot other archs
    • 2.1.2. Disable kernel compiler optimizations
  • 2.2. GDB step debug kernel post-boot
  • 2.3. tmux
    • 2.3.1. tmux gem5
  • 2.4. GDB step debug kernel module
    • 2.4.1. GDB step debug kernel module insmodded by init on ARM
    • 2.4.2. GDB module_init
      • 2.4.2.1. GDB module_init step into it
      • 2.4.2.2. GDB module_init calculate entry address
      • 2.4.2.3. GDB module_init break at the end of sys_init_module
      • 2.4.2.4. GDB module_init add trap instruction
    • 2.4.3. Bypass lx-symbols
  • 2.5. GDB step debug early boot
    • 2.5.1. Linux kernel entry point
      • 2.5.1.1. arm64 secondary CPU entry point
    • 2.5.2. Linux kernel arch-agnostic entry point
    • 2.5.3. Linux kernel early boot messages
  • 2.6. GDB step debug userland processes
    • 2.6.1. GDB step debug userland custom init
    • 2.6.2. GDB step debug userland BusyBox init
    • 2.6.3. GDB step debug userland non-init
      • 2.6.3.1. GDB step debug userland non-init without --gdb-wait
  • 2.7. GDB call
  • 2.8. GDB view ARM system registers
  • 2.9. GDB step debug multicore userland
  • 2.10. Linux kernel GDB scripts
    • 2.10.1. lx-ps
      • 2.10.1.1. CONFIG_PID_IN_CONTEXTIDR
  • 2.11. Debug the GDB remote protocol
    • 2.11.1. Remote 'g' packet reply is too long
• 3. KGDB
  • 3.1. KGDB ARM
  • 3.2. KGDB kernel modules
  • 3.3. KDB
    • 3.3.1. KDB graphic
    • 3.3.2. KDB ARM
• 4. gdbserver
  • 4.1. gdbserver BusyBox
  • 4.2. gdbserver libc
  • 4.3. gdbserver dynamic loader
• 5. CPU architecture
  • 5.1. x86_64
    • 5.1.1. ring0
  • 5.2. arm
    • 5.2.1. Run arm executable in aarch64
  • 5.3. MIPS
  • 5.4. Other architectures
• 6. init
  • 6.1. Replace init
    • 6.1.1. poweroff.out
    • 6.1.2. sleep_forever.out
    • 6.1.3. time_boot.out
  • 6.2. Run command at the end of BusyBox init
  • 6.3. Path to init
  • 6.4. Init environment
    • 6.4.1. init arguments
    • 6.4.2. init environment env
    • 6.4.3. BusyBox shell init environment
      • 6.4.3.1. BusyBox shell initrc files
• 7. initrd
  • 7.1. initrd in desktop distros
  • 7.2. initramfs
  • 7.3. rootfs
    • 7.3.1. /dev/root
  • 7.4. gem5 initrd
  • 7.5. gem5 initramfs
• 8. Device tree
  • 8.1. DTB files
  • 8.2. Device tree syntax
  • 8.3. Get device tree from a running kernel
  • 8.4. Device tree emulator generation
• 9. KVM
  • 9.1. KVM arm
  • 9.2. gem5 KVM
• 10. User mode simulation
  • 10.1. QEMU user mode getting started
    • 10.1.1. User mode GDB
  • 10.2. User mode tests
  • 10.3. User mode Buildroot executables
  • 10.4. User mode simulation with glibc
    • 10.4.1. FATAL: kernel too old failure in userland simulation
    • 10.4.2. stack smashing detected when using glibc
  • 10.5. User mode static executables
    • 10.5.1. User mode static executables with dynamic libraries
      • 10.5.1.1. C++ static and pthreads
  • 10.6. syscall emulation mode program stdin
  • 10.7. gem5 syscall emulation mode
    • 10.7.1. gem5 dynamic linked executables in syscall emulation
    • 10.7.2. gem5 syscall emulation exit status
    • 10.7.3. gem5 syscall emulation mode syscall tracing
    • 10.7.4. gem5 syscall emulation multithreading
    • 10.7.5. gem5 syscall emulation multiple executables
      • 10.7.5.1. gem5 syscall emulation --smt
  • 10.8. QEMU user mode quirks
    • 10.8.1. QEMU user mode does not show stdout immediately
      • 10.8.1.1. QEMU user mode does not show errors
• 11. Kernel module utilities
  • 11.1. insmod
  • 11.2. myinsmod
  • 11.3. modprobe
  • 11.4. kmod
    • 11.4.1. module-init-tools
    • 11.4.2. kmod modprobe
• 12. Filesystems
  • 12.1. OverlayFS
  • 12.2. Secondary disk
• 13. Graphics
  • 13.1. QEMU text mode
    • 13.1.1. Quit QEMU from text mode
  • 13.2. QEMU graphic mode
    • 13.2.1. Scroll up in graphic mode
    • 13.2.2. QEMU Graphic mode arm
      • 13.2.2.1. QEMU graphic mode arm terminal
      • 13.2.2.2. QEMU graphic mode arm terminal implementation
      • 13.2.2.3. QEMU graphic mode arm VGA
  • 13.3. gem5 graphic mode
    • 13.3.1. Graphic mode gem5 aarch64
    • 13.3.2. gem5 graphic mode DP650
    • 13.3.3. gem5 graphic mode internals
  • 13.4. X11 Buildroot
    • 13.4.1. X11 Buildroot mouse not moving
    • 13.4.2. X11 Buildroot ARM
• 14. Networking
  • 14.1. Enable networking
  • 14.2. ping
  • 14.3. Guest host networking
    • 14.3.1. Host to guest networking
      • 14.3.1.1. nc host to guest
      • 14.3.1.2. ssh into guest
      • 14.3.1.3. gem5 host to guest networking
    • 14.3.2. Guest to host networking
  • 14.4. 9P
    • 14.4.1. 9P vs NFS
    • 14.4.2. 9P getting started
    • 14.4.3. gem5 9P
    • 14.4.4. NFS
• 15. Operating systems
• 16. Linux kernel
  • 16.1. Linux kernel configuration
    • 16.1.1. Modify kernel config
    • 16.1.2. Find the kernel config
    • 16.1.3. About our Linux kernel configs
      • 16.1.3.1. About Buildroot’s kernel configs
        • 16.1.3.1.1. Linux kernel defconfig
        • 16.1.3.1.2. Linux kernel min config
      • 16.1.3.2. Notable alternate gem5 kernel configs
  • 16.2. Kernel version
    • 16.2.1. Find the kernel version
    • 16.2.2. Update the Linux kernel
      • 16.2.2.1. Update the Linux kernel LKMC procedure
    • 16.2.3. Downgrade the Linux kernel
  • 16.3. Kernel command line parameters
    • 16.3.1. Kernel command line parameters escaping
    • 16.3.2. Kernel command line parameters definition points
    • 16.3.3. rw
    • 16.3.4. norandmaps
  • 16.4. printk
    • 16.4.1. /proc/sys/kernel/printk
    • 16.4.2. ignore_loglevel
    • 16.4.3. pr_debug
      • 16.4.3.1. pr_debug != printk(KERN_DEBUG
  • 16.5. Kernel module APIs
    • 16.5.1. Kernel module parameters
      • 16.5.1.1. modprobe.conf
    • 16.5.2. Kernel module dependencies
      • 16.5.2.1. Kernel module dependencies with modprobe
    • 16.5.3. MODULE_INFO
    • 16.5.4. vermagic
    • 16.5.5. init_module
    • 16.5.6. Floating point in kernel modules
  • 16.6. Kernel panic and oops
    • 16.6.1. Kernel panic
      • 16.6.1.1. Kernel module stack trace to source line
      • 16.6.1.2. BUG_ON
      • 16.6.1.3. Exit emulator on panic
        • 16.6.1.3.1. Exit QEMU on panic
        • 16.6.1.3.2. Exit gem5 on panic
      • 16.6.1.4. Reboot on panic
      • 16.6.1.5. Panic trace show addresses instead of symbols
    • 16.6.2. Kernel oops
    • 16.6.3. dump_stack
    • 16.6.4. WARN_ON
    • 16.6.5. not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
  • 16.7. Pseudo filesystems
    • 16.7.1. debugfs
    • 16.7.2. procfs
      • 16.7.2.1. /proc/version
    • 16.7.3. sysfs
    • 16.7.4. Character devices
      • 16.7.4.1. Automatically create character device file on insmod
  • 16.8. Pseudo files
    • 16.8.1. File operations
    • 16.8.2. seq_file
      • 16.8.2.1. seq_file single_open
    • 16.8.3. poll
    • 16.8.4. ioctl
    • 16.8.5. mmap
    • 16.8.6. Anonymous inode
    • 16.8.7. netlink sockets
  • 16.9. kthread
    • 16.9.1. kthreads
    • 16.9.2. sleep
    • 16.9.3. Workqueues
      • 16.9.3.1. Workqueue from workqueue
    • 16.9.4. schedule
    • 16.9.5. Wait queues
  • 16.10. Timers
  • 16.11. IRQ
    • 16.11.1. irq.ko
    • 16.11.2. dummy-irq
    • 16.11.3. /proc/interrupts
  • 16.12. Kernel utility functions
    • 16.12.1. kstrto
    • 16.12.2. virt_to_phys
      • 16.12.2.1. Userland physical address experiments
        • 16.12.2.1.1. QEMU xp
        • 16.12.2.1.2. /dev/mem
        • 16.12.2.1.3. pagemap_dump.out
  • 16.13. Linux kernel tracing
    • 16.13.1. CONFIG_PROC_EVENTS
      • 16.13.1.1. CONFIG_PROC_EVENTS aarch64
    • 16.13.2. ftrace
      • 16.13.2.1. ftrace system calls
      • 16.13.2.2. trace-cmd
    • 16.13.3. Kprobes
    • 16.13.4. Count boot instructions
  • 16.14. Linux kernel hardening
    • 16.14.1. CONFIG_FORTIFY_SOURCE
    • 16.14.2. Linux security modules
      • 16.14.2.1. SELinux
  • 16.15. User mode Linux
  • 16.16. UIO
  • 16.17. Linux kernel interactive stuff
    • 16.17.1. Linux kernel console fun
    • 16.17.2. Linux kernel magic keys
      • 16.17.2.1. Ctrl Alt Del
      • 16.17.2.2. SysRq
    • 16.17.3. TTY
      • 16.17.3.1. Start a getty from outside of init
      • 16.17.3.2. console kernel boot parameter
    • 16.17.4. CONFIG_LOGO
  • 16.18. DRM
    • 16.18.1. kmscube
    • 16.18.2. kmscon
    • 16.18.3. libdri2
  • 16.19. Linux kernel testing
    • 16.19.1. Linux Test Project
    • 16.19.2. stress
  • 16.20. Linux kernel build system
    • 16.20.1. vmlinux vs bzImage vs zImage vs Image
  • 16.21. Virtio
  • 16.22. Kernel modules
    • 16.22.1. dump_regs
• 17. FreeBSD
• 18. RTOS
  • 18.1. Zephyr
  • 18.2. ARM Mbed
• 19. Xen
• 20. U-Boot
• 21. Emulators
• 22. QEMU
  • 22.1. Introduction to QEMU
  • 22.2. Binary translation
  • 22.3. Disk persistency
    • 22.3.1. gem5 disk persistency
  • 22.4. gem5 qcow2
  • 22.5. Snapshot
    • 22.5.1. Snapshot internals
  • 22.6. Device models
    • 22.6.1. PCI
      • 22.6.1.1. QEMU edu PCI device
      • 22.6.1.2. Manipulate PCI registers directly
      • 22.6.1.3. pciutils
      • 22.6.1.4. Introduction to PCI
      • 22.6.1.5. PCI BFD
      • 22.6.1.6. PCI BAR
    • 22.6.2. GPIO
    • 22.6.3. LEDs
    • 22.6.4. gem5 educational hardware models
  • 22.7. QEMU monitor
    • 22.7.1. QEMU monitor from guest
    • 22.7.2. QEMU monitor from GDB
  • 22.8. Debug the emulator
    • 22.8.1. Reverse debug the emulator
    • 22.8.2. Debug gem5 Python scripts
  • 22.9. Tracing
    • 22.9.1. QEMU -d tracing
    • 22.9.2. QEMU trace register values
    • 22.9.3. QEMU trace memory accesses
    • 22.9.4. Trace source lines
    • 22.9.5. QEMU record and replay
      • 22.9.5.1. QEMU reverse debugging
    • 22.9.6. QEMU trace multicore
    • 22.9.7. QEMU get guest instruction count
    • 22.9.8. gem5 tracing
      • 22.9.8.1. gem5 trace internals
      • 22.9.8.2. gem5 ExecAll trace format
      • 22.9.8.3. gem5 Registers trace format
      • 22.9.8.4. gem5 TARMAC traces
      • 22.9.8.5. gem5 tracing internals
  • 22.10. QEMU GUI is unresponsive
• 23. gem5
  • 23.1. gem5 vs QEMU
  • 23.2. gem5 run benchmark
    • 23.2.1. Skip extra benchmark instructions
  • 23.3. gem5 system parameters
    • 23.3.1. Number of cores
      • 23.3.1.1. QEMU user mode multithreading
      • 23.3.1.2. gem5 ARM full system with more than 8 cores
    • 23.3.2. gem5 cache size
    • 23.3.3. gem5 DRAM model
      • 23.3.3.1. gem5 memory latency
      • 23.3.3.2. Memory size
      • 23.3.3.3. gem5 DRAM setup
    • 23.3.4. gem5 disk and network latency
    • 23.3.5. gem5 clock frequency
  • 23.4. gem5 kernel command line parameters
  • 23.5. gem5 GDB step debug
    • 23.5.1. gem5 GDB step debug kernel
    • 23.5.2. gem5 GDB step debug userland process
    • 23.5.3. gem5 GDB step debug secondary cores
  • 23.6. gem5 checkpoint
    • 23.6.1. gem5 checkpoint userland minimal example
    • 23.6.2. gem5 checkpoint internals
    • 23.6.3. gem5 checkpoint restore and run a different script
    • 23.6.4. gem5 restore checkpoint with a different CPU
      • 23.6.4.1. gem5 fast forward
    • 23.6.5. gem5 checkpoint upgrader
  • 23.7. Pass extra options to gem5
  • 23.8. m5ops
    • 23.8.1. gem5 m5 executable
      • 23.8.1.1. m5 exit
      • 23.8.1.2. m5 dumpstats
      • 23.8.1.3. m5 fail
      • 23.8.1.4. m5 writefile
      • 23.8.1.5. m5 readfile
      • 23.8.1.6. m5 initparam
      • 23.8.1.7. m5 execfile
    • 23.8.2. m5ops instructions
      • 23.8.2.1. m5ops magic addresses
      • 23.8.2.2. m5ops instructions interface
      • 23.8.2.3. m5op annotations
  • 23.9. gem5 arm Linux kernel patches
    • 23.9.1. gem5 arm Linux kernel patches boot speedup
  • 23.10. m5out directory
    • 23.10.1. gem5 m5out/system.terminal file
    • 23.10.2. gem5 m5out/system.workload.dmesg file
    • 23.10.3. gem5 m5out/stats.txt file
      • 23.10.3.1. gem5 HDF5 statistics
      • 23.10.3.2. gem5 only dump selected stats
      • 23.10.3.3. Meaning of each gem5 stat
      • 23.10.3.4. gem5 stats internals
    • 23.10.4. gem5 config.ini
      • 23.10.4.1. gem5 config.dot
  • 23.11. m5term
  • 23.12. gem5 Python scripts without rebuild
  • 23.13. gem5 fs_bigLITTLE
  • 23.14. gem5 in-tree tests
    • 23.14.1. gem5 unit tests
    • 23.14.2. gem5 regression tests
  • 23.15. gem5 simulate() limit reached
  • 23.16. gem5 build options
    • 23.16.1. gem5 debug build
    • 23.16.2. gem5 fast build
    • 23.16.3. gem5 prof and perf builds
    • 23.16.4. gem5 clang build
    • 23.16.5. gem5 sanitation build
    • 23.16.6. gem5 Ruby build
      • 23.16.6.1. gem5 Ruby MI_example protocol
      • 23.16.6.2. gem5 crossbar interconnect
    • 23.16.7. gem5 Python 3 build
  • 23.17. gem5 CPU types
    • 23.17.1. List of gem5 CPU types
      • 23.17.1.1. gem5 BaseSimpleCPU
        • 23.17.1.1.1. gem5 AtomicSimpleCPU
        • 23.17.1.1.2. gem5 TimingSimpleCPU
      • 23.17.1.2. gem5 MinorCPU
      • 23.17.1.3. gem5 DerivO3CPU
        • 23.17.1.3.1. gem5 DerivO3CPU pipeline stages
        • 23.17.1.3.2. gem5 util/o3-pipeview.py O3 pipeline viewer
        • 23.17.1.3.3. gem5 Konata O3 pipeline viewer
    • 23.17.2. gem5 ARM RSK
  • 23.18. gem5 ARM platforms
  • 23.19. gem5 upstream images
  • 23.20. gem5 bootloaders
  • 23.21. gem5 memory system
    • 23.21.1. gem5 port system
      • 23.21.1.1. gem5 functional vs atomic vs timing memory requests
        • 23.21.1.1.1. gem5 functional requests
    • 23.21.2. gem5 Packet vs Request
      • 23.21.2.1. gem5 Packet
        • 23.21.2.1.1. gem5 MemCmd
      • 23.21.2.2. gem5 Request
        • 23.21.2.2.1. gem5 Request in AtomicSimpleCPU
        • 23.21.2.2.2. gem5 Request in TimingSimpleCPU
    • 23.21.3. gem5 MSHR
    • 23.21.4. gem5 CommMonitor
    • 23.21.5. gem5 SimpleMemory
  • 23.22. gem5 internals
    • 23.22.1. gem5 Eclipse configuration
    • 23.22.2. gem5 Python C++ interaction
    • 23.22.3. gem5 entry point
      • 23.22.3.1. gem5 m5.objects module
    • 23.22.4. gem5 event queue
      • 23.22.4.1. gem5 event queue AtomicSimpleCPU syscall emulation freestanding example analysis
        • 23.22.4.1.1. AtomicSimpleCPU initial events

1. Getting started

Each child section describes a possible different setup for this repo.

If you don’t know which one to go for, start with QEMU Buildroot setup getting started.

Design goals of this project are documented at: [design-goals].

1.1. Should you waste your life with systems programming?

Being the hardcore person who fully understands an important complex system such as a computer, it does have a nice ring to it doesn’t it?

But before you dedicate your life to this nonsense, do consider the following points:

• almost all contributions to the kernel are done by large companies, and if you are not an employee in one of them, you are likely not going to be able to do much.

  This can be inferred by the fact that the devices/ directory is by far the largest in the kernel.

  The kernel is of course just an interface to hardware, and the hardware developers start developing their kernel stuff even before specs are publicly released, both to help with hardware development and to have things working when the announcement is made.

  Furthermore, I believe that there are in-tree devices which have never been properly publicly documented. Linus is of course fine with this, since code == documentation for him, but it is not as easy for mere mortals.

  There are some less hardware bound higher level layers in the kernel which might not require being in a hardware company, and a few people must be living off it.

  But of course, those are heavily motivated by the underlying hardware characteristics, and it is very likely that most of the people working there were previously at a hardware company.

  In that sense, therefore, the kernel is not as open as one might want to believe.

  Of course, if there is some super useful and undocumented hardware that is just waiting there to be reverse engineered, then that’s a much juicier target :-)

• it is impossible to become rich with this knowledge.

  This is partly implied by the fact that you need to be in a big company to make useful low level things, and therefore you will only be a tiny cog in the engine.

  The key problem is that the entry cost of hardware design is just too insanely high for startups in general.

• Is learning this the most useful thing that you think can do for society?

  Or are you just learning it for job security and having a nice sounding title?

  I’m not a huge fan of the person, but I think Jobs said it right: https://www.youtube.com/watch?v=FF-tKLISfPE

  First determine the useful goal, and then backtrack down to the most efficient thing you can do to reach it.

• there are two things that sadden me compared to physics-based engineering:

  • you will never become eternally famous. All tech disappears sooner or later, while laws of nature, at least as useful approximations, stay unchanged.

  • every problem that you face is caused by imperfections introduced by other humans.

    It is much easier to accept limitations of physics, and even natural selection in biology, which are not produced by a sentient being (?).

  Physics-based engineering, just like low level hardware, is of course completely closed source however, since wrestling against the laws of physics is about the most expensive thing humans can do, so there’s also a downside to it.

Are you fine with those points, and ready to continue wasting your life with this crap?

Good. In that case, read on, and let’s have some fun together ;-)

Related: [soft-topics].

1.2. QEMU Buildroot setup

1.2.1. QEMU Buildroot setup getting started

This setup has been mostly tested on Ubuntu. For other host operating systems see: [supported-hosts]. For greater stability, consider using the latest release instead of master: https://github.com/cirosantilli/linux-kernel-module-cheat/releases

Reserve 12Gb of disk and run:

git clone https://github.com/cirosantilli/linux-kernel-module-cheat
cd linux-kernel-module-cheat
./build --download-dependencies qemu-buildroot
./run

You don’t need to clone recursively even though we have .git submodules: download-dependencies fetches just the submodules that you need for this build to save time.

If something goes wrong, see: [common-build-issues] and use our issue tracker: https://github.com/cirosantilli/linux-kernel-module-cheat/issues

The initial build will take a while (30 minutes to 2 hours) to clone and build, see [benchmark-builds] for more details.

If you don’t want to wait, you could also try the following faster but much more limited methods:

• Prebuilt setup

• Host kernel module setup

but you will soon find that they are simply not enough if you anywhere near serious about systems programming.

After ./run, QEMU opens up leaving you in the /lkmc/ directory, and you can start playing with the kernel modules inside the simulated system:

insmod hello.ko
insmod hello2.ko
rmmod hello
rmmod hello2

This should print to the screen:

hello init
hello2 init
hello cleanup
hello2 cleanup

which are printk messages from init and cleanup methods of those modules.

Sources:

• kernel_modules/hello.c

• kernel_modules/hello2.c

Quit QEMU with:

Ctrl-A X

See also: Section 13.1.1, “Quit QEMU from text mode”.

All available modules can be found in the kernel_modules directory.

It is super easy to build for different CPU architectures, just use the --arch option:

./build --arch aarch64 --download-dependencies qemu-buildroot
./run --arch aarch64

To avoid typing --arch aarch64 many times, you can set the default arch as explained at: [default-command-line-arguments]

I now urge you to read the following sections which contain widely applicable information:

• [run-command-after-boot]

• [clean-the-build]

• Build the documentation

• Linux kernel

  • printk

  • Kernel command line parameters

Once you use GDB step debug and tmux, your terminal will look a bit like this:

[    1.451857] input: AT Translated Set 2 keyboard as /devices/platform/i8042/s1│loading @0xffffffffc0000000: ../kernel_modules-1.0//timer.ko
[    1.454310] ledtrig-cpu: registered to indicate activity on CPUs             │(gdb) b lkmc_timer_callback
[    1.455621] usbcore: registered new interface driver usbhid                  │Breakpoint 1 at 0xffffffffc0000000: file /home/ciro/bak/git/linux-kernel-module
[    1.455811] usbhid: USB HID core driver                                      │-cheat/out/x86_64/buildroot/build/kernel_modules-1.0/./timer.c, line 28.
[    1.462044] NET: Registered protocol family 10                               │(gdb) c
[    1.467911] Segment Routing with IPv6                                        │Continuing.
[    1.468407] sit: IPv6, IPv4 and MPLS over IPv4 tunneling driver              │
[    1.470859] NET: Registered protocol family 17                               │Breakpoint 1, lkmc_timer_callback (data=0xffffffffc0002000 <mytimer>)
[    1.472017] 9pnet: Installing 9P2000 support                                 │    at /linux-kernel-module-cheat//out/x86_64/buildroot/build/
[    1.475461] sched_clock: Marking stable (1473574872, 0)->(1554017593, -80442)│kernel_modules-1.0/./timer.c:28
[    1.479419] ALSA device list:                                                │28      {
[    1.479567]   No soundcards found.                                           │(gdb) c
[    1.619187] ata2.00: ATAPI: QEMU DVD-ROM, 2.5+, max UDMA/100                 │Continuing.
[    1.622954] ata2.00: configured for MWDMA2                                   │
[    1.644048] scsi 1:0:0:0: CD-ROM            QEMU     QEMU DVD-ROM     2.5+ P5│Breakpoint 1, lkmc_timer_callback (data=0xffffffffc0002000 <mytimer>)
[    1.741966] tsc: Refined TSC clocksource calibration: 2904.010 MHz           │    at /linux-kernel-module-cheat//out/x86_64/buildroot/build/
[    1.742796] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x29dc0f4s│kernel_modules-1.0/./timer.c:28
[    1.743648] clocksource: Switched to clocksource tsc                         │28      {
[    2.072945] input: ImExPS/2 Generic Explorer Mouse as /devices/platform/i8043│(gdb) bt
[    2.078641] EXT4-fs (vda): couldn't mount as ext3 due to feature incompatibis│#0  lkmc_timer_callback (data=0xffffffffc0002000 <mytimer>)
[    2.080350] EXT4-fs (vda): mounting ext2 file system using the ext4 subsystem│    at /linux-kernel-module-cheat//out/x86_64/buildroot/build/
[    2.088978] EXT4-fs (vda): mounted filesystem without journal. Opts: (null)  │kernel_modules-1.0/./timer.c:28
[    2.089872] VFS: Mounted root (ext2 filesystem) readonly on device 254:0.    │#1  0xffffffff810ab494 in call_timer_fn (timer=0xffffffffc0002000 <mytimer>,
[    2.097168] devtmpfs: mounted                                                │    fn=0xffffffffc0000000 <lkmc_timer_callback>) at kernel/time/timer.c:1326
[    2.126472] Freeing unused kernel memory: 1264K                              │#2  0xffffffff810ab71f in expire_timers (head=<optimized out>,
[    2.126706] Write protecting the kernel read-only data: 16384k               │    base=<optimized out>) at kernel/time/timer.c:1363
[    2.129388] Freeing unused kernel memory: 2024K                              │#3  __run_timers (base=<optimized out>) at kernel/time/timer.c:1666
[    2.139370] Freeing unused kernel memory: 1284K                              │#4  run_timer_softirq (h=<optimized out>) at kernel/time/timer.c:1692
[    2.246231] EXT4-fs (vda): warning: mounting unchecked fs, running e2fsck isd│#5  0xffffffff81a000cc in __do_softirq () at kernel/softirq.c:285
[    2.259574] EXT4-fs (vda): re-mounted. Opts: block_validity,barrier,user_xatr│#6  0xffffffff810577cc in invoke_softirq () at kernel/softirq.c:365
hello S98                                                                       │#7  irq_exit () at kernel/softirq.c:405
                                                                                │#8  0xffffffff818021ba in exiting_irq () at ./arch/x86/include/asm/apic.h:541
Apr 15 23:59:23 login[49]: root login on 'console'                              │#9  smp_apic_timer_interrupt (regs=<optimized out>)
hello /root/.profile                                                            │    at arch/x86/kernel/apic/apic.c:1052
# insmod /timer.ko                                                              │#10 0xffffffff8180190f in apic_timer_interrupt ()
[    6.791945] timer: loading out-of-tree module taints kernel.                 │    at arch/x86/entry/entry_64.S:857
# [    7.821621] 4294894248                                                     │#11 0xffffffff82003df8 in init_thread_union ()
[    8.851385] 4294894504                                                       │#12 0x0000000000000000 in ?? ()
                                                                                │(gdb)

1.2.2. How to hack stuff

Besides a seamless initial build, this project also aims to make it effortless to modify and rebuild several major components of the system, to serve as an awesome development setup.

1.2.2.1. Your first Linux kernel hack

Let’s hack up the Linux kernel entry point, which is an easy place to start.

Open the file:

vim submodules/linux/init/main.c

and find the start_kernel function, then add there a:

pr_info("I'VE HACKED THE LINUX KERNEL!!!");

Then rebuild the Linux kernel, quit QEMU and reboot the modified kernel:

./build-linux
./run

and, surely enough, your message has appeared at the beginning of the boot:

<6>[    0.000000] I'VE HACKED THE LINUX KERNEL!!!

So you are now officially a Linux kernel hacker, way to go!

We could have used just build to rebuild the kernel as in the initial build instead of build-linux, but building just the required individual components is preferred during development:

• saves a few seconds from parsing Make scripts and reading timestamps

• makes it easier to understand what is being done in more detail

• allows passing more specific options to customize the build

The build script is just a lightweight wrapper that calls the smaller build scripts, and you can see what ./build does with:

./build --dry-run

see also: Dry run to get commands for your project.

When you reach difficulties, QEMU makes it possible to easily GDB step debug the Linux kernel source code, see: Section 2, “GDB step debug”.

1.2.2.2. Your first kernel module hack

Edit kernel_modules/hello.c to contain:

pr_info("hello init hacked\n");

and rebuild with:

./build-modules

Now there are two ways to test it out: the fast way, and the safe way.

The fast way is, without quitting or rebooting QEMU, just directly re-insert the module with:

insmod /mnt/9p/out_rootfs_overlay/lkmc/hello.ko

and the new pr_info message should now show on the terminal at the end of the boot.

This works because we have a 9P mount there setup by default, which mounts the host directory that contains the build outputs on the guest:

ls "$(./getvar out_rootfs_overlay_dir)"

The fast method is slightly risky because your previously insmodded buggy kernel module attempt might have corrupted the kernel memory, which could affect future runs.

Such failures are however unlikely, and you should be fine if you don’t see anything weird happening.

The safe way, is to fist quit QEMU, rebuild the modules, put them in the root filesystem, and then reboot:

./build-modules
./build-buildroot
./run --eval-after 'insmod hello.ko'

./build-buildroot is required after ./build-modules because it re-generates the root filesystem with the modules that we compiled at ./build-modules.

You can see that ./build does that as well, by running:

./build --dry-run

See also: Dry run to get commands for your project.

--eval-after is optional: you could just type insmod hello.ko in the terminal, but this makes it run automatically at the end of boot, and then drops you into a shell.

If the guest and host are the same arch, typically x86_64, you can speed up boot further with KVM:

./run --kvm

All of this put together makes the safe procedure acceptably fast for regular development as well.

It is also easy to GDB step debug kernel modules with our setup, see: Section 2.4, “GDB step debug kernel module”.

1.2.2.3. Your first glibc hack

We use glibc as our default libc now, and it is tracked as an unmodified submodule at submodules/glibc, at the exact same version that Buildroot has it, which can be found at: package/glibc/glibc.mk. Buildroot 2018.05 applies no patches.

Let’s hack up the puts function:

./build-buildroot -- glibc-reconfigure

with the patch:

diff --git a/libio/ioputs.c b/libio/ioputs.c
index 706b20b492..23185948f3 100644
--- a/libio/ioputs.c
+++ b/libio/ioputs.c
@@ -38,8 +38,9 @@ _IO_puts (const char *str)
   if ((_IO_vtable_offset (_IO_stdout) != 0
        || _IO_fwide (_IO_stdout, -1) == -1)
       && _IO_sputn (_IO_stdout, str, len) == len
+      && _IO_sputn (_IO_stdout, " hacked", 7) == 7
       && _IO_putc_unlocked ('\n', _IO_stdout) != EOF)
-    result = MIN (INT_MAX, len + 1);
+    result = MIN (INT_MAX, len + 1 + 7);

   _IO_release_lock (_IO_stdout);
   return result;

And then:

./run --eval-after './c/hello.out'

outputs:

hello hacked

Lol!

We can also test our hacked glibc on User mode simulation with:

./run --userland userland/c/hello.c

I just noticed that this is actually a good way to develop glibc for other archs.

In this example, we got away without recompiling the userland program because we made a change that did not affect the glibc ABI, see this answer for an introduction to ABI stability: https://stackoverflow.com/questions/2171177/what-is-an-application-binary-interface-abi/54967743#54967743

Note that for arch agnostic features that don’t rely on bleeding kernel changes that you host doesn’t yet have, you can develop glibc natively as explained at:

• https://stackoverflow.com/questions/10412684/how-to-compile-my-own-glibc-c-standard-library-from-source-and-use-it/52454710#52454710

• https://stackoverflow.com/questions/847179/multiple-glibc-libraries-on-a-single-host/52454603#52454603

• https://stackoverflow.com/questions/2856438/how-can-i-link-to-a-specific-glibc-version/52550158#52550158 more focus on symbol versioning, but no one knows how to do it, so I answered

Tested on a30ed0f047523ff2368d421ee2cce0800682c44e + 1.

1.2.2.4. Your first Binutils hack

Have you ever felt that a single inc instruction was not enough? Really? Me too!

So let’s hack the [gnu-gas-assembler], which is part of GNU Binutils, to add a new shiny version of inc called…​ myinc!

GCC uses GNU GAS as its backend, so we will test out new mnemonic with an [gcc-inline-assembly] test program: userland/arch/x86_64/binutils_hack.c, which is just a copy of userland/arch/x86_64/binutils_nohack.c but with myinc instead of inc.

The inline assembly is disabled with an #ifdef, so first modify the source to enable that.

Then, try to build userland:

./build-userland

and watch it fail with:

binutils_hack.c:8: Error: no such instruction: `myinc %rax'

Now, edit the file

vim submodules/binutils-gdb/opcodes/i386-tbl.h

and add a copy of the "inc" instruction just next to it, but with the new name "myinc":

diff --git a/opcodes/i386-tbl.h b/opcodes/i386-tbl.h
index af583ce578..3cc341f303 100644
--- a/opcodes/i386-tbl.h
+++ b/opcodes/i386-tbl.h
@@ -1502,6 +1502,19 @@ const insn_template i386_optab[] =
     { { { 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
 	  0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0,
 	  1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0 } } } },
+  { "myinc", 1, 0xfe, 0x0, 1,
+    { { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+        0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0 } },
+    { 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+      0, 1, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 0,
+      0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+      0, 0, 0, 0, 0, 0 },
+    { { { 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
+	  0, 0, 1, 1, 1, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0,
+	  1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 0 } } } },
   { "sub", 2, 0x28, None, 1,
     { { 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
         0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,

Finally, rebuild Binutils, userland and test our program with User mode simulation:

./build-buildroot -- host-binutils-rebuild
./build-userland --static
./run --static --userland userland/arch/x86_64/binutils_hack.c

and we se that myinc worked since the assert did not fail!

Tested on b60784d59bee993bf0de5cde6c6380dd69420dda + 1.

1.2.2.5. Your first GCC hack

OK, now time to hack GCC.

For convenience, let’s use the User mode simulation.

If we run the program userland/c/gcc_hack.c:

./build-userland --static
./run --static --userland userland/c/gcc_hack.c

it produces the normal boring output:

i = 2
j = 0

So how about we swap ++ and -- to make things more fun?

Open the file:

vim submodules/gcc/gcc/c/c-parser.c

and find the function c_parser_postfix_expression_after_primary.

In that function, swap case CPP_PLUS_PLUS and case CPP_MINUS_MINUS:

diff --git a/gcc/c/c-parser.c b/gcc/c/c-parser.c
index 101afb8e35f..89535d1759a 100644
--- a/gcc/c/c-parser.c
+++ b/gcc/c/c-parser.c
@@ -8529,7 +8529,7 @@ c_parser_postfix_expression_after_primary (c_parser *parser,
 		expr.original_type = DECL_BIT_FIELD_TYPE (field);
 	    }
 	  break;
-	case CPP_PLUS_PLUS:
+	case CPP_MINUS_MINUS:
 	  /* Postincrement.  */
 	  start = expr.get_start ();
 	  finish = c_parser_peek_token (parser)->get_finish ();
@@ -8548,7 +8548,7 @@ c_parser_postfix_expression_after_primary (c_parser *parser,
 	  expr.original_code = ERROR_MARK;
 	  expr.original_type = NULL;
 	  break;
-	case CPP_MINUS_MINUS:
+	case CPP_PLUS_PLUS:
 	  /* Postdecrement.  */
 	  start = expr.get_start ();
 	  finish = c_parser_peek_token (parser)->get_finish ();

Now rebuild GCC, the program and re-run it:

./build-buildroot -- host-gcc-final-rebuild
./build-userland --static
./run --static --userland userland/c/gcc_hack.c

and the new ouptut is now:

i = 2
j = 0

We need to use the ugly -final thing because GCC has to packages in Buildroot, -initial and -final: https://stackoverflow.com/questions/54992977/how-to-select-an-override-srcdir-source-for-gcc-when-building-buildroot No one is able to example precisely with a minimal example why this is required:

• https://stackoverflow.com/questions/39883865/why-multiple-passes-for-building-linux-from-scratch-lfs

• https://stackoverflow.com/questions/27457835/why-do-cross-compilers-have-a-two-stage-compilation

1.2.3. About the QEMU Buildroot setup

What QEMU and Buildroot are:

• [introduction-to-buildroot]

• Introduction to QEMU

This is our reference setup, and the best supported one, use it unless you have good reason not to.

It was historically the first one we did, and all sections have been tested with this setup unless explicitly noted.

Read the following sections for further introductory material:

• Introduction to QEMU

• [introduction-to-buildroot]

1.3. Dry run to get commands for your project

One of the major features of this repository is that we try to support the --dry-run option really well for all scripts.

This option, as the name suggests, outputs the external commands that would be run (or more precisely: equivalent commands), without actually running them.

This allows you to just clone this repository and get full working commands to integrate into your project, without having to build or use this setup further!

For example, we can obtain a QEMU run for the file userland/c/hello.c in User mode simulation by adding --dry-run to the normal command:

./run --dry-run --userland userland/c/hello.c

which as of LKMC a18f28e263c91362519ef550150b5c9d75fa3679 + 1 outputs:

+ /path/to/linux-kernel-module-cheat/out/qemu/default/opt/x86_64-linux-user/qemu-x86_64 \
  -L /path/to/linux-kernel-module-cheat/out/buildroot/build/default/x86_64/target \
  -r 5.2.1 \
  -seed 0 \
  -trace enable=load_file,file=/path/to/linux-kernel-module-cheat/out/run/qemu/x86_64/0/trace.bin \
  -cpu max \
  /path/to/linux-kernel-module-cheat/out/userland/default/x86_64/c/hello.out \
;

So observe that the command contains:

• +: sign to differentiate it from program stdout, much like bash -x output. This is not a valid part of the generated Bash command however.

• the actual command nicely, indented and with arguments broken one per line, but with continuing backslashes so you can just copy paste into a terminal

  For setups that don’t support the newline e.g. Eclipse debugging, you can turn them off with --print-cmd-oneline

• ;: both a valid part of the Bash command, and a visual mark the end of the command

For the specific case of running emulators such as QEMU, the last command is also automatically placed in a file for your convenience and later inspection:

cat "$(./getvar run_dir)/run.sh"

Since we need this so often, the last run command is also stored for convenience at:

cat out/run.sh

although this won’t of course work well for [simultaneous-runs].

Furthermore, --dry-run also automatically specifies, in valid Bash shell syntax:

• environment variables used to run the command with syntax + ENV_VAR_1=abc ENV_VAR_2=def ./some/command

• change in working directory with + cd /some/new/path && ./some/command

1.4. gem5 Buildroot setup

1.4.1. About the gem5 Buildroot setup

This setup is like the QEMU Buildroot setup, but it uses gem5 instead of QEMU as a system simulator.

QEMU tries to run as fast as possible and give correct results at the end, but it does not tell us how many CPU cycles it takes to do something, just the number of instructions it ran. This kind of simulation is known as functional simulation.

The number of instructions executed is a very poor estimator of performance because in modern computers, a lot of time is spent waiting for memory requests rather than the instructions themselves.

gem5 on the other hand, can simulate the system in more detail than QEMU, including:

• simplified CPU pipeline

• caches

• DRAM timing

and can therefore be used to estimate system performance, see: Section 23.2, “gem5 run benchmark” for an example.

The downside of gem5 much slower than QEMU because of the greater simulation detail.

See gem5 vs QEMU for a more thorough comparison.

1.4.2. gem5 Buildroot setup getting started

For the most part, if you just add the --emulator gem5 option or *-gem5 suffix to all commands and everything should magically work.

If you haven’t built Buildroot yet for QEMU Buildroot setup, you can build from the beginning with:

./build --download-dependencies gem5-buildroot
./run --emulator gem5

If you have already built previously, don’t be afraid: gem5 and QEMU use almost the same root filesystem and kernel, so ./build will be fast.

Remember that the gem5 boot is considerably slower than QEMU since the simulation is more detailed.

If you have a relatively new GCC version and the gem5 build fails on your machine, see: [gem5-build-broken-on-recent-compiler-version].

To get a terminal, either open a new shell and run:

./gem5-shell

You can quit the shell without killing gem5 by typing tilde followed by a period:

~.

If you are inside tmux, which I highly recommend, you can both run gem5 stdout and open the guest terminal on a split window with:

./run --emulator gem5 --tmux

See also: Section 2.3.1, “tmux gem5”.

At the end of boot, it might not be very clear that you have the shell since some printk messages may appear in front of the prompt like this:

# <6>[    1.215329] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x1cd486fa865, max_idle_ns: 440795259574 ns
<6>[    1.215351] clocksource: Switched to clocksource tsc

but if you look closely, the PS1 prompt marker # is there already, just hit enter and a clear prompt line will appear.

If you forgot to open the shell and gem5 exit, you can inspect the terminal output post-mortem at:

less "$(./getvar --emulator gem5 m5out_dir)/system.pc.com_1.device"

More gem5 information is present at: Section 23, “gem5”

Good next steps are:

• gem5 run benchmark: how to run a benchmark in gem5 full system, including how to boot Linux, checkpoint and restore to skip the boot on a fast CPU

• m5out directory: understand the output files that gem5 produces, which contain information about your run

• m5ops: magic guest instructions used to control gem5

• [add-new-files-to-the-buildroot-image]: how to add your own files to the image if you have a benchmark that we don’t already support out of the box (also send a pull request!)

1.5. Docker host setup

This repository has been tested inside clean Docker containers.

This is a good option if you are on a Linux host, but the native setup failed due to your weird host distribution, and you have better things to do with your life than to debug it. See also: [supported-hosts].

For example, to do a QEMU Buildroot setup inside Docker, run:

sudo apt-get install docker
./run-docker create && \
./run-docker sh -- ./build --download-dependencies qemu-buildroot
./run-docker sh

You are now left inside a shell in the Docker! From there, just run as usual:

./run

The host git top level directory is mounted inside the guest with a Docker volume, which means for example that you can use your host’s GUI text editor directly on the files. Just don’t forget that if you nuke that directory on the guest, then it gets nuked on the host as well!

Command breakdown:

• ./run-docker create: create the image and container.

  Needed only the very first time you use Docker, or if you run ./run-docker DESTROY to restart for scratch, or save some disk space.

  The image and container name is lkmc. The container shows under:

  docker ps -a

  and the image shows under:

  docker images

• ./run-docker sh: open a shell on the container.

  If it has not been started previously, start it. This can also be done explicitly with:

  ./run-docker start

  Quit the shell as usual with Ctrl-D

  This can be called multiple times from different host terminals to open multiple shells.

• ./run-docker stop: stop the container.

  This might save a bit of CPU and RAM once you stop working on this project, but it should not be a lot.

• ./run-docker DESTROY: delete the container and image.

  This doesn’t really clean the build, since we mount the guest’s working directory on the host git top-level, so you basically just got rid of the apt-get installs.

  To actually delete the Docker build, run on host:

  # sudo rm -rf out.docker

To use GDB step debug from inside Docker, you need a second shell inside the container. You can either do that from another shell with:

./run-docker sh

or even better, by starting a tmux session inside the container. We install tmux by default in the container.

You can also start a second shell and run a command in it at the same time with:

./run-docker sh -- ./run-gdb start_kernel

To use QEMU graphic mode from Docker, run:

./run --graphic --vnc

and then on host:

sudo apt-get install vinagre
./vnc

TODO make files created inside Docker be owned by the current user in host instead of root:

• https://stackoverflow.com/questions/33681396/how-do-i-write-to-a-volume-container-as-non-root-in-docker

• https://stackoverflow.com/questions/23544282/what-is-the-best-way-to-manage-permissions-for-docker-shared-volumes

• https://stackoverflow.com/questions/31779802/shared-volume-file-permissions-ownership-docker

1.6. Prebuilt setup

1.6.1. About the prebuilt setup

This setup uses prebuilt binaries that we upload to GitHub from time to time.

We don’t currently provide a full prebuilt because it would be too big to host freely, notably because of the cross toolchain.

Our prebuilts currently include:

• QEMU Buildroot setup binaries

  • Linux kernel

  • root filesystem

• Baremetal setup binaries for QEMU

For more details, see our our release procedure.

Advantage of this setup: saves time and disk space on the initial install, which is expensive in largely due to building the toolchain.

The limitations are severe however:

• can’t GDB step debug the kernel, since the source and cross toolchain with GDB are not available. Buildroot cannot easily use a host toolchain: [prebuilt-toolchain].

  Maybe we could work around this by just downloading the kernel source somehow, and using a host prebuilt GDB, but we felt that it would be too messy and unreliable.

• you won’t get the latest version of this repository. Our [travis] attempt to automate builds failed, and storing a release for every commit would likely make GitHub mad at us anyway.

• gem5 is not currently supported. The major blocking point is how to avoid distributing the kernel images twice: once for gem5 which uses vmlinux, and once for QEMU which uses arch/* images, see also:

  • https://github.com/cirosantilli/linux-kernel-module-cheat/issues/79

  • vmlinux vs bzImage vs zImage vs Image.

This setup might be good enough for those developing simulators, as that requires less image modification. But once again, if you are serious about this, why not just let your computer build the full featured setup while you take a coffee or a nap? :-)

1.6.2. Prebuilt setup getting started

Checkout to the latest tag and use the Ubuntu packaged QEMU to boot Linux:

sudo apt-get install qemu-system-x86
git clone https://github.com/cirosantilli/linux-kernel-module-cheat
cd linux-kernel-module-cheat
git checkout "$(git rev-list --tags --max-count=1)"
./release-download-latest
unzip lkmc-*.zip
./run --qemu-which host

You have to checkout to the latest tag to ensure that the scripts match the release format: https://stackoverflow.com/questions/1404796/how-to-get-the-latest-tag-name-in-current-branch-in-git

This is known not to work for aarch64 on an Ubuntu 16.04 host with QEMU 2.5.0, presumably because QEMU is too old, the terminal does not show any output. I haven’t investigated why.

Or to run a baremetal example instead:

./run \
  --arch aarch64 \
  --baremetal userland/c/hello.c \
  --qemu-which host \
;

Be saner and use our custom built QEMU instead:

./build --download-dependencies qemu
./run

To build the kernel modules as in Your first kernel module hack do:

git submodule update --depth 1 --init --recursive "$(./getvar linux_source_dir)"
./build-linux --no-modules-install -- modules_prepare
./build-modules --gcc-which host
./run

TODO: for now the only way to test those modules out without building Buildroot is with 9p, since we currently rely on Buildroot to manipulate the root filesystem.

Command explanation:

• modules_prepare does the minimal build procedure required on the kernel for us to be able to compile the kernel modules, and is way faster than doing a full kernel build. A full kernel build would also work however.

• --gcc-which host selects your host Ubuntu packaged GCC, since you don’t have the Buildroot toolchain

• --no-modules-install is required otherwise the make modules_install target we run by default fails, since the kernel wasn’t built

To modify the Linux kernel, build and use it as usual:

git submodule update --depth 1 --init --recursive "$(./getvar linux_source_dir)"
./build-linux
./run

1.7. Host kernel module setup

THIS IS DANGEROUS (AND FUN), YOU HAVE BEEN WARNED

This method runs the kernel modules directly on your host computer without a VM, and saves you the compilation time and disk usage of the virtual machine method.

It has however severe limitations:

• can’t control which kernel version and build options to use. So some of the modules will likely not compile because of kernel API changes, since the Linux kernel does not have a stable kernel module API.

• bugs can easily break you system. E.g.:

  • segfaults can trivially lead to a kernel crash, and require a reboot

  • your disk could get erased. Yes, this can also happen with sudo from userland. But you should not use sudo when developing newbie programs. And for the kernel you don’t have the choice not to use sudo.

  • even more subtle system corruption such as not being able to rmmod

• can’t control which hardware is used, notably the CPU architecture

• can’t step debug it with GDB easily. The alternatives are JTAG or KGDB, but those are less reliable, and require extra hardware.

Still interested?

./build-modules --host

Compilation will likely fail for some modules because of kernel or toolchain differences that we can’t control on the host.

The best workaround is to compile just your modules with:

./build-modules --host -- hello hello2

which is equivalent to:

./build-modules \
  --gcc-which host \
  --host \
  -- \
  kernel_modules/hello.c \
  kernel_modules/hello2.c \
;

Or just remove the .c extension from the failing files and try again:

cd "$(./getvar kernel_modules_source_dir)"
mv broken.c broken.c~

Once you manage to compile, and have come to terms with the fact that this may blow up your host, try it out with:

cd "$(./getvar kernel_modules_build_host_subdir)"
sudo insmod hello.ko

# Our module is there.
sudo lsmod | grep hello

# Last message should be: hello init
dmesg -T

sudo rmmod hello

# Last message should be: hello exit
dmesg -T

# Not present anymore
sudo lsmod | grep hello

1.7.1. Hello host

Minimal host build system example:

cd hello_host_kernel_module
make
sudo insmod hello.ko
dmesg
sudo rmmod hello.ko
dmesg

1.8. Userland setup

1.8.1. About the userland setup

In order to test the kernel and emulators, userland content in the form of executables and scripts is of course required, and we store it mostly under:

• userland/

• [rootfs-overlay]

• [add-new-buildroot-packages]

When we started this repository, it only contained content that interacted very closely with the kernel, or that had required performance analysis.

However, we soon started to notice that this had an increasing overlap with other userland test repositories: we were duplicating build and test infrastructure and even some examples.

Therefore, we decided to consolidate other userland tutorials that we had scattered around into this repository.

Notable userland content included / moving into this repository includes:

• [userland-assembly]

• [c]

• [cpp]

• [posix]

• [algorithms]

1.8.2. Userland setup getting started

There are several ways to run our [userland-content], notably:

• natively on the host as shown at: Section 1.8.2.1, “Userland setup getting started natively”

  Can only run examples compatible with your host CPU architecture and OS, but has the fastest setup and runtimes.

• from user mode simulation with:

  • the host prebuilt toolchain: Section 1.8.2.2, “Userland setup getting started with prebuilt toolchain and QEMU user mode”

  • the Buildroot toolchain you built yourself: Section 10.1, “QEMU user mode getting started”

  This setup:

  • can run most examples, including those for other CPU architectures, with the notable exception of examples that rely on kernel modules

  • can run reproducible approximate performance experiments with gem5, see e.g. [bst-vs-heap-vs-hashmap]

• from full system simulation as shown at: Section 1.2.1, “QEMU Buildroot setup getting started”.

  This is the most reproducible and controlled environment, and all examples work there. But also the slower one to setup.

1.8.2.1. Userland setup getting started natively

With this setup, we will use the host toolchain and execute executables directly on the host.

No toolchain build is required, so you can just download your distro toolchain and jump straight into it.

Build, run and example, and clean it in-tree with:

sudo apt-get install gcc
cd userland
./build c/hello
./c/hello.out
./build --clean

Source: userland/c/hello.c.

Build an entire directory and test it:

cd userland
./build c
./test c

Build the current directory and test it:

cd userland/c
./build
./test

As mentioned at [userland-libs-directory], tests under userland/libs require certain optional libraries to be installed, and are not built or tested by default.

You can install those libraries with:

cd linux-kernel-module-cheat
./build --download-dependencies userland-host

and then build the examples and test with:

./build --package-all
./test --package-all

Pass custom compiler options:

./build --ccflags='-foptimize-sibling-calls -foptimize-strlen' --force-rebuild

Here we used --force-rebuild to force rebuild since the sources weren’t modified since the last build.

Some CLI options have more specialized flags, e.g. -O for the [optimization-level-of-a-build]:

./build --optimization-level 3 --force-rebuild

See also User mode static executables for --static.

The build scripts inside userland/ are just symlinks to build-userland-in-tree which you can also use from toplevel as:

./build-userland-in-tree
./build-userland-in-tree userland/c
./build-userland-in-tree userland/c/hello.c

build-userland-in-tree is in turn just a thin wrapper around build-userland:

./build-userland --gcc-which host --in-tree userland/c

So you can use any option supported by build-userland script freely with build-userland-in-tree and build.

The situation is analogous for userland/test, test-executables-in-tree and test-executables, which are further documented at: Section 10.2, “User mode tests”.

Do a more clean out-of-tree build instead and run the program:

./build-userland --gcc-which host --userland-build-id host
./run --emulator native --userland userland/c/hello.c --userland-build-id host

Here we:

• put the host executables in a separate build variant to avoid conflict with Buildroot builds.

• ran with the --emulator native option to run the program natively

In this case you can debub the program with:

./run --debug-vm --emulator native --userland userland/c/hello.c --userland-build-id host

as shown at: Section 22.8, “Debug the emulator”, although direct GDB host usage works as well of course.

1.8.2.2. Userland setup getting started with prebuilt toolchain and QEMU user mode

If you are lazy to built the Buildroot toolchain and QEMU, but want to run e.g. ARM [userland-assembly] in User mode simulation, you can get away on Ubuntu 18.04 with just:

sudo apt-get install gcc-aarch64-linux-gnu qemu-system-aarch64
./build-userland \
  --arch aarch64 \
  --gcc-which host \
  --userland-build-id host \
;
./run \
  --arch aarch64 \
  --qemu-which host \
  --userland-build-id host \
  --userland userland/c/command_line_arguments.c \
  --cli-args 'asdf "qw er"' \
;

where:

• --gcc-which host: use the host toolchain.

  We must pass this to ./run as well because QEMU must know which dynamic libraries to use. See also: Section 10.5, “User mode static executables”.

• --userland-build-id host: put the host built into a [build-variants]

This present the usual trade-offs of using prebuilts as mentioned at: Section 1.6, “Prebuilt setup”.

Other functionality are analogous, e.g. testing:

./test-executables \
  --arch aarch64 \
  --gcc-which host \
  --qemu-which host \
  --userland-build-id host \
;

and User mode GDB:

./run \
  --arch aarch64 \
  --gdb \
  --gcc-which host \
  --qemu-which host \
  --userland-build-id host \
  --userland userland/c/command_line_arguments.c \
  --cli-args 'asdf "qw er"' \
;

1.8.2.3. Userland setup getting started full system

First ensure that QEMU Buildroot setup is working.

After doing that setup, you can already execute your userland programs from inside QEMU: the only missing step is how to rebuild executables and run them.

And the answer is exactly analogous to what is shown at: Section 1.2.2.2, “Your first kernel module hack”

For example, if we modify userland/c/hello.c to print out something different, we can just rebuild it with:

./build-userland

Source: build-userland. ./build calls that script automatically for us when doing the initial full build.

Now, run the program either without rebooting use the 9P mount:

/mnt/9p/out_rootfs_overlay/c/hello.out

or shutdown QEMU, add the executable to the root filesystem:

./build-buildroot

reboot and use the root filesystem as usual:

./hello.out

1.9. Baremetal setup

1.9.1. About the baremetal setup

This setup does not use the Linux kernel nor Buildroot at all: it just runs your very own minimal OS.

x86_64 is not currently supported, only arm and aarch64: I had made some x86 bare metal examples at: https://github.com/cirosantilli/x86-bare-metal-examples but I’m lazy to port them here now. Pull requests are welcome.

The main reason this setup is included in this project, despite the word "Linux" being on the project name, is that a lot of the emulator boilerplate can be reused for both use cases.

This setup allows you to make a tiny OS and that runs just a few instructions, use it to fully control the CPU to better understand the simulators for example, or develop your own OS if you are into that.

You can also use C and a subset of the C standard library because we enable Newlib by default. See also:

• https://electronics.stackexchange.com/questions/223929/c-standard-libraries-on-bare-metal/400077#400077

• https://stackoverflow.com/questions/13063055/does-a-libc-os-exist/59771531#59771531

Our C bare-metal compiler is built with crosstool-NG. If you have already built Buildroot previously, you will end up with two GCCs installed. Unfortunately I don’t see a solution for this, since we need separate toolchains for Newlib on baremetal and glibc on Linux: https://stackoverflow.com/questions/38956680/difference-between-arm-none-eabi-and-arm-linux-gnueabi/38989869#38989869

1.9.2. Baremetal setup getting started

Every .c file inside baremetal/ and .S file inside baremetal/arch/<arch>/ generates a separate baremetal image.

For example, to run baremetal/arch/aarch64/dump_regs.c in QEMU do:

./build --arch aarch64 --download-dependencies qemu-baremetal
./run --arch aarch64 --baremetal baremetal/arch/aarch64/dump_regs.c

And the terminal prints the values of certain system registers. This example prints registers that are only accessible from EL1 or higher, and thus could not be run in userland.

In addition to the examples under baremetal/, several of the userland examples can also be run in baremetal! This is largely due to the awesomeness of Newlib.

The examples that work include most C examples that don’t rely on complicated syscalls such as threads, and almost all the [userland-assembly] examples.

The exact list of userland programs that work in baremetal is specified in [path-properties] with the baremetal property, but you can also easily find it out with a baremetal test dry run:

./test-executables --arch aarch64 --dry-run --mode baremetal

For example, we can run the C hello world userland/c/hello.c simply as:

./run --arch aarch64 --baremetal userland/c/hello.c

and that outputs to the serial port the string:

hello

which QEMU shows on the host terminal.

To modify a baremetal program, simply edit the file, e.g.

vim userland/c/hello.c

and rebuild:

./build-baremetal --arch aarch64
./run --arch aarch64 --baremetal userland/c/hello.c

./build qemu-baremetal that we run previously is only needed for the initial build. That script calls build-baremetal for us, in addition to building prerequisites such as QEMU and crosstool-NG.

./build-baremetal uses crosstool-NG, and so it must be preceded by build-crosstool-ng, which ./build qemu-baremetal also calls.

Now let’s run userland/arch/aarch64/add.S:

./run --arch aarch64 --baremetal userland/arch/aarch64/add.S

This time, the terminal does not print anything, which indicates success: if you look into the source, you will see that we just have an assertion there.

You can see a sample assertion fail in userland/c/assert_fail.c:

./run --arch aarch64 --baremetal userland/c/assert_fail.c

and the terminal contains:

lkmc_exit_status_134
error: simulation error detected by parsing logs

and the exit status of our script is 1:

echo $?

You can run all the baremetal examples in one go and check that all assertions passed with:

./test-executables --arch aarch64 --mode baremetal

To use gem5 instead of QEMU do:

./build --download-dependencies gem5-baremetal
./run --arch aarch64 --baremetal userland/c/hello.c --emulator gem5

and then as usual open a shell with:

./gem5-shell

Or as usual, tmux users can do both in one go with:

./run --arch aarch64 --baremetal userland/c/hello.c --emulator gem5 --tmux

TODO: the carriage returns are a bit different than in QEMU, see: [gem5-baremetal-carriage-return].

Note that ./build-baremetal requires the --emulator gem5 option, and generates separate executable images for both, as can be seen from:

echo "$(./getvar --arch aarch64 --baremetal userland/c/hello.c --emulator qemu image)"
echo "$(./getvar --arch aarch64 --baremetal userland/c/hello.c --emulator gem5 image)"

This is unlike the Linux kernel that has a single image for both QEMU and gem5:

echo "$(./getvar --arch aarch64 --emulator qemu image)"
echo "$(./getvar --arch aarch64 --emulator gem5 image)"

The reason for that is that on baremetal we don’t parse the device tress from memory like the Linux kernel does, which tells the kernel for example the UART address, and many other system parameters.

gem5 also supports the RealViewPBX machine, which represents an older hardware compared to the default VExpress_GEM5_V1:

./build-baremetal --arch aarch64 --emulator gem5 --machine RealViewPBX
./run --arch aarch64 --baremetal userland/c/hello.c --emulator gem5 --machine RealViewPBX

see also: Section 23.18, “gem5 ARM platforms”.

This generates yet new separate images with new magic constants:

echo "$(./getvar --arch aarch64 --baremetal userland/c/hello.c --emulator gem5 --machine VExpress_GEM5_V1 image)"
echo "$(./getvar --arch aarch64 --baremetal userland/c/hello.c --emulator gem5 --machine RealViewPBX      image)"

But just stick to newer and better VExpress_GEM5_V1 unless you have a good reason to use RealViewPBX.

When doing baremetal programming, it is likely that you will want to learn userland assembly first, see: [userland-assembly].

For more information on baremetal, see the section: [baremetal].

The following subjects are particularly important:

• Tracing

• [baremetal-gdb-step-debug]

1.10. Build the documentation

You don’t need to depend on GitHub.

For a quick and dirty build, install Asciidoctor however you like and build:

asciidoctor README.adoc
xdg-open README.html

For development, you will want to do a more controlled build with extra error checking as follows.

For the initial build do:

./build --download-dependencies docs

which also downloads build dependencies.

Then the following times just to the faster:

./build-doc

Source: build-doc

The HTML output is located at:

xdg-open out/README.html

More information about our documentation internals can be found at: [documentation]

2. GDB step debug

2.1. GDB step debug kernel boot

--gdb-wait makes QEMU and gem5 wait for a GDB connection, otherwise we could accidentally go past the point we want to break at:

./run --gdb-wait

Say you want to break at start_kernel. So on another shell:

./run-gdb start_kernel

or at a given line:

./run-gdb init/main.c:1088

Now QEMU will stop there, and you can use the normal GDB commands:

list
next
continue

See also:

• https://stackoverflow.com/questions/11408041/how-to-debug-the-linux-kernel-with-gdb-and-qemu/33203642#33203642

• https://stackoverflow.com/questions/4943857/linux-kernel-live-debugging-how-its-done-and-what-tools-are-used/42316607#42316607

2.1.1. GDB step debug kernel boot other archs

Just don’t forget to pass --arch to ./run-gdb, e.g.:

./run --arch aarch64 --gdb-wait

and:

./run-gdb --arch aarch64 start_kernel

2.1.2. Disable kernel compiler optimizations

https://stackoverflow.com/questions/29151235/how-to-de-optimize-the-linux-kernel-to-and-compile-it-with-o0

O=0 is an impossible dream, O=2 being the default.

So get ready for some weird jumps, and <value optimized out> fun. Why, Linux, why.

The -O level of some other userland content can be controlled as explained at: [optimization-level-of-a-build].

2.2. GDB step debug kernel post-boot

Let’s observe the kernel write system call as it reacts to some userland actions.

Start QEMU with just:

./run

and after boot inside a shell run:

./count.sh

which counts to infinity to stdout. Source: rootfs_overlay/lkmc/count.sh.

Then in another shell, run:

./run-gdb

and then hit:

Ctrl-C
break __x64_sys_write
continue
continue
continue

And you now control the counting on the first shell from GDB!

Before v4.17, the symbol name was just sys_write, the change happened at d5a00528b58cdb2c71206e18bd021e34c4eab878. As of Linux v 4.19, the function is called sys_write in arm, and __arm64_sys_write in aarch64. One good way to find it if the name changes again is to try:

rbreak .*sys_write

or just have a quick look at the sources!

When you hit Ctrl-C, if we happen to be inside kernel code at that point, which is very likely if there are no heavy background tasks waiting, and we are just waiting on a sleep type system call of the command prompt, we can already see the source for the random place inside the kernel where we stopped.

2.3. tmux

tmux just makes things even more fun by allowing us to see both the terminal for:

• emulator stdout

• GDB step debug

at once without dragging windows around!

First start tmux with:

tmux

Now that you are inside a shell inside tmux, you can start GDB simply with:

./run --gdb

which is just a convenient shortcut for:

./run --gdb-wait --tmux --tmux-args start_kernel

This splits the terminal into two panes:

• left: usual QEMU with terminal

• right: GDB

and focuses on the GDB pane.

Now you can navigate with the usual tmux shortcuts:

• switch between the two panes with: Ctrl-B O

• close either pane by killing its terminal with Ctrl-D as usual

See the tmux manual for further details:

man tmux

To start again, switch back to the QEMU pane with Ctrl-O, kill the emulator, and re-run:

./run --gdb

This automatically clears the GDB pane, and starts a new one.

The option --tmux-args determines which options will be passed to the program running on the second tmux pane, and is equivalent to:

This is equivalent to:

./run --gdb-wait
./run-gdb start_kernel

Due to Python’s CLI parsing quicks, if the run-gdb arguments start with a dash -, you have to use the = sign, e.g. to GDB step debug early boot:

./run --gdb --tmux-args=--no-continue

Bibliography: https://unix.stackexchange.com/questions/152738/how-to-split-a-new-window-and-run-a-command-in-this-new-window-using-tmux/432111#432111

2.3.1. tmux gem5

If you are using gem5 instead of QEMU, --tmux has a different effect by default: it opens the gem5 terminal instead of the debugger:

./run --emulator gem5 --tmux

To open a new pane with GDB instead of the terminal, use:

./run --gdb

which is equivalent to:

./run --emulator gem5 --gdb-wait --tmux --tmux-args start_kernel --tmux-program gdb

--tmux-program implies --tmux, so we can just write:

./run --emulator gem5 --gdb-wait --tmux-program gdb

If you also want to see both GDB and the terminal with gem5, then you will need to open a separate shell manually as usual with ./gem5-shell.

From inside tmux, you can create new terminals on a new window with Ctrl-B C split a pane yet again vertically with Ctrl-B % or horizontally with Ctrl-B ".

2.4. GDB step debug kernel module

https://stackoverflow.com/questions/28607538/how-to-debug-linux-kernel-modules-with-qemu/44095831#44095831

Loadable kernel modules are a bit trickier since the kernel can place them at different memory locations depending on load order.

So we cannot set the breakpoints before insmod.

However, the Linux kernel GDB scripts offer the lx-symbols command, which takes care of that beautifully for us.

Shell 1:

./run

Wait for the boot to end and run:

insmod timer.ko

Source: kernel_modules/timer.c.

This prints a message to dmesg every second.

Shell 2:

./run-gdb

In GDB, hit Ctrl-C, and note how it says:

scanning for modules in /root/linux-kernel-module-cheat/out/kernel_modules/x86_64/kernel_modules
loading @0xffffffffc0000000: /root/linux-kernel-module-cheat/out/kernel_modules/x86_64/kernel_modules/timer.ko

That’s lx-symbols working! Now simply:

break lkmc_timer_callback
continue
continue
continue

and we now control the callback from GDB!

Just don’t forget to remove your breakpoints after rmmod, or they will point to stale memory locations.

TODO: why does break work_func for insmod kthread.ko not very well? Sometimes it breaks but not others.

2.4.1. GDB step debug kernel module insmodded by init on ARM

TODO on arm 51e31cdc2933a774c2a0dc62664ad8acec1d2dbe it does not always work, and lx-symbols fails with the message:

loading vmlinux
Traceback (most recent call last):
  File "/linux-kernel-module-cheat//out/arm/buildroot/build/linux-custom/scripts/gdb/linux/symbols.py", line 163, in invoke
    self.load_all_symbols()
  File "/linux-kernel-module-cheat//out/arm/buildroot/build/linux-custom/scripts/gdb/linux/symbols.py", line 150, in load_all_symbols
    [self.load_module_symbols(module) for module in module_list]
  File "/linux-kernel-module-cheat//out/arm/buildroot/build/linux-custom/scripts/gdb/linux/symbols.py", line 110, in load_module_symbols
    module_name = module['name'].string()
gdb.MemoryError: Cannot access memory at address 0xbf0000cc
Error occurred in Python command: Cannot access memory at address 0xbf0000cc

Can’t reproduce on x86_64 and aarch64 are fine.

It is kind of random: if you just insmod manually and then immediately ./run-gdb --arch arm, then it usually works.

But this fails most of the time: shell 1:

./run --arch arm --eval-after 'insmod hello.ko'

shell 2:

./run-gdb --arch arm

then hit Ctrl-C on shell 2, and voila.

Then:

cat /proc/modules

says that the load address is:

0xbf000000

so it is close to the failing 0xbf0000cc.

readelf:

./run-toolchain readelf -- -s "$(./getvar kernel_modules_build_subdir)/hello.ko"

does not give any interesting hits at cc, no symbol was placed that far.

2.4.2. GDB module_init

TODO find a more convenient method. We have working methods, but they are not ideal.

This is not very easy, since by the time the module finishes loading, and lx-symbols can work properly, module_init has already finished running!

Possibly asked at:

• https://stackoverflow.com/questions/37059320/debug-a-kernel-module-being-loaded

• https://stackoverflow.com/questions/11888412/debug-the-init-module-call-of-a-linux-kernel-module

2.4.2.1. GDB module_init step into it

This is the best method we’ve found so far.

The kernel calls module_init synchronously, therefore it is not hard to step into that call.

As of 4.16, the call happens in do_one_initcall, so we can do in shell 1:

./run

shell 2 after boot finishes (because there are other calls to do_init_module at boot, presumably for the built-in modules):

./run-gdb do_one_initcall

then step until the line:

833         ret = fn();

which does the actual call, and then step into it.

For the next time, you can also put a breakpoint there directly:

./run-gdb init/main.c:833

How we found this out: first we got GDB module_init calculate entry address working, and then we did a bt. AKA cheating :-)

2.4.2.2. GDB module_init calculate entry address

This works, but is a bit annoying.

The key observation is that the load address of kernel modules is deterministic: there is a pre allocated memory region https://www.kernel.org/doc/Documentation/x86/x86_64/mm.txt "module mapping space" filled from bottom up.

So once we find the address the first time, we can just reuse it afterwards, as long as we don’t modify the module.

Do a fresh boot and get the module:

./run --eval-after './pr_debug.sh;insmod fops.ko;./linux/poweroff.out'

The boot must be fresh, because the load address changes every time we insert, even after removing previous modules.

The base address shows on terminal:

0xffffffffc0000000 .text

Now let’s find the offset of myinit:

./run-toolchain readelf -- \
  -s "$(./getvar kernel_modules_build_subdir)/fops.ko" | \
  grep myinit

which gives:

    30: 0000000000000240    43 FUNC    LOCAL  DEFAULT    2 myinit

so the offset address is 0x240 and we deduce that the function will be placed at:

0xffffffffc0000000 + 0x240 = 0xffffffffc0000240

Now we can just do a fresh boot on shell 1:

./run --eval 'insmod fops.ko;./linux/poweroff.out' --gdb-wait

and on shell 2:

./run-gdb '*0xffffffffc0000240'

GDB then breaks, and lx-symbols works.

2.4.2.3. GDB module_init break at the end of sys_init_module

TODO not working. This could be potentially very convenient.

The idea here is to break at a point late enough inside sys_init_module, at which point lx-symbols can be called and do its magic.

Beware that there are both sys_init_module and sys_finit_module syscalls, and insmod uses fmodule_init by default.

Both call do_module_init however, which is what lx-symbols hooks to.

If we try:

b sys_finit_module

then hitting:

n

does not break, and insertion happens, likely because of optimizations? Disable kernel compiler optimizations

Then we try:

b do_init_module

A naive:

fin

also fails to break!

Finally, in despair we notice that pr_debug prints the kernel load address as explained at Bypass lx-symbols.

So, if we set a breakpoint just after that message is printed by searching where that happens on the Linux source code, we must be able to get the correct load address before init_module happens.

2.4.2.4. GDB module_init add trap instruction

This is another possibility: we could modify the module source by adding a trap instruction of some kind.

This appears to be described at: https://www.linuxjournal.com/article/4525

But it refers to a gdbstart script which is not in the tree anymore and beyond my git log capabilities.

And just adding:

asm( " int $3");

directly gives an oops as I’d expect.

2.4.3. Bypass lx-symbols

Useless, but a good way to show how hardcore you are. Disable lx-symbols with:

./run-gdb --no-lxsymbols

From inside guest:

insmod timer.ko
cat /proc/modules

as mentioned at:

• https://stackoverflow.com/questions/6384605/how-to-get-address-of-a-kernel-module-loaded-using-insmod/6385818

• https://unix.stackexchange.com/questions/194405/get-base-address-and-size-of-a-loaded-kernel-module

This will give a line of form:

fops 2327 0 - Live 0xfffffffa00000000

And then tell GDB where the module was loaded with:

Ctrl-C
add-symbol-file ../../../rootfs_overlay/x86_64/timer.ko 0xffffffffc0000000
0xffffffffc0000000

Alternatively, if the module panics before you can read /proc/modules, there is a pr_debug which shows the load address:

echo 8 > /proc/sys/kernel/printk
echo 'file kernel/module.c +p' > /sys/kernel/debug/dynamic_debug/control
./linux/myinsmod.out hello.ko

And then search for a line of type:

[   84.877482]  0xfffffffa00000000 .text

Tested on 4f4749148273c282e80b58c59db1b47049e190bf + 1.

2.5. GDB step debug early boot

TODO successfully debug the very first instruction that the Linux kernel runs, before start_kernel!

Break at the very first instruction executed by QEMU:

./run-gdb --no-continue

Note however that early boot parts appear to be relocated in memory somehow, and therefore:

• you won’t see the source location in GDB, only assembly

• you won’t be able to break by symbol in those early locations

Further discussion at: Linux kernel entry point.

In the specific case of gem5 aarch64 at least:

• gem5 relocates the kernel in memory to a fixed location, see e.g. https://gem5.atlassian.net/browse/GEM5-787

• --param 'system.workload.early_kernel_symbols=True should in theory duplicate the symbols to the correct physical location, but it was broken at one point: https://gem5.atlassian.net/browse/GEM5-785

• gem5 executes directly from vmlinux, so there is no decompression code involved, so you actually immediately start running the "true" first instruction from head.S as described at: https://stackoverflow.com/questions/18266063/does-linux-kernel-have-main-function/33422401#33422401

• once the MMU gets turned on at kernel symbol __primary_switched, the virtual address matches the ELF symbols, and you start seeing correct symbols without the need for early_kernel_symbols. This can be observed clearly with function_trace = True: https://stackoverflow.com/questions/64049487/how-to-trace-executed-guest-function-symbol-names-with-their-timestamp-in-gem5/64049488#64049488 which produces:

  0: _kernel_flags_le_lo32 (12500)
  12500: __crc_tcp_add_backlog (1000)
  13500: __crc_crypto_alg_tested (6500)
  20000: __crc_tcp_add_backlog (10000)
  30000: __crc_crypto_alg_tested (500)
  30500: __crc_scsi_is_host_device (5000)
  35500: __crc_crypto_alg_tested (1500)
  37000: __crc_scsi_is_host_device (4000)
  41000: __crc_crypto_alg_tested (3000)
  44000: __crc_tcp_add_backlog (263500)
  307500: __crc_crypto_alg_tested (975500)
  1283000: __crc_tcp_add_backlog (77191500)
  78474500: __crc_crypto_alg_tested (1000)
  78475500: __crc_scsi_is_host_device (19500)
  78495000: __crc_crypto_alg_tested (500)
  78495500: __crc_scsi_is_host_device (13500)
  78509000: __primary_switched (14000)
  78523000: memset (21118000)
  99641000: __primary_switched (2500)
  99643500: start_kernel (11000)

  so we see that primary_switched is the first non-trash symbol (non-crc_* and non-kernel_flags*, which are just informative symbols, not actual executable code)

2.5.1. Linux kernel entry point

TODO https://stackoverflow.com/questions/2589845/what-are-the-first-operations-that-the-linux-kernel-executes-on-boot

As mentioned at: GDB step debug early boot, the very first kernel instructions executed appear to be placed into memory at a different location than that of the kernel ELF section.

As a result, we are unable to break on early symbols such as:

./run-gdb extract_kernel
./run-gdb main

gem5 ExecAll trace format>> however does show the right symbols however! This could be because gem5 uses vmlinux to boot, which QEMU uses the compressed version, and as mentioned on the Stack Overflow answer, the entry point is actually a tiny decompresser routine.

I also tried to hack run-gdb with:

@@ -81,7 +81,7 @@ else
 ${gdb} \
 -q \\
 -ex 'add-auto-load-safe-path $(pwd)' \\
--ex 'file vmlinux' \\
+-ex 'file arch/arm/boot/compressed/vmlinux' \\
 -ex 'target remote localhost:${port}' \\
 ${brk} \
 -ex 'continue' \\

and no I do have the symbols from arch/arm/boot/compressed/vmlinux', but the breaks still don’t work.

v4.19 also added a CONFIG_HAVE_KERNEL_UNCOMPRESSED=y option for having the kernel uncompressed which could make following the startup easier, but it is only available on s390. aarch64 however is already uncompressed by default, so might be the easiest one. See also: Section 16.20.1, “vmlinux vs bzImage vs zImage vs Image”.

You then need the associated KERNEL_UNCOMPRESSED to enable it if available:

config KERNEL_UNCOMPRESSED
    bool "None"
    depends on HAVE_KERNEL_UNCOMPRESSED

2.5.1.1. arm64 secondary CPU entry point

In gem5 aarch64 Linux v4.18, experimentally the entry point of secondary CPUs seems to be secondary_holding_pen as shown at https://gist.github.com/cirosantilli2/34a7bc450fcb6c1c1a910369be1fdd90

What happens is that:

• the bootloader goes in in WFE

• the kernel writes the entry point to the secondary CPU (the address of secondary_holding_pen) with CPU0 at the address given to the kernel in the cpu-release-addr of the DTB

• the kernel wakes up the bootloader with a SEV, and the bootloader boots to the address the kernel told it

The CPU0 action happens at: https://github.com/cirosantilli/linux/blob/v5.7/arch/arm64/kernel/smp_spin_table.c:

Here’s the code that writes the address and does SEV:

static int smp_spin_table_cpu_prepare(unsigned int cpu)
{
	__le64 __iomem *release_addr;

	if (!cpu_release_addr[cpu])
		return -ENODEV;

	/*
	 * The cpu-release-addr may or may not be inside the linear mapping.
	 * As ioremap_cache will either give us a new mapping or reuse the
	 * existing linear mapping, we can use it to cover both cases. In
	 * either case the memory will be MT_NORMAL.
	 */
	release_addr = ioremap_cache(cpu_release_addr[cpu],
				     sizeof(*release_addr));
	if (!release_addr)
		return -ENOMEM;

	/*
	 * We write the release address as LE regardless of the native
	 * endianess of the kernel. Therefore, any boot-loaders that
	 * read this address need to convert this address to the
	 * boot-loader's endianess before jumping. This is mandated by
	 * the boot protocol.
	 */
	writeq_relaxed(__pa_symbol(secondary_holding_pen), release_addr);
	__flush_dcache_area((__force void *)release_addr,
			    sizeof(*release_addr));

	/*
	 * Send an event to wake up the secondary CPU.
	 */
	sev();

and here’s the code that reads the value from the DTB:

static int smp_spin_table_cpu_init(unsigned int cpu)
{
	struct device_node *dn;
	int ret;

	dn = of_get_cpu_node(cpu, NULL);
	if (!dn)
		return -ENODEV;

	/*
	 * Determine the address from which the CPU is polling.
	 */
	ret = of_property_read_u64(dn, "cpu-release-addr",
				   &cpu_release_addr[cpu]);

2.5.2. Linux kernel arch-agnostic entry point

start_kernel is the first C function to be executed basically: https://stackoverflow.com/questions/18266063/does-kernel-have-main-function/33422401#33422401

For the earlier arch-specific entry point, see: Linux kernel entry point.

2.5.3. Linux kernel early boot messages

When booting Linux on a slow emulator like gem5, what you observe is that:

• first nothing shows for a while

• then at once, a bunch of message lines show at once followed on aarch64 Linux 5.4.3 by:

  [    0.081311] printk: console [ttyAMA0] enabled

This means of course that all the previous messages had been generated earlier and stored, but were only printed to the terminal once the terminal itself was enabled.

Notably for example the very first message:

[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x410fd070]

happens very early in the boot process.

If you get a failure before that, it will be hard to see the print messages.

One possible solution is to parse the dmesg buffer, gem5 actually implements that: gem5 m5out/system.workload.dmesg file.

2.6. GDB step debug userland processes

QEMU’s -gdb GDB breakpoints are set on virtual addresses, so you can in theory debug userland processes as well.

• https://stackoverflow.com/questions/26271901/is-it-possible-to-use-gdb-and-qemu-to-debug-linux-user-space-programs-and-kernel

• https://stackoverflow.com/questions/16273614/debug-init-on-qemu-using-gdb

You will generally want to use gdbserver for this as it is more reliable, but this method can overcome the following limitations of gdbserver:

• the emulator does not support host to guest networking. This seems to be the case for gem5 as explained at: Section 14.3.1.3, “gem5 host to guest networking”

• cannot see the start of the init process easily

• gdbserver alters the working of the kernel, and makes your run less representative

Known limitations of direct userland debugging:

• the kernel might switch context to another process or to the kernel itself e.g. on a system call, and then TODO confirm the PIC would go to weird places and source code would be missing.

  Solutions to this are being researched at: Section 2.10.1, “lx-ps”.

• TODO step into shared libraries. If I attempt to load them explicitly:

  (gdb) sharedlibrary ../../staging/lib/libc.so.0
  No loaded shared libraries match the pattern `../../staging/lib/libc.so.0'.

  since GDB does not know that libc is loaded.

2.6.1. GDB step debug userland custom init

This is the userland debug setup most likely to work, since at init time there is only one userland executable running.

For executables from the userland/ directory such as userland/posix/count.c:

• Shell 1:

  ./run --gdb-wait --kernel-cli 'init=/lkmc/posix/count.out'

• Shell 2:

  ./run-gdb --userland userland/posix/count.c main

  Alternatively, we could also pass the full path to the executable:

  ./run-gdb --userland "$(./getvar userland_build_dir)/posix/count.out" main

  Path resolution is analogous to that of ./run --baremetal.

Then, as soon as boot ends, we are left inside a debug session that looks just like what gdbserver would produce.

2.6.2. GDB step debug userland BusyBox init

BusyBox custom init process:

• Shell 1:

  ./run --gdb-wait --kernel-cli 'init=/bin/ls'

• Shell 2:

  ./run-gdb --userland "$(./getvar buildroot_build_build_dir)"/busybox-*/busybox ls_main

This follows BusyBox' convention of calling the main for each executable as <exec>_main since the busybox executable has many "mains".

BusyBox default init process:

• Shell 1:

  ./run --gdb-wait

• Shell 2:

  ./run-gdb --userland "$(./getvar buildroot_build_build_dir)"/busybox-*/busybox init_main

init cannot be debugged with gdbserver without modifying the source, or else /sbin/init exits early with:

"must be run as PID 1"

2.6.3. GDB step debug userland non-init

Non-init process:

• Shell 1:

  ./run --gdb-wait

• Shell 2:

  ./run-gdb --userland userland/linux/rand_check.c main

• Shell 1 after the boot finishes:

  ./linux/rand_check.out

This is the least reliable setup as there might be other processes that use the given virtual address.

2.6.3.1. GDB step debug userland non-init without --gdb-wait

TODO: if I try GDB step debug userland non-init without --gdb-wait and the break main that we do inside ./run-gdb says:

Cannot access memory at address 0x10604

and then GDB never breaks. Tested at ac8663a44a450c3eadafe14031186813f90c21e4 + 1.

The exact behaviour seems to depend on the architecture:

• arm: happens always

• x86_64: appears to happen only if you try to connect GDB as fast as possible, before init has been reached.

• aarch64: could not observe the problem

We have also double checked the address with:

./run-toolchain --arch arm readelf -- \
  -s "$(./getvar --arch arm userland_build_dir)/linux/myinsmod.out" | \
  grep main

and from GDB:

info line main

and both give:

000105fc

which is just 8 bytes before 0x10604.

gdbserver also says 0x10604.

However, if do a Ctrl-C in GDB, and then a direct:

b *0x000105fc

it works. Why?!

On GEM5, x86 can also give the Cannot access memory at address, so maybe it is also unreliable on QEMU, and works just by coincidence.

2.7. GDB call

GDB can call functions as explained at: https://stackoverflow.com/questions/1354731/how-to-evaluate-functions-in-gdb

However this is failing for us:

• some symbols are not visible to call even though b sees them

• for those that are, call fails with an E14 error

E.g.: if we break on __x64_sys_write on count.sh:

>>> call printk(0, "asdf")
Could not fetch register "orig_rax"; remote failure reply 'E14'
>>> b printk
Breakpoint 2 at 0xffffffff81091bca: file kernel/printk/printk.c, line 1824.
>>> call fdget_pos(fd)
No symbol "fdget_pos" in current context.
>>> b fdget_pos
Breakpoint 3 at 0xffffffff811615e3: fdget_pos. (9 locations)
>>>

even though fdget_pos is the first thing __x64_sys_write does:

581 SYSCALL_DEFINE3(write, unsigned int, fd, const char __user *, buf,
582         size_t, count)
583 {
584     struct fd f = fdget_pos(fd);

I also noticed that I get the same error:

Could not fetch register "orig_rax"; remote failure reply 'E14'

when trying to use:

fin

on many (all?) functions.

See also: https://github.com/cirosantilli/linux-kernel-module-cheat/issues/19

2.8. GDB view ARM system registers

info all-registers shows some of them.

The implementation is described at: https://stackoverflow.com/questions/46415059/how-to-observe-aarch64-system-registers-in-qemu/53043044#53043044

2.9. GDB step debug multicore userland

For a more minimal baremetal multicore setup, see: [arm-baremetal-multicore].

We can set and get which cores the Linux kernel allows a program to run on with sched_getaffinity and sched_setaffinity:

./run --cpus 2 --eval-after './linux/sched_getaffinity.out'

Source: userland/linux/sched_getaffinity.c

Sample output:

sched_getaffinity = 1 1
sched_getcpu = 1
sched_getaffinity = 1 0
sched_getcpu = 0

Which shows us that:

• initially:

  • all 2 cores were enabled as shown by sched_getaffinity = 1 1

  • the process was randomly assigned to run on core 1 (the second one) as shown by sched_getcpu = 1. If we run this several times, it will also run on core 0 sometimes.

• then we restrict the affinity to just core 0, and we see that the program was actually moved to core 0

The number of cores is modified as explained at: Section 23.3.1, “Number of cores”

taskset from the util-linux package sets the initial core affinity of a program:

./build-buildroot \
  --config 'BR2_PACKAGE_UTIL_LINUX=y' \
  --config 'BR2_PACKAGE_UTIL_LINUX_SCHEDUTILS=y' \
;
./run --eval-after 'taskset -c 1,1 ./linux/sched_getaffinity.out'

output:

sched_getaffinity = 0 1
sched_getcpu = 1
sched_getaffinity = 1 0
sched_getcpu = 0

so we see that the affinity was restricted to the second core from the start.

Let’s do a QEMU observation to justify this example being in the repository with userland breakpoints.

We will run our ./linux/sched_getaffinity.out infinitely many times, on core 0 and core 1 alternatively:

./run \
  --cpus 2 \
  --eval-after 'i=0; while true; do taskset -c $i,$i ./linux/sched_getaffinity.out; i=$((! $i)); done' \
  --gdb-wait \
;

on another shell:

./run-gdb --userland "$(./getvar userland_build_dir)/linux/sched_getaffinity.out" main

Then, inside GDB:

(gdb) info threads
  Id   Target Id         Frame
* 1    Thread 1 (CPU#0 [running]) main () at sched_getaffinity.c:30
  2    Thread 2 (CPU#1 [halted ]) native_safe_halt () at ./arch/x86/include/asm/irqflags.h:55
(gdb) c
(gdb) info threads
  Id   Target Id         Frame
  1    Thread 1 (CPU#0 [halted ]) native_safe_halt () at ./arch/x86/include/asm/irqflags.h:55
* 2    Thread 2 (CPU#1 [running]) main () at sched_getaffinity.c:30
(gdb) c

and we observe that info threads shows the actual correct core on which the process was restricted to run by taskset!

We should also try it out with kernel modules: https://stackoverflow.com/questions/28347876/set-cpu-affinity-on-a-loadable-linux-kernel-module

TODO we then tried:

./run --cpus 2 --eval-after './linux/sched_getaffinity_threads.out'

and:

./run-gdb --userland "$(./getvar userland_build_dir)/linux/sched_getaffinity_threads.out"

to switch between two simultaneous live threads with different affinities, it just didn’t break on our threads:

b main_thread_0

Note that secondary cores in gem5 are kind of broken however: gem5 GDB step debug secondary cores.

Bibliography:

• https://stackoverflow.com/questions/10490756/how-to-use-sched-getaffinity-and-sched-setaffinity-in-linux-from-c/50117787#50117787

  • https://stackoverflow.com/questions/663958/how-to-control-which-core-a-process-runs-on/50210009#50210009

  • https://stackoverflow.com/questions/280909/cpu-affinity/54478296#54478296

  • https://unix.stackexchange.com/questions/73/how-can-i-set-the-processor-affinity-of-a-process-on-linux/441098#441098 (summary only)

• https://stackoverflow.com/questions/42800801/how-to-use-gdb-to-debug-qemu-with-smp-symmetric-multiple-processors

2.10. Linux kernel GDB scripts

We source the Linux kernel GDB scripts by default for lx-symbols, but they also contains some other goodies worth looking into.

Those scripts basically parse some in-kernel data structures to offer greater visibility with GDB.

All defined commands are prefixed by lx-, so to get a full list just try to tab complete that.

There aren’t as many as I’d like, and the ones that do exist are pretty self explanatory, but let’s give a few examples.

Show dmesg:

lx-dmesg

Show the Kernel command line parameters:

lx-cmdline

Dump the device tree to a fdtdump.dtb file in the current directory:

lx-fdtdump
pwd

List inserted kernel modules:

lx-lsmod

Sample output:

Address            Module                  Size  Used by
0xffffff80006d0000 hello                  16384  0

Bibliography:

• https://events.static.linuxfound.org/sites/events/files/slides/Debugging%20the%20Linux%20Kernel%20with%20GDB.pdf

• https://wiki.linaro.org/LandingTeams/ST/GDB

2.10.1. lx-ps

List all processes:

lx-ps

Sample output:

0xffff88000ed08000 1 init
0xffff88000ed08ac0 2 kthreadd

The second and third fields are obviously PID and process name.

The first one is more interesting, and contains the address of the task_struct in memory.

This can be confirmed with:

p ((struct task_struct)*0xffff88000ed08000

which contains the correct PID for all threads I’ve tried:

pid = 1,

TODO get the PC of the kthreads: https://stackoverflow.com/questions/26030910/find-program-counter-of-process-in-kernel Then we would be able to see where the threads are stopped in the code!

On ARM, I tried:

task_pt_regs((struct thread_info *)((struct task_struct)*0xffffffc00e8f8000))->uregs[ARM_pc]

but task_pt_regs is a #define and GDB cannot see defines without -ggdb3: https://stackoverflow.com/questions/2934006/how-do-i-print-a-defined-constant-in-gdb which are apparently not set?

Bibliography:

• https://stackoverflow.com/questions/9561546/thread-aware-gdb-for-kernel

• https://wiki.linaro.org/LandingTeams/ST/GDB

• https://events.static.linuxfound.org/sites/events/files/slides/Debugging%20the%20Linux%20Kernel%20with%20GDB.pdf presentation: https://www.youtube.com/watch?v=pqn5hIrz3A8

2.10.1.1. CONFIG_PID_IN_CONTEXTIDR

https://stackoverflow.com/questions/54133479/accessing-logical-software-thread-id-in-gem5 on ARM the kernel can store an indication of PID in the CONTEXTIDR_EL1 register, making that much easier to observe from simulators.

In particular, gem5 prints that number out by default on ExecAll messages!

Let’s test it out with [linux-kernel-build-variants] + gem5 checkpoint restore and run a different script:

./build-linux --arch aarch64 --linux-build-id CONFIG_PID_IN_CONTEXTIDR --config 'CONFIG_PID_IN_CONTEXTIDR=y'
# Checkpoint run.
./run --arch aarch64 --emulator gem5 --linux-build-id CONFIG_PID_IN_CONTEXTIDR --eval './gem5.sh'
# Trace run.
./run \
  --arch aarch64 \
  --emulator gem5 \
  --gem5-readfile 'posix/getpid.out; posix/getpid.out' \
  --gem5-restore 1 \
  --linux-build-id CONFIG_PID_IN_CONTEXTIDR \
  --trace FmtFlag,ExecAll,-ExecSymbol \
;

The terminal runs both programs which output their PID to stdout:

pid=44
pid=45

By quickly inspecting the trace.txt file, we immediately notice that the system.cpu: A<n> part of the logs, which used to always be system.cpu: A0, now has a few different values! Nice!

We can briefly summarize those values by removing repetitions:

cut -d' ' -f4 "$(./getvar --arch aarch64 --emulator gem5 trace_txt_file)" | uniq -c

gives:

  97227 A39
 147476 A38
 222052 A40
      1 terminal
1117724 A40
  27529 A31
  43868 A40
  27487 A31
 138349 A40
  13781 A38
 231246 A40
  25536 A38
  28337 A40
 214799 A38
 963561 A41
  92603 A38
  27511 A31
 224384 A38
 564949 A42
 182360 A38
 729009 A43
   8398 A23
  20200 A10
 636848 A43
 187995 A44
  27529 A31
  70071 A44
  16981 A0
 623806 A44
  16981 A0
 139319 A44
  24487 A0
 174986 A44
  25420 A0
  89611 A44
  16981 A0
 183184 A44
  24728 A0
  89608 A44
  17226 A0
 899075 A44
  24974 A0
 250608 A44
 137700 A43
1497997 A45
 227485 A43
 138147 A38
 482646 A46

I’m not smart enough to be able to deduce all of those IDs, but we can at least see that:

• A44 and A45 are there as expected from stdout!

• A39 must be the end of the execution of m5 checkpoint

• so we guess that A38 is the shell as it comes next

• the weird "terminal" line is 336969745500: system.terminal: attach terminal 0

• which is the shell PID? I should have printed that as well :-)

• why are there so many other PIDs? This was supposed to be a silent system without daemons!

• A0 is presumably the kernel. However we see process switches without going into A0, so I’m not sure how, it appears to count kernel instructions as part of processes

• A46 has to be the m5 exit call

Or if you want to have some real fun, try: baremetal/arch/aarch64/contextidr_el1.c:

./run --arch aarch64 --emulator gem5 --baremetal baremetal/arch/aarch64/contextidr_el1.c --trace-insts-stdout

in which we directly set the register ourselves! Output excerpt:

  31500: system.cpu: A0 T0 : @main+12    :   ldr   x0, [sp, #12]      : MemRead :  D=0x0000000000000001 A=0x82fffffc  flags=(IsInteger|IsMemRef|IsLoad)
  32000: system.cpu: A1 T0 : @main+16    :   msr   contextidr_el1, x0 : IntAlu :  D=0x0000000000000001  flags=(IsInteger|IsSerializeAfter|IsNonSpeculative)
  32500: system.cpu: A1 T0 : @main+20    :   ldr   x0, [sp, #12]      : MemRead :  D=0x0000000000000001 A=0x82fffffc  flags=(IsInteger|IsMemRef|IsLoad)
  33000: system.cpu: A1 T0 : @main+24    :   add   w0, w0, #1         : IntAlu :  D=0x0000000000000002  flags=(IsInteger)
  33500: system.cpu: A1 T0 : @main+28    :   str   x0, [sp, #12]      : MemWrite :  D=0x0000000000000002 A=0x82fffffc  flags=(IsInteger|IsMemRef|IsStore)
  34000: system.cpu: A1 T0 : @main+32    :   ldr   x0, [sp, #12]      : MemRead :  D=0x0000000000000002 A=0x82fffffc  flags=(IsInteger|IsMemRef|IsLoad)
  34500: system.cpu: A1 T0 : @main+36    :   subs   w0, #9            : IntAlu :  D=0x0000000000000000  flags=(IsInteger)
  35000: system.cpu: A1 T0 : @main+40    :   b.le   <main+12>         : IntAlu :   flags=(IsControl|IsDirectControl|IsCondControl)
  35500: system.cpu: A1 T0 : @main+12    :   ldr   x0, [sp, #12]      : MemRead :  D=0x0000000000000002 A=0x82fffffc  flags=(IsInteger|IsMemRef|IsLoad)
  36000: system.cpu: A2 T0 : @main+16    :   msr   contextidr_el1, x0 : IntAlu :  D=0x0000000000000002  flags=(IsInteger|IsSerializeAfter|IsNonSpeculative)
  36500: system.cpu: A2 T0 : @main+20    :   ldr   x0, [sp, #12]      : MemRead :  D=0x0000000000000002 A=0x82fffffc  flags=(IsInteger|IsMemRef|IsLoad)
  37000: system.cpu: A2 T0 : @main+24    :   add   w0, w0, #1         : IntAlu :  D=0x0000000000000003  flags=(IsInteger)
  37500: system.cpu: A2 T0 : @main+28    :   str   x0, [sp, #12]      : MemWrite :  D=0x0000000000000003 A=0x82fffffc  flags=(IsInteger|IsMemRef|IsStore)
  38000: system.cpu: A2 T0 : @main+32    :   ldr   x0, [sp, #12]      : MemRead :  D=0x0000000000000003 A=0x82fffffc  flags=(IsInteger|IsMemRef|IsLoad)
  38500: system.cpu: A2 T0 : @main+36    :   subs   w0, #9            : IntAlu :  D=0x0000000000000000  flags=(IsInteger)
  39000: system.cpu: A2 T0 : @main+40    :   b.le   <main+12>         : IntAlu :   flags=(IsControl|IsDirectControl|IsCondControl)
  39500: system.cpu: A2 T0 : @main+12    :   ldr   x0, [sp, #12]      : MemRead :  D=0x0000000000000003 A=0x82fffffc  flags=(IsInteger|IsMemRef|IsLoad)
  40000: system.cpu: A3 T0 : @main+16    :   msr   contextidr_el1, x0 : IntAlu :  D=0x0000000000000003  flags=(IsInteger|IsSerializeAfter|IsNonSpeculative)

[armarm8-fa] D13.2.27 "CONTEXTIDR_EL1, Context ID Register (EL1)" documents CONTEXTIDR_EL1 as:

Identifies the current Process Identifier.

The value of the whole of this register is called the Context ID and is used by:

• The debug logic, for Linked and Unlinked Context ID matching.

• The trace logic, to identify the current process.

The significance of this register is for debug and trace use only.

Tested on 145769fc387dc5ee63ec82e55e6b131d9c968538 + 1.

2.11. Debug the GDB remote protocol

For when it breaks again, or you want to add a new feature!

./run --debug
./run-gdb --before '-ex "set remotetimeout 99999" -ex "set debug remote 1"' start_kernel

See also: https://stackoverflow.com/questions/13496389/gdb-remote-protocol-how-to-analyse-packets

2.11.1. Remote 'g' packet reply is too long

This error means that the GDB server, e.g. in QEMU, sent more registers than the GDB client expected.

This can happen for the following reasons:

• you set the architecture of the client wrong, often 32 vs 64 bit as mentioned at: https://stackoverflow.com/questions/4896316/gdb-remote-cross-debugging-fails-with-remote-g-packet-reply-is-too-long

• there is a bug in the GDB server and the XML description does not match the number of registers actually sent

• the GDB server does not send XML target descriptions and your GDB expects a different number of registers by default. E.g., gem5 d4b3e064adeeace3c3e7d106801f95c14637c12f does not send the XML files

The XML target description format is described a bit further at: https://stackoverflow.com/questions/46415059/how-to-observe-aarch64-system-registers-in-qemu/53043044#53043044

3. KGDB

KGDB is kernel dark magic that allows you to GDB the kernel on real hardware without any extra hardware support.

It is useless with QEMU since we already have full system visibility with -gdb. So the goal of this setup is just to prepare you for what to expect when you will be in the treches of real hardware.

KGDB is cheaper than JTAG (free) and easier to setup (all you need is serial), but with less visibility as it depends on the kernel working, so e.g.: dies on panic, does not see boot sequence.

First run the kernel with:

./run --kgdb

this passes the following options on the kernel CLI:

kgdbwait kgdboc=ttyS1,115200

kgdbwait tells the kernel to wait for KGDB to connect.

So the kernel sets things up enough for KGDB to start working, and then boot pauses waiting for connection:

<6>[    4.866050] Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled
<6>[    4.893205] 00:05: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A
<6>[    4.916271] 00:06: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A
<6>[    4.987771] KGDB: Registered I/O driver kgdboc
<2>[    4.996053] KGDB: Waiting for connection from remote gdb...

Entering kdb (current=0x(____ptrval____), pid 1) on processor 0 due to Keyboard Entry
[0]kdb>

KGDB expects the connection at ttyS1, our second serial port after ttyS0 which contains the terminal.

The last line is the KDB prompt, and is covered at: Section 3.3, “KDB”. Typing now shows nothing because that prompt is expecting input from ttyS1.

Instead, we connect to the serial port ttyS1 with GDB:

./run-gdb --kgdb --no-continue

Once GDB connects, it is left inside the function kgdb_breakpoint.

So now we can set breakpoints and continue as usual.

For example, in GDB:

continue

Then in QEMU:

./count.sh &
./kgdb.sh

rootfs_overlay/lkmc/kgdb.sh pauses the kernel for KGDB, and gives control back to GDB.

And now in GDB we do the usual:

break __x64_sys_write
continue
continue
continue
continue

And now you can count from KGDB!

If you do: break __x64_sys_write immediately after ./run-gdb --kgdb, it fails with KGDB: BP remove failed: <address>. I think this is because it would break too early on the boot sequence, and KGDB is not yet ready.

See also:

• https://github.com/torvalds/linux/blob/v4.9/Documentation/DocBook/kgdb.tmpl

• https://stackoverflow.com/questions/22004616/qemu-kernel-debugging-with-kgdb/44197715#44197715

3.1. KGDB ARM

TODO: we would need a second serial for KGDB to work, but it is not currently supported on arm and aarch64 with -M virt that we use: https://unix.stackexchange.com/questions/479085/can-qemu-m-virt-on-arm-aarch64-have-multiple-serial-ttys-like-such-as-pl011-t/479340#479340

One possible workaround for this would be to use KDB ARM.

Main more generic question: https://stackoverflow.com/questions/14155577/how-to-use-kgdb-on-arm

3.2. KGDB kernel modules

Just works as you would expect:

insmod timer.ko
./kgdb.sh

In GDB:

break lkmc_timer_callback
continue
continue
continue

and you now control the count.

3.3. KDB

KDB is a way to use KDB directly in your main console, without GDB.

Advantage over KGDB: you can do everything in one serial. This can actually be important if you only have one serial for both shell and .

Disadvantage: not as much functionality as GDB, especially when you use Python scripts. Notably, TODO confirm you can’t see the the kernel source code and line step as from GDB, since the kernel source is not available on guest (ah, if only debugging information supported full source, or if the kernel had a crazy mechanism to embed it).

Run QEMU as:

./run --kdb

This passes kgdboc=ttyS0 to the Linux CLI, therefore using our main console. Then QEMU:

[0]kdb> go

And now the kdb> prompt is responsive because it is listening to the main console.

After boot finishes, run the usual:

./count.sh &
./kgdb.sh

And you are back in KDB. Now you can count with:

[0]kdb> bp __x64_sys_write
[0]kdb> go
[0]kdb> go
[0]kdb> go
[0]kdb> go

And you will break whenever __x64_sys_write is hit.

You can get see further commands with:

[0]kdb> help

The other KDB commands allow you to step instructions, view memory, registers and some higher level kernel runtime data similar to the superior GDB Python scripts.

3.3.1. KDB graphic

You can also use KDB directly from the graphic window with:

./run --graphic --kdb

This setup could be used to debug the kernel on machines without serial, such as modern desktops.

This works because --graphics adds kbd (which stands for KeyBoarD!) to kgdboc.

3.3.2. KDB ARM

TODO neither arm and aarch64 are working as of 1cd1e58b023791606498ca509256cc48e95e4f5b + 1.

arm seems to place and hit the breakpoint correctly, but no matter how many go commands I do, the count.sh stdout simply does not show.

aarch64 seems to place the breakpoint correctly, but after the first go the kernel oopses with warning:

WARNING: CPU: 0 PID: 46 at /root/linux-kernel-module-cheat/submodules/linux/kernel/smp.c:416 smp_call_function_many+0xdc/0x358

and stack trace:

smp_call_function_many+0xdc/0x358
kick_all_cpus_sync+0x30/0x38
kgdb_flush_swbreak_addr+0x3c/0x48
dbg_deactivate_sw_breakpoints+0x7c/0xb8
kgdb_cpu_enter+0x284/0x6a8
kgdb_handle_exception+0x138/0x240
kgdb_brk_fn+0x2c/0x40
brk_handler+0x7c/0xc8
do_debug_exception+0xa4/0x1c0
el1_dbg+0x18/0x78
__arm64_sys_write+0x0/0x30
el0_svc_handler+0x74/0x90
el0_svc+0x8/0xc

My theory is that every serious ARM developer has JTAG, and no one ever tests this, and the kernel code is just broken.

4. gdbserver

Step debug userland processes to understand how they are talking to the kernel.

First build gdbserver into the root filesystem:

./build-buildroot --config 'BR2_PACKAGE_GDB=y'

Then on guest, to debug userland/linux/rand_check.c:

./gdbserver.sh ./c/command_line_arguments.out asdf qwer

Source: rootfs_overlay/lkmc/gdbserver.sh.

And on host:

./run-gdb --gdbserver --userland userland/c/command_line_arguments.c main

or alternatively with the path to the executable itself:

./run --gdbserver --userland "$(./getvar userland_build_dir)/c/command_line_arguments.out"

Bibliography: https://reverseengineering.stackexchange.com/questions/8829/cross-debugging-for-arm-mips-elf-with-qemu-toolchain/16214#16214

4.1. gdbserver BusyBox

Analogous to GDB step debug userland processes:

./gdbserver.sh ls

on host you need:

./run-gdb --gdbserver --userland "$(./getvar buildroot_build_build_dir)"/busybox-*/busybox ls_main

4.2. gdbserver libc

Our setup gives you the rare opportunity to step debug libc and other system libraries.

For example in the guest:

./gdbserver.sh ./posix/count.out

Then on host:

./run-gdb --gdbserver --userland userland/posix/count.c main

and inside GDB:

break sleep
continue

And you are now left inside the sleep function of our default libc implementation uclibc libc/unistd/sleep.c!

You can also step into the sleep call:

step

This is made possible by the GDB command that we use by default:

set sysroot ${common_buildroot_build_dir}/staging

which automatically finds unstripped shared libraries on the host for us.

See also: https://stackoverflow.com/questions/8611194/debugging-shared-libraries-with-gdbserver/45252113#45252113

4.3. gdbserver dynamic loader

TODO: try to step debug the dynamic loader. Would be even easier if starti is available: https://stackoverflow.com/questions/10483544/stopping-at-the-first-machine-code-instruction-in-gdb

Bibliography: https://stackoverflow.com/questions/20114565/gdb-step-into-dynamic-linkerld-so-code

5. CPU architecture

The portability of the kernel and toolchains is amazing: change an option and most things magically work on completely different hardware.

To use arm instead of x86 for example:

./build-buildroot --arch arm
./run --arch arm

Debug:

./run --arch arm --gdb-wait
# On another terminal.
./run-gdb --arch arm

We also have one letter shorthand names for the architectures and --arch option:

# aarch64
./run -a A
# arm
./run -a a
# x86_64
./run -a x

Known quirks of the supported architectures are documented in this section.

5.1. x86_64

5.1.1. ring0

This example illustrates how reading from the x86 control registers with mov crX, rax can only be done from kernel land on ring0.

From kernel land:

insmod ring0.ko

works and output the registers, for example:

cr0 = 0xFFFF880080050033
cr2 = 0xFFFFFFFF006A0008
cr3 = 0xFFFFF0DCDC000

However if we try to do it from userland:

./ring0.out

stdout gives:

Segmentation fault

and dmesg outputs:

traps: ring0.out[55] general protection ip:40054c sp:7fffffffec20 error:0 in ring0.out[400000+1000]

Sources:

• kernel_modules/ring0.c

• lkmc/ring0.h

• userland/arch/x86_64/ring0.c

In both cases, we attempt to run the exact same code which is shared on the ring0.h header file.

Bibliography:

• https://stackoverflow.com/questions/7415515/how-to-access-the-control-registers-cr0-cr2-cr3-from-a-program-getting-segmenta/7419306#7419306

• https://stackoverflow.com/questions/18717016/what-are-ring-0-and-ring-3-in-the-context-of-operating-systems/44483439#44483439

5.2. arm

5.2.1. Run arm executable in aarch64

TODO Can you run arm executables in the aarch64 guest? https://stackoverflow.com/questions/22460589/armv8-running-legacy-32-bit-applications-on-64-bit-os/51466709#51466709

I’ve tried:

./run-toolchain --arch aarch64 gcc -- -static ~/test/hello_world.c -o "$(./getvar p9_dir)/a.out"
./run --arch aarch64 --eval-after '/mnt/9p/data/a.out'

but it fails with:

a.out: line 1: syntax error: unexpected word (expecting ")")

5.3. MIPS

We used to "support" it until f8c0502bb2680f2dbe7c1f3d7958f60265347005 (it booted) but dropped since one was testing it often.

If you want to revive and maintain it, send a pull request.

5.4. Other architectures

It should not be too hard to port this repository to any architecture that Buildroot supports. Pull requests are welcome.

6. init

When the Linux kernel finishes booting, it runs an executable as the first and only userland process. This executable is called the init program.

The init process is then responsible for setting up the entire userland (or destroying everything when you want to have fun).

This typically means reading some configuration files (e.g. /etc/initrc) and forking a bunch of userland executables based on those files, including the very interactive shell that we end up on.

systemd provides a "popular" init implementation for desktop distros as of 2017.

BusyBox provides its own minimalistic init implementation which Buildroot, and therefore this repo, uses by default.

The init program can be either an executable shell text file, or a compiled ELF file. It becomes easy to accept this once you see that the exec system call handles both cases equally: https://unix.stackexchange.com/questions/174062/can-the-init-process-be-a-shell-script-in-linux/395375#395375

The init executable is searched for in a list of paths in the root filesystem, including /init, /sbin/init and a few others. For more details see: Section 6.3, “Path to init”

6.1. Replace init

To have more control over the system, you can replace BusyBox’s init with your own.

The most direct way to replace init with our own is to just use the init= command line parameter directly:

./run --kernel-cli 'init=/lkmc/count.sh'

This just counts every second forever and does not give you a shell.

This method is not very flexible however, as it is hard to reliably pass multiple commands and command line arguments to the init with it, as explained at: Section 6.4, “Init environment”.

For this reason, we have created a more robust helper method with the --eval option:

./run --eval 'echo "asdf qwer";insmod hello.ko;./linux/poweroff.out'

It is basically a shortcut for:

./run --kernel-cli 'init=/lkmc/eval_base64.sh - lkmc_eval="insmod hello.ko;./linux/poweroff.out"'

Source: rootfs_overlay/lkmc/eval_base64.sh.

This allows quoting and newlines by base64 encoding on host, and decoding on guest, see: Section 16.3.1, “Kernel command line parameters escaping”.

It also automatically chooses between init= and rcinit= for you, see: Section 6.3, “Path to init”

--eval replaces BusyBox' init completely, which makes things more minimal, but also has has the following consequences:

• /etc/fstab mounts are not done, notably /proc and /sys, test it out with:

  ./run --eval 'echo asdf;ls /proc;ls /sys;echo qwer'

• no shell is launched at the end of boot for you to interact with the system. You could explicitly add a sh at the end of your commands however:

  ./run --eval 'echo hello;sh'

The best way to overcome those limitations is to use: Section 6.2, “Run command at the end of BusyBox init”

If the script is large, you can add it to a gitignored file and pass that to --eval as in:

echo '
cd /lkmc
insmod hello.ko
./linux/poweroff.out
' > data/gitignore.sh
./run --eval "$(cat data/gitignore.sh)"

or add it to a file to the root filesystem guest and rebuild:

echo '#!/bin/sh
cd /lkmc
insmod hello.ko
./linux/poweroff.out
' > rootfs_overlay/lkmc/gitignore.sh
chmod +x rootfs_overlay/lkmc/gitignore.sh
./build-buildroot
./run --kernel-cli 'init=/lkmc/gitignore.sh'

Remember that if your init returns, the kernel will panic, there are just two non-panic possibilities:

• run forever in a loop or long sleep

• poweroff the machine

6.1.1. poweroff.out

Just using BusyBox' poweroff at the end of the init does not work and the kernel panics:

./run --eval poweroff

because BusyBox' poweroff tries to do some fancy stuff like killing init, likely to allow userland to shutdown nicely.

But this fails when we are init itself!

BusyBox' poweroff works more brutally and effectively if you add -f:

./run --eval 'poweroff -f'

but why not just use our minimal ./linux/poweroff.out and be done with it?

./run --eval './linux/poweroff.out'

Source: userland/linux/poweroff.c

This also illustrates how to shutdown the computer from C: https://stackoverflow.com/questions/28812514/how-to-shutdown-linux-using-c-or-qt-without-call-to-system

6.1.2. sleep_forever.out

I dare you to guess what this does:

./run --eval './posix/sleep_forever.out'

Source: userland/posix/sleep_forever.c

This executable is a convenient simple init that does not panic and sleeps instead.

6.1.3. time_boot.out

Get a reasonable answer to "how long does boot take in guest time?":

./run --eval-after './linux/time_boot.c'

Source: userland/linux/time_boot.c

That executable writes to dmesg directly through /dev/kmsg a message of type:

[    2.188242] /path/to/linux-kernel-module-cheat/userland/linux/time_boot.c

which tells us that boot took 2.188242 seconds based on the dmesg timestamp.

Bibliography: https://stackoverflow.com/questions/12683169/measure-time-taken-for-linux-kernel-from-bootup-to-userpace/46517014#46517014

6.2. Run command at the end of BusyBox init

Use the --eval-after option is for you rely on something that BusyBox' init set up for you like /etc/fstab:

./run --eval-after 'echo asdf;ls /proc;ls /sys;echo qwer'

After the commands run, you are left on an interactive shell.

The above command is basically equivalent to:

./run --kernel-cli-after-dash 'lkmc_eval="insmod hello.ko;./linux/poweroff.out;"'

where the lkmc_eval option gets evaled by our default rootfs_overlay/etc/init.d/S98 startup script.

Except that --eval-after is smarter and uses base64 encoding.

Alternatively, you can also add the comamdns to run to a new init.d entry to run at the end o the BusyBox init:

cp rootfs_overlay/etc/init.d/S98 rootfs_overlay/etc/init.d/S99.gitignore
vim rootfs_overlay/etc/init.d/S99.gitignore
./build-buildroot
./run

and they will be run automatically before the login prompt.

Scripts under /etc/init.d are run by /etc/init.d/rcS, which gets called by the line ::sysinit:/etc/init.d/rcS in /etc/inittab.

6.3. Path to init

The init is selected at:

• initrd or initramfs system: /init, a custom one can be set with the rdinit= kernel command line parameter

• otherwise: default is /sbin/init, followed by some other paths, a custom one can be set with init=

More details: https://unix.stackexchange.com/questions/30414/what-can-make-passing-init-path-to-program-to-the-kernel-not-start-program-as-i/430614#430614

The final init that actually got selected is shown on Linux v5.9.2 a line of type:

<6>[    0.309984] Run /sbin/init as init process

at the very end of the boot logs.

6.4. Init environment

Documented at https://www.kernel.org/doc/html/v4.14/admin-guide/kernel-parameters.html:

The kernel parses parameters from the kernel command line up to "-"; if it doesn’t recognize a parameter and it doesn’t contain a '.', the parameter gets passed to init: parameters with '=' go into init’s environment, others are passed as command line arguments to init. Everything after "-" is passed as an argument to init.

And you can try it out with:

./run --kernel-cli 'init=/lkmc/linux/init_env_poweroff.out' --kernel-cli-after-dash 'asdf=qwer zxcv'

From the generated QEMU command, we see that the kernel CLI at LKMC 69f5745d3df11d5c741551009df86ea6c61a09cf now contains:

init=/lkmc/linux/init_env_poweroff.out console=ttyS0 - lkmc_home=/lkmc asdf=qwer zxcv

and the init program outputs:

args:
/lkmc/linux/init_env_poweroff.out
-
zxcv

env:
HOME=/
TERM=linux
lkmc_home=/lkmc
asdf=qwer

Source: userland/linux/init_env_poweroff.c.

As of the Linux kernel v5.7 (possibly earlier, I’ve skipped a few releases), boot also shows the init arguments and environment very clearly, which is a great addition:

<6>[    0.309984] Run /sbin/init as init process
<7>[    0.309991]   with arguments:
<7>[    0.309997]     /sbin/init
<7>[    0.310004]     nokaslr
<7>[    0.310010]     -
<7>[    0.310016]   with environment:
<7>[    0.310022]     HOME=/
<7>[    0.310028]     TERM=linux
<7>[    0.310035]     earlyprintk=pl011,0x1c090000
<7>[    0.310041]     lkmc_home=/lkmc

6.4.1. init arguments

The annoying dash - gets passed as a parameter to init, which makes it impossible to use this method for most non custom executables.

Arguments with dots that come after - are still treated specially (of the form subsystem.somevalue) and disappear, from args, e.g.:

./run --kernel-cli 'init=/lkmc/linux/init_env_poweroff.out' --kernel-cli-after-dash '/lkmc/linux/poweroff.out'

outputs:

args
/lkmc/linux/init_env_poweroff.out
-
ab

so see how a.b is gone.

The simple workaround is to just create a shell script that does it, e.g. as we’ve done at: rootfs_overlay/lkmc/gem5_exit.sh.

6.4.2. init environment env

Wait, where do HOME and TERM come from? (greps the kernel). Ah, OK, the kernel sets those by default: https://github.com/torvalds/linux/blob/94710cac0ef4ee177a63b5227664b38c95bbf703/init/main.c#L173

const char *envp_init[MAX_INIT_ENVS+2] = { "HOME=/", "TERM=linux", NULL, };

6.4.3. BusyBox shell init environment

On top of the Linux kernel, the BusyBox /bin/sh shell will also define other variables.

We can explore the shenanigans that the shell adds on top of the Linux kernel with:

./run --kernel-cli 'init=/bin/sh'

From there we observe that:

env

gives:

SHLVL=1
HOME=/
TERM=linux
PWD=/

therefore adding SHLVL and PWD to the default kernel exported variables.

Furthermore, to increase confusion, if you list all non-exported shell variables https://askubuntu.com/questions/275965/how-to-list-all-variables-names-and-their-current-values with:

set

then it shows more variables, notably:

PATH='/sbin:/usr/sbin:/bin:/usr/bin'

6.4.3.1. BusyBox shell initrc files

Login shells source some default files, notably:

/etc/profile
$HOME/.profile

In our case, HOME is set to / presumably by init at: https://git.busybox.net/busybox/tree/init/init.c?id=5059653882dbd86e3bbf48389f9f81b0fac8cd0a#n1114

We provide /.profile from rootfs_overlay/.profile, and use the default BusyBox /etc/profile.

The shell knows that it is a login shell if the first character of argv[0] is -, see also: https://stackoverflow.com/questions/2050961/is-argv0-name-of-executable-an-accepted-standard-or-just-a-common-conventi/42291142#42291142

When we use just init=/bin/sh, the Linux kernel sets argv[0] to /bin/sh, which does not start with -.

However, if you use ::respawn:-/bin/sh on inttab described at TTY, BusyBox' init sets argv[0][0] to -, and so does getty. This can be observed with:

cat /proc/$$/cmdline

where $$ is the PID of the shell itself: https://stackoverflow.com/questions/21063765/get-pid-in-shell-bash

Bibliography: https://unix.stackexchange.com/questions/176027/ash-profile-configuration-file

7. initrd

The kernel can boot from an CPIO file, which is a directory serialization format much like tar: https://superuser.com/questions/343915/tar-vs-cpio-what-is-the-difference

The bootloader, which for us is provided by QEMU itself, is then configured to put that CPIO into memory, and tell the kernel that it is there.

This is very similar to the kernel image itself, which already gets put into memory by the QEMU -kernel option.

With this setup, you don’t even need to give a root filesystem to the kernel: it just does everything in memory in a ramfs.

To enable initrd instead of the default ext2 disk image, do:

./build-buildroot --initrd
./run --initrd

By looking at the QEMU run command generated, you can see that we didn’t give the -drive option at all:

cat "$(./getvar run_dir)/run.sh"

Instead, we used the QEMU -initrd option to point to the .cpio filesystem that Buildroot generated for us.

Try removing that -initrd option to watch the kernel panic without rootfs at the end of boot.

When using .cpio, there can be no filesystem persistency across boots, since all file operations happen in memory in a tmpfs:

date >f
poweroff
cat f
# can't open 'f': No such file or directory

which can be good for automated tests, as it ensures that you are using a pristine unmodified system image every time.

Not however that we already disable disk persistency by default on ext2 filesystems even without --initrd: Section 22.3, “Disk persistency”.

One downside of this method is that it has to put the entire filesystem into memory, and could lead to a panic:

end Kernel panic - not syncing: Out of memory and no killable processes...

This can be solved by increasing the memory as explained at Memory size:

./run --initrd --memory 256M

The main ingredients to get initrd working are:

• BR2_TARGET_ROOTFS_CPIO=y: make Buildroot generate images/rootfs.cpio in addition to the other images.

  It is also possible to compress that image with other options.

• qemu -initrd: make QEMU put the image into memory and tell the kernel about it.

• CONFIG_BLK_DEV_INITRD=y: Compile the kernel with initrd support, see also: https://unix.stackexchange.com/questions/67462/linux-kernel-is-not-finding-the-initrd-correctly/424496#424496

  Buildroot forces that option when BR2_TARGET_ROOTFS_CPIO=y is given

TODO: how does the bootloader inform the kernel where to find initrd? https://unix.stackexchange.com/questions/89923/how-does-linux-load-the-initrd-image

7.1. initrd in desktop distros

Most modern desktop distributions have an initrd in their root disk to do early setup.

The rationale for this is described at: https://en.wikipedia.org/wiki/Initial_ramdisk

One obvious use case is having an encrypted root filesystem: you keep the initrd in an unencrypted partition, and then setup decryption from there.

I think GRUB then knows read common disk formats, and then loads that initrd to memory with a /boot/grub/grub.cfg directive of type:

initrd /initrd.img-4.4.0-108-generic

Related: https://stackoverflow.com/questions/6405083/initrd-and-booting-the-linux-kernel

7.2. initramfs

initramfs is just like initrd, but you also glue the image directly to the kernel image itself using the kernel’s build system.

Try it out with:

./build-buildroot --initramfs
./build-linux --initramfs
./run --initramfs

Notice how we had to rebuild the Linux kernel this time around as well after Buildroot, since in that build we will be gluing the CPIO to the kernel image.

Now, once again, if we look at the QEMU run command generated, we see all that QEMU needs is the -kernel option, no -drive not even -initrd! Pretty cool:

cat "$(./getvar run_dir)/run.sh"

It is also interesting to observe how this increases the size of the kernel image if you do a:

ls -lh "$(./getvar linux_image)"

before and after using initramfs, since the .cpio is now glued to the kernel image.

Don’t forget that to stop using initramfs, you must rebuild the kernel without --initramfs to get rid of the attached CPIO image:

./build-linux
./run

Alternatively, consider using [linux-kernel-build-variants] if you need to switch between initramfs and non initramfs often:

./build-buildroot --initramfs
./build-linux --initramfs --linux-build-id initramfs
./run --initramfs --linux-build-id

Setting up initramfs is very easy: our scripts just set CONFIG_INITRAMFS_SOURCE to point to the CPIO path.

http://nairobi-embedded.org/initramfs_tutorial.html shows a full manual setup.

7.3. rootfs

This is how /proc/mounts shows the root filesystem:

• hard disk: /dev/root on / type ext2 (rw,relatime,block_validity,barrier,user_xattr). That file does not exist however.

• initrd: rootfs on / type rootfs (rw)

• initramfs: rootfs on / type rootfs (rw)

TODO: understand /dev/root better:

• https://unix.stackexchange.com/questions/295060/why-on-some-linux-systems-does-the-root-filesystem-appear-as-dev-root-instead

• https://superuser.com/questions/1213770/how-do-you-determine-the-root-device-if-dev-root-is-missing

7.3.1. /dev/root

See: Section 7.3, “rootfs”

7.4. gem5 initrd

TODO we were not able to get it working yet: https://stackoverflow.com/questions/49261801/how-to-boot-the-linux-kernel-with-initrd-or-initramfs-with-gem5

This would require gem5 to load the CPIO into memory, just like QEMU. Grepping initrd shows some ARM hits under:

src/arch/arm/linux/atag.hh

but they are commented out.

7.5. gem5 initramfs

This could in theory be easier to make work than initrd since the emulator does not have to do anything special.

However, it didn’t: boot fails at the end because it does not see the initramfs, but rather tries to open our dummy root filesystem, which unsurprisingly does not have a format in a way that the kernel understands:

VFS: Cannot open root device "sda" or unknown-block(8,0): error -5

We think that this might be because gem5 boots directly vmlinux, and not from the final compressed images that contain the attached rootfs such as bzImage, which is what QEMU does, see also: Section 16.20.1, “vmlinux vs bzImage vs zImage vs Image”.

To do this failed test, we automatically pass a dummy disk image as of gem5 7fa4c946386e7207ad5859e8ade0bbfc14000d91 since the scripts don’t handle a missing --disk-image well, much like is currently done for [baremetal].

Interestingly, using initramfs significantly slows down the gem5 boot, even though it did not work. For example, we’ve observed a 4x slowdown of as 17062a2e8b6e7888a14c3506e9415989362c58bf for aarch64. This must be because expanding the large attached CPIO must be expensive. We can clearly see from the kernel logs that the kernel just hangs at a point after the message PCI: CLS 0 bytes, default 64 for a long time before proceeding further.

8. Device tree

The device tree is a Linux kernel defined data structure that serves to inform the kernel how the hardware is setup.

Device trees serve to reduce the need for hardware vendors to patch the kernel: they just provide a device tree file instead, which is much simpler.

x86 does not use it device trees, but many other archs to, notably ARM.

This is notably because ARM boards:

• typically don’t have discoverable hardware extensions like PCI, but rather just put everything on an SoC with magic register addresses

• are made by a wide variety of vendors due to ARM’s licensing business model, which increases variability

The Linux kernel itself has several device trees under ./arch/<arch>/boot/dts, see also: https://stackoverflow.com/questions/21670967/how-to-compile-dts-linux-device-tree-source-files-to-dtb/42839737#42839737

8.1. DTB files

Files that contain device trees have the .dtb extension when compiled, and .dts when in text form.

You can convert between those formats with:

"$(./getvar buildroot_host_dir)"/bin/dtc -I dtb -O dts -o a.dts a.dtb
"$(./getvar buildroot_host_dir)"/bin/dtc -I dts -O dtb -o a.dtb a.dts

Buildroot builds the tool due to BR2_PACKAGE_HOST_DTC=y.

On Ubuntu 18.04, the package is named:

sudo apt-get install device-tree-compiler

See also: https://stackoverflow.com/questions/14000736/tool-to-visualize-the-device-tree-file-dtb-used-by-the-linux-kernel/39931834#39931834

Device tree files are provided to the emulator just like the root filesystem and the Linux kernel image.

In real hardware, those components are also often provided separately. For example, on the Raspberry Pi 2, the SD card must contain two partitions:

• the first contains all magic files, including the Linux kernel and the device tree

• the second contains the root filesystem

See also: https://stackoverflow.com/questions/29837892/how-to-run-a-c-program-with-no-os-on-the-raspberry-pi/40063032#40063032

8.2. Device tree syntax

Good format descriptions:

• https://www.raspberrypi.org/documentation/configuration/device-tree.md

Minimal example

/dts-v1/;

/ {
    a;
};

Check correctness with:

dtc a.dts

Separate nodes are simply merged by node path, e.g.:

/dts-v1/;

/ {
    a;
};

/ {
    b;
};

then dtc a.dts gives:

/dts-v1/;

/ {
        a;
        b;
};

8.3. Get device tree from a running kernel

https://unix.stackexchange.com/questions/265890/is-it-possible-to-get-the-information-for-a-device-tree-using-sys-of-a-running/330926#330926

This is specially interesting because QEMU and gem5 are capable of generating DTBs that match the selected machine depending on dynamic command line parameters for some types of machines.

So observing the device tree from the guest allows to easily see what the emulator has generated.

Compile the dtc tool into the root filesystem:

./build-buildroot \
  --arch aarch64 \
  --config 'BR2_PACKAGE_DTC=y' \
  --config 'BR2_PACKAGE_DTC_PROGRAMS=y' \
;

-M virt for example, which we use by default for aarch64, boots just fine without the -dtb option:

./run --arch aarch64

Then, from inside the guest:

dtc -I fs -O dts /sys/firmware/devicetree/base

contains:

        cpus {
                #address-cells = <0x1>;
                #size-cells = <0x0>;

                cpu@0 {
                        compatible = "arm,cortex-a57";
                        device_type = "cpu";
                        reg = <0x0>;
                };
        };

8.4. Device tree emulator generation

Since emulators know everything about the hardware, they can automatically generate device trees for us, which is very convenient.

This is the case for both QEMU and gem5.

For example, if we increase the number of cores to 2:

./run --arch aarch64 --cpus 2

QEMU automatically adds a second CPU to the DTB!

                cpu@0 {
                cpu@1 {

The action seems to be happening at: hw/arm/virt.c.

You can dump the DTB QEMU generated with:

./run --arch aarch64 -- -machine dumpdtb=dtb.dtb

as mentioned at: https://lists.gnu.org/archive/html/qemu-discuss/2017-02/msg00051.html

gem5 fs_bigLITTLE 2a9573f5942b5416fb0570cf5cb6cdecba733392 can also generate its own DTB.

gem5 can generate DTBs on ARM with --generate-dtb. The generated DTB is placed in the m5out directory named as system.dtb.

9. KVM

KVM is Linux kernel interface that greatly speeds up execution of virtual machines.

You can make QEMU or gem5 by passing enabling KVM with:

./run --kvm

KVM works by running userland instructions natively directly on the real hardware instead of running a software simulation of those instructions.

Therefore, KVM only works if you the host architecture is the same as the guest architecture. This means that this will likely only work for x86 guests since almost all development machines are x86 nowadays. Unless you are running an ARM desktop for some weird reason :-)

We don’t enable KVM by default because:

• it limits visibility, since more things are running natively:

  • can’t use GDB

  • can’t do instruction tracing

  • on gem5, you lose cycle counts and therefor any notion of performance

• QEMU kernel boots are already fast enough for most purposes without it

One important use case for KVM is to fast forward gem5 execution, often to skip boot, take a gem5 checkpoint, and then move on to a more detailed and slow simulation

9.1. KVM arm

TODO: we haven’t gotten it to work yet, but it should be doable, and this is an outline of how to do it. Just don’t expect this to tested very often for now.

We can test KVM on arm by running this repository inside an Ubuntu arm QEMU VM.

This produces no speedup of course, since the VM is already slow since it cannot use KVM on the x86 host.

First, obtain an Ubuntu arm64 virtual machine as explained at: https://askubuntu.com/questions/281763/is-there-any-prebuilt-qemu-ubuntu-image32bit-online/1081171#1081171

Then, from inside that image:

sudo apt-get install git
git clone https://github.com/cirosantilli/linux-kernel-module-cheat
cd linux-kernel-module-cheat
sudo ./setup -y

and then proceed exactly as in Prebuilt setup.

We don’t want to build the full Buildroot image inside the VM as that would be way too slow, thus the recommendation for the prebuilt setup.

TODO: do the right thing and cross compile QEMU and gem5. gem5’s Python parts might be a pain. QEMU should be easy: https://stackoverflow.com/questions/26514252/cross-compile-qemu-for-arm

9.2. gem5 KVM

While gem5 does have KVM, as of 2019 its support has not been very good, because debugging it is harder and people haven’t focused intensively on it.

X86 was broken with pending patches: https://www.mail-archive.com/gem5-users@gem5.org/msg15046.html It failed immediately on:

panic: KVM: Failed to enter virtualized mode (hw reason: 0x80000021)

also mentioned at:

• https://stackoverflow.com/questions/62687463/gem5-kvm-doesnt-work-with-error-0x80000021

• https://gem5-users.gem5.narkive.com/8DBihuUx/running-fs-py-with-x86kvmcpu-failed

Bibliography:

• ARM thread: https://stackoverflow.com/questions/53523087/how-to-run-gem5-on-kvm-on-arm-with-multiple-cores

10. User mode simulation

Both QEMU and gem5 have an user mode simulation mode in addition to full system simulation that we consider elsewhere in this project.

In QEMU, it is called just "user mode", and in gem5 it is called syscall emulation mode.

In both, the basic idea is the same.

User mode simulation takes regular userland executables of any arch as input and executes them directly, without booting a kernel.

Instead of simulating the full system, it translates normal instructions like in full system mode, but magically forwards system calls to the host OS.

Advantages over full system simulation:

• the simulation may run faster since you don’t have to simulate the Linux kernel and several device models

• you don’t need to build your own kernel or root filesystem, which saves time. You still need a toolchain however, but the pre-packaged ones may work fine.

Disadvantages:

• lower guest to host portability:

  • TODO confirm: host OS == guest OS?

  • TODO confirm: the host Linux kernel should be newer than the kernel the executable was built for.

    It may still work even if that is not the case, but could fail is a missing system call is reached.

    The target Linux kernel of the executable is a GCC toolchain build-time configuration.

  • emulator implementers have to keep up with libc changes, some of which break even a C hello world due setup code executed before main.

    See also: Section 10.4, “User mode simulation with glibc”

• cannot be used to test the Linux kernel or any devices, and results are less representative of a real system since we are faking more

10.1. QEMU user mode getting started

Let’s run userland/c/command_line_arguments.c built with the Buildroot toolchain on QEMU user mode:

./build user-mode-qemu
./run \
  --userland userland/c/command_line_arguments.c \
  --cli-args='asdf "qw er"' \
;

Output:

/path/to/linux-kernel-module-cheat/out/userland/default/x86_64/c/command_line_arguments.out
asdf
qw er

./run --userland path resolution is analogous to that of ./run --baremetal.

./build user-mode-qemu first builds Buildroot, and then runs ./build-userland, which is further documented at: Section 1.8, “Userland setup”. It also builds QEMU. If you ahve already done a QEMU Buildroot setup previously, this will be very fast.

If you modify the userland programs, rebuild simply with:

./build-userland

To rebuild just QEMU userland if you hack it, use:

./build-qemu --mode userland

The:

--mode userland

is needed because QEMU has two separate executables:

• qemu-x86_64 for userland

• qemu-system-x86_64 for full system

10.1.1. User mode GDB

It’s nice when the obvious just works, right?

./run \
  --arch aarch64 \
  --gdb-wait \
  --userland userland/c/command_line_arguments.c \
  --cli-args 'asdf "qw er"' \
;

and on another shell:

./run-gdb \
  --arch aarch64 \
  --userland userland/c/command_line_arguments.c \
  main \
;

Or alternatively, if you are using tmux, do everything in one go with:

./run \
  --arch aarch64 \
  --gdb \
  --userland userland/c/command_line_arguments.c \
  --cli-args 'asdf "qw er"' \
;

To stop at the very first instruction of a freestanding program, just use --no-continue. A good example of this is shown at: [freestanding-programs].

10.2. User mode tests

Automatically run all userland tests that can be run in user mode simulation, and check that they exit with status 0:

./build --all-archs test-executables-userland
./test-executables --all-archs --all-emulators

Or just for QEMU:

./build --all-archs test-executables-userland-qemu
./test-executables --all-archs --emulator qemu

Source: test-executables

This script skips a manually configured list of tests, notably:

• tests that depend on a full running kernel and cannot be run in user mode simulation, e.g. those that rely on kernel modules

• tests that require user interaction

• tests that take perceptible amounts of time

• known bugs we didn’t have time to fix ;-)

Tests under userland/libs/ are only run if --package or --package-all are given as described at [userland-libs-directory].

The gem5 tests require building statically with build id static, see also: Section 10.7, “gem5 syscall emulation mode”. TODO automate this better.

See: [test-this-repo] for more useful testing tips.

10.3. User mode Buildroot executables

If you followed QEMU Buildroot setup, you can now run the executables created by Buildroot directly as:

./run \
  --userland "$(./getvar buildroot_target_dir)/bin/echo" \
  --cli-args='asdf' \
;

To easily explore the userland executable environment interactively, you can do:

./run \
  --arch aarch64 \
  --userland "$(./getvar --arch aarch64 buildroot_target_dir)/bin/sh" \
  --terminal \
;

or:

./run \
  --arch aarch64 \
  --userland "$(./getvar --arch aarch64 buildroot_target_dir)/bin/sh" \
  --cli-args='-c "uname -a && pwd"' \
;

Here is an interesting examples of this: Section 16.19.1, “Linux Test Project”

10.4. User mode simulation with glibc

At 125d14805f769104f93c510bedaa685a52ec025d we moved Buildroot from uClibc to glibc, and caused some user mode pain, which we document here.

10.4.1. FATAL: kernel too old failure in userland simulation

glibc has a check for kernel version, likely obtained from the uname syscall, and if the kernel is not new enough, it quits.

Both gem5 and QEMU however allow setting the reported uname version from the command line, which we do to always match our toolchain.

QEMU by default copies the host uname value, but we always override it in our scripts.

Determining the right number to use for the kernel version is of course highly non-trivial and would require an extensive userland test suite, which most emulators don’t have.

./run --arch aarch64 --kernel-version 4.18 --userland userland/posix/uname.c

Source: userland/posix/uname.c.

The QEMU source that does this is at: https://github.com/qemu/qemu/blob/v3.1.0/linux-user/syscall.c#L8931

Bibliography:

• https://stackoverflow.com/questions/48959349/how-to-solve-fatal-kernel-too-old-when-running-gem5-in-syscall-emulation-se-m

• https://stackoverflow.com/questions/53085048/how-to-compile-and-run-an-executable-in-gem5-syscall-emulation-mode-with-se-py/53085049#53085049

• https://gem5-review.googlesource.com/c/public/gem5/+/15855

The ID is just hardcoded on the source:

10.4.2. stack smashing detected when using glibc

For some reason QEMU / glibc x86_64 picks up the host libc, which breaks things.

Other archs work as they different host libc is skipped. User mode static executables also work.

We have worked around this with with https://bugs.launchpad.net/qemu/+bug/1701798/comments/12 from the thread: https://bugs.launchpad.net/qemu/+bug/1701798 by creating the file: rootfs_overlay/etc/ld.so.cache which is a symlink to a file that cannot exist: /dev/null/nonexistent.

Reproduction:

rm -f "$(./getvar buildroot_target_dir)/etc/ld.so.cache"
./run --userland userland/c/hello.c
./run --userland userland/c/hello.c --qemu-which host

Outcome:

*** stack smashing detected ***: <unknown> terminated
qemu: uncaught target signal 6 (Aborted) - core dumped

To get things working again, restore ld.so.cache with:

./build-buildroot

I’ve also tested on an Ubuntu 16.04 guest and the failure is different one:

qemu: uncaught target signal 4 (Illegal instruction) - core dumped

A non-QEMU-specific example of stack smashing is shown at: https://stackoverflow.com/questions/1345670/stack-smashing-detected/51897264#51897264

Tested at: 2e32389ebf1bedd89c682aa7b8fe42c3c0cf96e5 + 1.

10.5. User mode static executables

Example:

./build-userland \
  --arch aarch64 \
  --static \
;
./run \
  --arch aarch64 \
  --static \
  --userland userland/c/command_line_arguments.c \
  --cli-args 'asdf "qw er"' \
;

Running dynamically linked executables in QEMU requires pointing it to the root filesystem with the -L option so that it can find the dynamic linker and shared libraries, see also:

• https://stackoverflow.com/questions/54802670/using-dynamic-linker-with-qemu-arm/64551293#64551293

• https://stackoverflow.com/questions/khow-to-gdb-step-debug-a-dynamically-linked-executable-in-qemu-user-mode

We pass -L by default, so everything just works.

However, in case something goes wrong, you can also try statically linked executables, since this mechanism tends to be a bit more stable, for example:

• QEMU x86_64 guest on x86_64 host was failing with stack smashing detected when using glibc, but we found a workaround

• gem5 user only supported static executables in the past, as mentioned at: Section 10.7, “gem5 syscall emulation mode”

Running statically linked executables sometimes makes things break:

• User mode static executables with dynamic libraries

• TODO understand why:

  ./run --static --userland userland/c/file_write_read.c

  fails our assertion that the data was read back correctly:

  Assertion `strcmp(data, output) == 0' faile

10.5.1. User mode static executables with dynamic libraries

One limitation of static executables is that Buildroot mostly only builds dynamic versions of libraries (the libc is an exception).

So programs that rely on those libraries might not compile as GCC can’t find the .a version of the library.

For example, if we try to build [blas] statically:

./build-userland --package openblas --static -- userland/libs/openblas/hello.c

it fails with:

ld: cannot find -lopenblas

10.5.1.1. C++ static and pthreads

g++ and pthreads also causes issues:

• https://stackoverflow.com/questions/35116327/when-g-static-link-pthread-cause-segmentation-fault-why

• https://stackoverflow.com/questions/58848694/gcc-whole-archive-recipe-for-static-linking-to-pthread-stopped-working-in-rec

As a consequence, the following just hangs as of LKMC ca0403849e03844a328029d70c08556155dc1cd0 + 1 the example userland/cpp/atomic/std_atomic.cpp:

./run --userland userland/cpp/atomic/std_atomic.cpp --static

And before that, it used to fail with other randomly different errors, e.g.:

qemu-x86_64: /path/to/linux-kernel-module-cheat/submodules/qemu/accel/tcg/cpu-exec.c:700: cpu_exec: Assertion `!have_mmap_lock()' failed.
qemu-x86_64: /path/to/linux-kernel-module-cheat/submodules/qemu/accel/tcg/cpu-exec.c:700: cpu_exec: Assertion `!have_mmap_lock()' failed.

And a native Ubuntu 18.04 AMD64 run with static compilation segfaults.

As of LKMC f5d4998ff51a548ed3f5153aacb0411d22022058 the aarch64 error:

./run --arch aarch64 --userland userland/cpp/atomic/fail.cpp --static

is:

terminate called after throwing an instance of 'std::system_error'
  what():  Unknown error 16781344
qemu: uncaught target signal 6 (Aborted) - core dumped

The workaround:

-pthread -Wl,--whole-archive -lpthread -Wl,--no-whole-archive

fixes some of the problems, but not all TODO which were missing?, so we are just skipping those tests for now.

10.6. syscall emulation mode program stdin

The following work on both QEMU and gem5 as of LKMC 99d6bc6bc19d4c7f62b172643be95d9c43c26145 + 1. Interactive input:

./run --userland userland/c/getchar.c

Source: userland/c/getchar.c

A line of type should show:

enter a character:

and after pressing say a and Enter, we get:

you entered: a

Note however that due to QEMU user mode does not show stdout immediately we don’t really see the initial enter a character line.

Non-interactive input from a file by forwarding emulators stdin implicitly through our Python scripts:

printf a > f.tmp
./run --userland userland/c/getchar.c < f.tmp

Input from a file by explicitly requesting our scripts to use it via the Python API:

printf a > f.tmp
./run --emulator gem5 --userland userland/c/getchar.c --stdin-file f.tmp

This is especially useful when running tests that require stdin input.

10.7. gem5 syscall emulation mode

Less robust than QEMU’s, but still usable:

• https://stackoverflow.com/questions/48986597/when-should-you-use-full-system-fs-vs-syscall-emulation-se-with-userland-program

There are much more unimplemented syscalls in gem5 than in QEMU. Many of those are trivial to implement however.

So let’s just play with some static ones:

./build-userland --arch aarch64
./run \
  --arch aarch64 \
  --emulator gem5 \
  --userland userland/c/command_line_arguments.c \
  --cli-args 'asdf "qw er"' \
;

TODO: how to escape spaces on the command line arguments?

GDB step debug also works normally on gem5:

./run \
  --arch aarch64 \
  --emulator gem5 \
  --gdb-wait \
  --userland userland/c/command_line_arguments.c \
  --cli-args 'asdf "qw er"' \
;
./run-gdb \
  --arch aarch64 \
  --emulator gem5 \
  --userland userland/c/command_line_arguments.c \
  main \
;

10.7.1. gem5 dynamic linked executables in syscall emulation

Support for dynamic linking was added in November 2019:

• https://stackoverflow.com/questions/50542222/how-to-run-a-dynamically-linked-executable-syscall-emulation-mode-se-py-in-gem5/50696098#50696098

• https://stackoverflow.com/questions/64547306/cannot-open-lib-ld-linux-aarch64-so-1-in-qemu-or-gem5/64551313#64551313

Note that as shown at [benchmark-emulators-on-userland-executables], the dynamic version runs 200x more instructions, which might have an impact on smaller simulations in detailed CPUs.

10.7.2. gem5 syscall emulation exit status

As of gem5 7fa4c946386e7207ad5859e8ade0bbfc14000d91, the crappy se.py script does not forward the exit status of syscall emulation mode, you can test it with:

./run --dry-run --emulator gem5 --userland userland/c/false.c

Source: userland/c/false.c.

Then manually run the generated gem5 CLI, and do:

echo $?

and the output is always 0.

Instead, it just outputs a message to stdout just like for m5 fail:

Simulated exit code not 0! Exit code is 1

which we parse in run and then exit with the correct result ourselves…​

Related thread: https://stackoverflow.com/questions/56032347/is-there-a-way-to-identify-if-gem5-run-got-over-successfully

10.7.3. gem5 syscall emulation mode syscall tracing

Since gem5 has to implement syscalls itself in syscall emulation mode, it can of course clearly see which syscalls are being made, and we can log them for debug purposes with gem5 tracing, e.g.:

./run \
  --emulator gem5 \
  --userland userland/arch/x86_64/freestanding/linux/hello.S \
  --trace-stdout \
  --trace ExecAll,SyscallBase,SyscallVerbose \
;

the trace as of f2eeceb1cde13a5ff740727526bf916b356cee38 + 1 contains:

      0: system.cpu A0 T0 : @asm_main_after_prologue    : mov   rdi, 0x1
      0: system.cpu A0 T0 : @asm_main_after_prologue.0  :   MOV_R_I : limm   rax, 0x1 : IntAlu :  D=0x0000000000000001  flags=(IsInteger|IsMicroop|IsLastMicroop|IsFirstMicroop)
   1000: system.cpu A0 T0 : @asm_main_after_prologue+7    : mov rdi, 0x1
   1000: system.cpu A0 T0 : @asm_main_after_prologue+7.0  :   MOV_R_I : limm   rdi, 0x1 : IntAlu :  D=0x0000000000000001  flags=(IsInteger|IsMicroop|IsLastMicroop|IsFirstMicroop)
   2000: system.cpu A0 T0 : @asm_main_after_prologue+14    : lea        rsi, DS:[rip + 0x19]
   2000: system.cpu A0 T0 : @asm_main_after_prologue+14.0  :   LEA_R_P : rdip   t7, %ctrl153,  : IntAlu :  D=0x000000000040008d  flags=(IsInteger|IsMicroop|IsDelayedCommit|IsFirstMicroop)
   2500: system.cpu A0 T0 : @asm_main_after_prologue+14.1  :   LEA_R_P : lea   rsi, DS:[t7 + 0x19] : IntAlu :  D=0x00000000004000a6  flags=(IsInteger|IsMicroop|IsLastMicroop)
   3500: system.cpu A0 T0 : @asm_main_after_prologue+21    : mov        rdi, 0x6
   3500: system.cpu A0 T0 : @asm_main_after_prologue+21.0  :   MOV_R_I : limm   rdx, 0x6 : IntAlu :  D=0x0000000000000006  flags=(IsInteger|IsMicroop|IsLastMicroop|IsFirstMicroop)
   4000: system.cpu: T0 : syscall write called w/arguments 1, 4194470, 6, 0, 0, 0
hello
   4000: system.cpu: T0 : syscall write returns 6
   4000: system.cpu A0 T0 : @asm_main_after_prologue+28    :   syscall    eax           : IntAlu :   flags=(IsInteger|IsSerializeAfter|IsNonSpeculative|IsSyscall)
   5000: system.cpu A0 T0 : @asm_main_after_prologue+30    : mov        rdi, 0x3c
   5000: system.cpu A0 T0 : @asm_main_after_prologue+30.0  :   MOV_R_I : limm   rax, 0x3c : IntAlu :  D=0x000000000000003c  flags=(IsInteger|IsMicroop|IsLastMicroop|IsFirstMicroop)
   6000: system.cpu A0 T0 : @asm_main_after_prologue+37    : mov        rdi, 0
   6000: system.cpu A0 T0 : @asm_main_after_prologue+37.0  :   MOV_R_I : limm   rdi, 0  : IntAlu :  D=0x0000000000000000  flags=(IsInteger|IsMicroop|IsLastMicroop|IsFirstMicroop)
   6500: system.cpu: T0 : syscall exit called w/arguments 0, 4194470, 6, 0, 0, 0
   6500: system.cpu: T0 : syscall exit returns 0
   6500: system.cpu A0 T0 : @asm_main_after_prologue+44    :   syscall    eax           : IntAlu :   flags=(IsInteger|IsSerializeAfter|IsNonSpeculative|IsSyscall)

so we see that two syscall lines were added for each syscall, showing the syscall inputs and exit status, just like a mini strace!

10.7.4. gem5 syscall emulation multithreading

gem5 user mode multithreading has been particularly flaky compared to QEMU’s, but work is being put into improving it.

In gem5 syscall simulation, the fork syscall checks if there is a free CPU, and if there is a free one, the new threads runs on that CPU.

Otherwise, the fork call, and therefore higher level interfaces to fork such as pthread_create also fail and return a failure return status in the guest.

For example, if we use just one CPU for userland/posix/pthread_self.c which spawns one thread besides main:

./run --cpus 1 --emulator gem5 --userland userland/posix/pthread_self.c --cli-args 1

fails with this error message coming from the guest stderr:

pthread_create: Resource temporarily unavailable

It works however if we add on extra CPU:

./run --cpus 2 --emulator gem5 --userland userland/posix/pthread_self.c --cli-args 1

Once threads exit, their CPU is freed and becomes available for new fork calls: For example, the following run spawns a thread, joins it, and then spawns again, and 2 CPUs are enough:

./run --cpus 2 --emulator gem5 --userland userland/posix/pthread_self.c --cli-args '1 2'

because at each point in time, only up to two threads are running.

gem5 syscall emulation does show the expected number of cores when queried, e.g.:

./run --cpus 1 --userland userland/cpp/thread_hardware_concurrency.cpp --emulator gem5
./run --cpus 2 --userland userland/cpp/thread_hardware_concurrency.cpp --emulator gem5

outputs 1 and 2 respectively.

This can also be clearly by running sched_getcpu:

./run \
  --arch aarch64 \
  --cli-args  4 \
  --cpus 8 \
  --emulator gem5 \
  --userland userland/linux/sched_getcpu.c \
;

which necessarily produces an output containing the CPU numbers from 1 to 4 and no higher:

1
3
4
2

TODO why does the 2 come at the end here? Would be good to do a detailed assembly run analysis.

10.7.5. gem5 syscall emulation multiple executables

gem5 syscall emulation has the nice feature of allowing you to run multiple executables "at once".

Each executable starts running on the next free core much as if it had been forked right at the start of simulation: gem5 syscall emulation multithreading.

This can be useful to quickly create deterministic multi-CPU workload.

se.py --cmd takes a semicolon separated list, so we could do which LKMC exposes this by taking --userland multiple times as in:

./run \
  --arch aarch64 \
  --cpus 2 \
  --emulator gem5 \
  --userland userland/posix/getpid.c \
  --userland userland/posix/getpid.c \
;

We need at least one CPU per executable, just like when forking new processes.

The outcome of this is that we see two different pid messages printed to stdout:

pid=101
pid=100

since from [gem5-process] we can see that se.py sets up one different PID per executable starting at 100:

    workloads = options.cmd.split(';')
    idx = 0
    for wrkld in workloads:
        process = Process(pid = 100 + idx)

We can also see that these processes are running concurrently with gem5 tracing by hacking:

  --debug-flags ExecAll \
  --debug-file cout \

which starts with:

      0: system.cpu1: A0 T0 : @__end__+274873647040    :   add   x0, sp, #0         : IntAlu :  D=0x0000007ffffefde0  flags=(IsInteger)
      0: system.cpu0: A0 T0 : @__end__+274873647040    :   add   x0, sp, #0         : IntAlu :  D=0x0000007ffffefde0  flags=(IsInteger)
    500: system.cpu0: A0 T0 : @__end__+274873647044    :   bl   <__end__+274873649648> : IntAlu :  D=0x0000004000001008  flags=(IsInteger|IsControl|IsDirectControl|IsUncondControl|IsCall)
    500: system.cpu1: A0 T0 : @__end__+274873647044    :   bl   <__end__+274873649648> : IntAlu :  D=0x0000004000001008  flags=(IsInteger|IsControl|IsDirectControl|IsUncondControl|IsCall)

and therefore shows one instruction running on each CPU for each process at the same time.

10.7.5.1. gem5 syscall emulation --smt

gem5 b1623cb2087873f64197e503ab8894b5e4d4c7b4 syscall emulation has an --smt option presumably for [hardware-threads] but it has been neglected forever it seems: https://github.com/cirosantilli/linux-kernel-module-cheat/issues/104

If we start from the manually hacked working command from gem5 syscall emulation multiple executables and try to add:

--cpu 1 --cpu-type Derivo3CPU --caches

We choose DerivO3CPU because of the se.py assert:

example/se.py:115:        assert(options.cpu_type == "DerivO3CPU")

But then that fails with:

gem5.opt: /path/to/linux-kernel-module-cheat/out/gem5/master3/build/ARM/cpu/o3/cpu.cc:205: FullO3CPU<Impl>::FullO3CPU(DerivO3CPUParams*) [with Impl = O3CPUImpl]: Assertion `params->numPhysVecPredRegs >= numThreads * TheISA::NumVecPredRegs' failed.
Program aborted at tick 0

10.8. QEMU user mode quirks

10.8.1. QEMU user mode does not show stdout immediately

At 8d8307ac0710164701f6e14c99a69ee172ccbb70 + 1, I noticed that if you run userland/posix/count.c:

./run --userland userland/posix/count_to.c --cli-args 3

it first waits for 3 seconds, then the program exits, and then it dumps all the stdout at once, instead of counting once every second as expected.

The same can be reproduced by copying the raw QEMU command and piping it through tee, so I don’t think it is a bug in our setup:

/path/to/linux-kernel-module-cheat/out/qemu/default/x86_64-linux-user/qemu-x86_64 \
  -L /path/to/linux-kernel-module-cheat/out/buildroot/build/default/x86_64/target \
  /path/to/linux-kernel-module-cheat/out/userland/default/x86_64/posix/count.out \
  3 \
| tee

TODO: investigate further and then possibly post on QEMU mailing list.

10.8.1.1. QEMU user mode does not show errors

Similarly to QEMU user mode does not show stdout immediately, QEMU error messages do not show at all through pipes.

In particular, it does not say anything if you pass it a non-existing executable:

qemu-x86_64 asdf | cat

So we just check ourselves manually

11. Kernel module utilities

11.1. insmod

Provided by BusyBox:

./run --eval-after 'insmod hello.ko'

11.2. myinsmod

If you are feeling raw, you can insert and remove modules with our own minimal module inserter and remover!

# init_module
./linux/myinsmod.out hello.ko
# finit_module
./linux/myinsmod.out hello.ko "" 1
./linux/myrmmod.out hello

which teaches you how it is done from C code.

Source:

• userland/linux/myinsmod.c

• userland/linux/myrmmod.c

The Linux kernel offers two system calls for module insertion:

• init_module

• finit_module

and:

man init_module

documents that:

The finit_module() system call is like init_module(), but reads the module to be loaded from the file descriptor fd. It is useful when the authenticity of a kernel module can be determined from its location in the filesystem; in cases where that is possible, the overhead of using cryptographically signed modules to determine the authenticity of a module can be avoided. The param_values argument is as for init_module().

finit is newer and was added only in v3.8. More rationale: https://lwn.net/Articles/519010/

Bibliography: https://stackoverflow.com/questions/5947286/how-to-load-linux-kernel-modules-from-c-code

11.3. modprobe

Implemented as a BusyBox applet by default: https://git.busybox.net/busybox/tree/modutils/modprobe.c?h=1_29_stable

modprobe searches for modules installed under:

ls /lib/modules/<kernel_version>

and specified in the modules.order file.

This is the default install path for CONFIG_SOME_MOD=m modules built with make modules_install in the Linux kernel tree, with root path given by INSTALL_MOD_PATH, and therefore canonical in that sense.

Currently, there are only two kinds of kernel modules that you can try out with modprobe:

• modules built with Buildroot, see: [kernel-modules-buildroot-package]

• modules built from the kernel tree itself, see: Section 16.11.2, “dummy-irq”

We are not installing out custom ./build-modules modules there, because:

• we don’t know the right way. Why is there no install or install_modules target for kernel modules?

  This can of course be solved by running Buildroot in verbose mode, and copying whatever it is doing, initial exploration at: https://stackoverflow.com/questions/22783793/how-to-install-kernel-modules-from-source-code-error-while-make-process/53169078#53169078

• we would have to think how to not have to include the kernel modules twice in the root filesystem, but still have 9P working for fast development as described at: Section 1.2.2.2, “Your first kernel module hack”

11.4. kmod

The more "reference" kernel.org implementation of lsmod, insmod, rmmod, etc.: https://git.kernel.org/pub/scm/utils/kernel/kmod/kmod.git

Default implementation on desktop distros such as Ubuntu 16.04, where e.g.:

ls -l /bin/lsmod

gives:

lrwxrwxrwx 1 root root 4 Jul 25 15:35 /bin/lsmod -> kmod

and:

dpkg -l | grep -Ei

contains:

ii  kmod                                        22-1ubuntu5                                         amd64        tools for managing Linux kernel modules

BusyBox also implements its own version of those executables, see e.g. modprobe. Here we will only describe features that differ from kmod to the BusyBox implementation.

11.4.1. module-init-tools

Name of a predecessor set of tools.

11.4.2. kmod modprobe

kmod’s modprobe can also load modules under different names to avoid conflicts, e.g.:

sudo modprobe vmhgfs -o vm_hgfs

12. Filesystems

12.1. OverlayFS

OverlayFS is a filesystem merged in the Linux kernel in 3.18.

As the name suggests, OverlayFS allows you to merge multiple directories into one. The following minimal runnable examples should give you an intuition on how it works:

• https://askubuntu.com/questions/109413/how-do-i-use-overlayfs/1075564#1075564

• https://stackoverflow.com/questions/31044982/how-to-use-multiple-lower-layers-in-overlayfs/52792397#52792397

We are very interested in this filesystem because we are looking for a way to make host cross compiled executables appear on the guest root / without reboot.

This would have several advantages:

• makes it faster to test modified guest programs

  • not rebooting is fundamental for gem5, where the reboot is very costly.

  • no need to regenerate the root filesystem at all and reboot

  • overcomes the check_bin_arch problem as shown at: [rpath]

• we could keep the base root filesystem very small, which implies:

  • less host disk usage, no need to copy the entire ./getvar out_rootfs_overlay_dir to the image again

  • no need to worry about [br2-target-rootfs-ext2-size]

We can already make host files appear on the guest with 9P, but they appear on a subdirectory instead of the root.

If they would appear on the root instead, that would be even more awesome, because you would just use the exact same paths relative to the root transparently.

For example, we wouldn’t have to mess around with variables such as PATH and LD_LIBRARY_PATH.

The idea is to:

• 9P mount our overlay directory ./getvar out_rootfs_overlay_dir on the guest, which we already do at /mnt/9p/out_rootfs_overlay

• then create an overlay with that directory and the root, and chroot into it.

  I was unable to mount directly to / avoid the chroot: https://stackoverflow.com/questions/41119656/how-can-i-overlayfs-the-root-filesystem-on-linux https://unix.stackexchange.com/questions/316018/how-to-use-overlayfs-to-protect-the-root-filesystem ** https://unix.stackexchange.com/questions/420646/mount-root-as-overlayfs

We already have a prototype of this running from fstab on guest at /mnt/overlay, but it has the following shortcomings:

• changes to underlying filesystems are not visible on the overlay unless you remount with mount -r remount /mnt/overlay, as mentioned on the kernel docs:

  Changes to the underlying filesystems while part of a mounted overlay
  filesystem are not allowed.  If the underlying filesystem is changed,
  the behavior of the overlay is undefined, though it will not result in
  a crash or deadlock.

  This makes everything very inconvenient if you are inside chroot action. You would have to leave chroot, remount, then come back.

• the overlay does not contain sub-filesystems, e.g. /proc. We would have to re-mount them. But should be doable with some automation.

Even more awesome than chroot would be to pivot_root, but I couldn’t get that working either:

• https://stackoverflow.com/questions/28015688/pivot-root-device-or-resource-busy

• https://unix.stackexchange.com/questions/179788/pivot-root-device-or-resource-busy

12.2. Secondary disk

A simpler and possibly less overhead alternative to 9P would be to generate a secondary disk image with the benchmark you want to rebuild.

Then you can umount and re-mount on guest without reboot.

To build the secondary disk image run build-disk2:

./build-disk2

This will put the entire [out-rootfs-overlay-dir] into a squashfs filesystem.

Then, if that filesystem is present, ./run will automatically pass it as the second disk on the command line.

For example, from inside QEMU, you can mount that disk with:

mkdir /mnt/vdb
mount /dev/vdb /mnt/vdb
/mnt/vdb/lkmc/c/hello.out

To update the secondary disk while a simulation is running to avoid rebooting, first unmount in the guest:

umount /mnt/vdb

and then on the host:

# Edit the file.
vim userland/c/hello.c
./build-userland
./build-disk2

and now you can re-run the updated version of the executable on the guest after remounting it.

gem5 fs.py support for multiple disks is discussed at: https://stackoverflow.com/questions/50862906/how-to-attach-multiple-disk-images-in-a-simulation-with-gem5-fs-py/51037661#51037661

13. Graphics

Both QEMU and gem5 are capable of outputting graphics to the screen, and taking mouse and keyboard input.

https://unix.stackexchange.com/questions/307390/what-is-the-difference-between-ttys0-ttyusb0-and-ttyama0-in-linux

13.1. QEMU text mode

Text mode is the default mode for QEMU.

The opposite of text mode is QEMU graphic mode

In text mode, we just show the serial console directly on the current terminal, without opening a QEMU GUI window.

You cannot see any graphics from text mode, but text operations in this mode, including:

• scrolling up: Section 13.2.1, “Scroll up in graphic mode”

• copy paste to and from the terminal

making this a good default, unless you really need to use with graphics.

Text mode works by sending the terminal character by character to a serial device.

This is different from a display screen, where each character is a bunch of pixels, and it would be much harder to convert that into actual terminal text.

For more details, see:

• https://unix.stackexchange.com/questions/307390/what-is-the-difference-between-ttys0-ttyusb0-and-ttyama0-in-linux

• TTY

Note that you can still see an image even in text mode with the VNC:

./run --vnc

and on another terminal:

./vnc

but there is not terminal on the VNC window, just the CONFIG_LOGO penguin.

13.1.1. Quit QEMU from text mode

https://superuser.com/questions/1087859/how-to-quit-the-qemu-monitor-when-not-using-a-gui

However, our QEMU setup captures Ctrl + C and other common signals and sends them to the guest, which makes it hard to quit QEMU for the first time since there is no GUI either.

The simplest way to quit QEMU, is to do:

Ctrl-A X

Alternative methods include:

• quit command on the QEMU monitor

• pkill qemu

13.2. QEMU graphic mode

Enable graphic mode with:

./run --graphic

Outcome: you see a penguin due to CONFIG_LOGO.

For a more exciting GUI experience, see: Section 13.4, “X11 Buildroot”

Text mode is the default due to the following considerable advantages:

• copy and paste commands and stdout output to / from host

• get full panic traces when you start making the kernel crash :-) See also: https://unix.stackexchange.com/questions/208260/how-to-scroll-up-after-a-kernel-panic

• have a large scroll buffer, and be able to search it, e.g. by using tmux on host

• one less window floating around to think about in addition to your shell :-)

• graphics mode has only been properly tested on x86_64.

Text mode has the following limitations over graphics mode:

• you can’t see graphics such as those produced by X11 Buildroot

• very early kernel messages such as early console in extract_kernel only show on the GUI, since at such early stages, not even the serial has been setup.

x86_64 has a VGA device enabled by default, as can be seen as:

./qemu-monitor info qtree

and the Linux kernel picks it up through the fbdev graphics system as can be seen from:

cat /dev/urandom > /dev/fb0

flooding the screen with colors. See also: https://superuser.com/questions/223094/how-do-i-know-if-i-have-kms-enabled

13.2.1. Scroll up in graphic mode

Scroll up in QEMU graphic mode:

Shift-PgUp

but I never managed to increase that buffer:

• https://askubuntu.com/questions/709697/how-to-increase-scrollback-lines-in-ubuntu14-04-2-server-edition

• https://unix.stackexchange.com/questions/346018/how-to-increase-the-scrollback-buffer-size-for-tty

The superior alternative is to use text mode and GNU screen or tmux.

13.2.2. QEMU Graphic mode arm

13.2.2.1. QEMU graphic mode arm terminal

TODO: on arm, we see the penguin and some boot messages, but don’t get a shell at then end:

./run --arch aarch64 --graphic

I think it does not work because the graphic window is DRM only, i.e.:

cat /dev/urandom > /dev/fb0

fails with:

cat: write error: No space left on device

and has no effect, and the Linux kernel does not appear to have a built-in DRM console as it does for fbdev with fbcon.

There is however one out-of-tree implementation: kmscon.

13.2.2.2. QEMU graphic mode arm terminal implementation

arm and aarch64 rely on the QEMU CLI option:

-device virtio-gpu-pci

and the kernel config options:

CONFIG_DRM=y
CONFIG_DRM_VIRTIO_GPU=y

Unlike x86, arm and aarch64 don’t have a display device attached by default, thus the need for virtio-gpu-pci.

See also https://wiki.qemu.org/Documentation/Platforms/ARM (recently edited and corrected by yours truly…​ :-)).

13.2.2.3. QEMU graphic mode arm VGA

TODO: how to use VGA on ARM? https://stackoverflow.com/questions/20811203/how-can-i-output-to-vga-through-qemu-arm Tried:

-device VGA

But https://github.com/qemu/qemu/blob/v2.12.0/docs/config/mach-virt-graphical.cfg#L264 says:

# We use virtio-gpu because the legacy VGA framebuffer is
# very troublesome on aarch64, and virtio-gpu is the only
# video device that doesn't implement it.

so maybe it is not possible?

13.3. gem5 graphic mode

gem5 does not have a "text mode", since it cannot redirect the Linux terminal to same host terminal where the executable is running: you are always forced to connect to the terminal with gem-shell.

TODO could not get it working on x86_64, only ARM.

Overview: https://stackoverflow.com/questions/50364863/how-to-get-graphical-gui-output-and-user-touch-keyboard-mouse-input-in-a-ful/50364864#50364864

More concretely, first build the kernel with the gem5 arm Linux kernel patches, and then run:

./build-linux \
  --arch arm \
  --custom-config-file-gem5 \
  --linux-build-id gem5-v4.15 \
;
./run --arch arm --emulator gem5 --linux-build-id gem5-v4.15

and then on another shell:

vinagre localhost:5900

The CONFIG_LOGO penguin only appears after several seconds, together with kernel messages of type:

[    0.152755] [drm] found ARM HDLCD version r0p0
[    0.152790] hdlcd 2b000000.hdlcd: bound virt-encoder (ops 0x80935f94)
[    0.152795] [drm] Supports vblank timestamp caching Rev 2 (21.10.2013).
[    0.152799] [drm] No driver support for vblank timestamp query.
[    0.215179] Console: switching to colour frame buffer device 240x67
[    0.230389] hdlcd 2b000000.hdlcd: fb0:  frame buffer device
[    0.230509] [drm] Initialized hdlcd 1.0.0 20151021 for 2b000000.hdlcd on minor 0

The port 5900 is incremented by one if you already have something running on that port, gem5 stdout tells us the right port on stdout as:

system.vncserver: Listening for connections on port 5900

and when we connect it shows a message:

info: VNC client attached

Alternatively, you can also dump each new frame to an image file with --frame-capture:

./run \
  --arch arm \
  --emulator gem5 \
  --linux-build-id gem5-v4.15 \
  -- --frame-capture \
;

This creates on compressed PNG whenever the screen image changes inside the m5out directory with filename of type:

frames_system.vncserver/fb.<frame-index>.<timestamp>.png.gz

It is fun to see how we get one new frame whenever the white underscore cursor appears and reappears under the penguin!

The last frame is always available uncompressed at: system.framebuffer.png.

TODO kmscube failed on aarch64 with:

kmscube[706]: unhandled level 2 translation fault (11) at 0x00000000, esr 0x92000006, in libgbm.so.1.0.0[7fbf6a6000+e000]

Tested on: 38fd6153d965ba20145f53dc1bb3ba34b336bde9

13.3.1. Graphic mode gem5 aarch64

For aarch64 we also need to configure the kernel with linux_config/display:

git -C "$(./getvar linux_source_dir)" fetch https://gem5.googlesource.com/arm/linux gem5/v4.15:gem5/v4.15
git -C "$(./getvar linux_source_dir)" checkout gem5/v4.15
./build-linux \
  --arch aarch64 \
  --config-fragment linux_config/display \
  --custom-config-file-gem5 \
  --linux-build-id gem5-v4.15 \
;
git -C "$(./getvar linux_source_dir)" checkout -
./run --arch aarch64 --emulator gem5 --linux-build-id gem5-v4.15

This is because the gem5 aarch64 defconfig does not enable HDLCD like the 32 bit one arm one for some reason.

13.3.2. gem5 graphic mode DP650

TODO get working. There is an unmerged patchset at: https://gem5-review.googlesource.com/c/public/gem5/+/11036/1

The DP650 is a newer display hardware than HDLCD. TODO is its interface publicly documented anywhere? Since it has a gem5 model and in-tree Linux kernel support, that information cannot be secret?

The key option to enable support in Linux is DRM_MALI_DISPLAY=y which we enable at linux_config/display.

Build the kernel exactly as for Graphic mode gem5 aarch64 and then run with:

./run --arch aarch64 --dp650 --emulator gem5 --linux-build-id gem5-v4.15

13.3.3. gem5 graphic mode internals

We cannot use mainline Linux because the gem5 arm Linux kernel patches are required at least to provide the CONFIG_DRM_VIRT_ENCODER option.

gem5 emulates the HDLCD ARM Holdings hardware for arm and aarch64.

The kernel uses HDLCD to implement the DRM interface, the required kernel config options are present at: linux_config/display.

TODO: minimize out the --custom-config-file. If we just remove it on arm: it does not work with a failing dmesg:

[    0.066208] [drm] found ARM HDLCD version r0p0
[    0.066241] hdlcd 2b000000.hdlcd: bound virt-encoder (ops drm_vencoder_ops)
[    0.066247] [drm] Supports vblank timestamp caching Rev 2 (21.10.2013).
[    0.066252] [drm] No driver support for vblank timestamp query.
[    0.066276] hdlcd 2b000000.hdlcd: Cannot do DMA to address 0x0000000000000000
[    0.066281] swiotlb: coherent allocation failed for device 2b000000.hdlcd size=8294400
[    0.066288] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.15.0 #1
[    0.066293] Hardware name: V2P-AARCH64 (DT)
[    0.066296] Call trace:
[    0.066301]  dump_backtrace+0x0/0x1b0
[    0.066306]  show_stack+0x24/0x30
[    0.066311]  dump_stack+0xb8/0xf0
[    0.066316]  swiotlb_alloc_coherent+0x17c/0x190
[    0.066321]  __dma_alloc+0x68/0x160
[    0.066325]  drm_gem_cma_create+0x98/0x120
[    0.066330]  drm_fbdev_cma_create+0x74/0x2e0
[    0.066335]  __drm_fb_helper_initial_config_and_unlock+0x1d8/0x3a0
[    0.066341]  drm_fb_helper_initial_config+0x4c/0x58
[    0.066347]  drm_fbdev_cma_init_with_funcs+0x98/0x148
[    0.066352]  drm_fbdev_cma_init+0x40/0x50
[    0.066357]  hdlcd_drm_bind+0x220/0x428
[    0.066362]  try_to_bring_up_master+0x21c/0x2b8
[    0.066367]  component_master_add_with_match+0xa8/0xf0
[    0.066372]  hdlcd_probe+0x60/0x78
[    0.066377]  platform_drv_probe+0x60/0xc8
[    0.066382]  driver_probe_device+0x30c/0x478
[    0.066388]  __driver_attach+0x10c/0x128
[    0.066393]  bus_for_each_dev+0x70/0xb0
[    0.066398]  driver_attach+0x30/0x40
[    0.066402]  bus_add_driver+0x1d0/0x298
[    0.066408]  driver_register+0x68/0x100
[    0.066413]  __platform_driver_register+0x54/0x60
[    0.066418]  hdlcd_platform_driver_init+0x20/0x28
[    0.066424]  do_one_initcall+0x44/0x130
[    0.066428]  kernel_init_freeable+0x13c/0x1d8
[    0.066433]  kernel_init+0x18/0x108
[    0.066438]  ret_from_fork+0x10/0x1c
[    0.066444] hdlcd 2b000000.hdlcd: Failed to set initial hw configuration.
[    0.066470] hdlcd 2b000000.hdlcd: master bind failed: -12
[    0.066477] hdlcd: probe of 2b000000.hdlcd failed with error -12

So what other options are missing from gem5_defconfig? It would be cool to minimize it out to better understand the options.

13.4. X11 Buildroot

Once you’ve seen the CONFIG_LOGO penguin as a sanity check, you can try to go for a cooler X11 Buildroot setup.

Build and run:

./build-buildroot --config-fragment buildroot_config/x11
./run --graphic

Inside QEMU:

startx

And then from the GUI you can start exciting graphical programs such as:

xcalc
xeyes

Outcome: Figure 1, “X11 Buildroot graphical user interface screenshot”

Figure 1. X11 Buildroot graphical user interface screenshot

We don’t build X11 by default because it takes a considerable amount of time (about 20%), and is not expected to be used by most users: you need to pass the -x flag to enable it.

More details: https://unix.stackexchange.com/questions/70931/how-to-install-x11-on-my-own-linux-buildroot-system/306116#306116

Not sure how well that graphics stack represents real systems, but if it does it would be a good way to understand how it works.

To x11 packages have an xserver prefix as in:

./build-buildroot --config-fragment buildroot_config/x11 -- xserver_xorg-server-reconfigure

the easiest way to find them out is to just list "$(./getvar buildroot_build_build_dir)/x*.

TODO as of: c2696c978d6ca88e8b8599c92b1beeda80eb62b2 I noticed that startx leads to a BUG_ON:

[    2.809104] WARNING: CPU: 0 PID: 51 at drivers/gpu/drm/ttm/ttm_bo_vm.c:304 ttm_bo_vm_open+0x37/0x40

13.4.1. X11 Buildroot mouse not moving

TODO 9076c1d9bcc13b6efdb8ef502274f846d8d4e6a1 I’m 100% sure that it was working before, but I didn’t run it forever, and it stopped working at some point. Needs bisection, on whatever commit last touched x11 stuff.

• https://askubuntu.com/questions/730891/how-can-i-get-a-mouse-cursor-in-qemu

• https://stackoverflow.com/questions/19665412/mouse-and-keyboard-not-working-in-qemu-emulator

-show-cursor did not help, I just get to see the host cursor, but the guest cursor still does not move.

Doing:

watch -n 1 grep i8042 /proc/interrupts

shows that interrupts do happen when mouse and keyboard presses are done, so I expect that it is some wrong either with:

• QEMU. Same behaviour if I try the host’s QEMU 2.10.1 however.

• X11 configuration. We do have BR2_PACKAGE_XDRIVER_XF86_INPUT_MOUSE=y.

/var/log/Xorg.0.log contains the following interesting lines:

[    27.549] (II) LoadModule: "mouse"
[    27.549] (II) Loading /usr/lib/xorg/modules/input/mouse_drv.so
[    27.590] (EE) <default pointer>: Cannot find which device to use.
[    27.590] (EE) <default pointer>: cannot open input device
[    27.590] (EE) PreInit returned 2 for "<default pointer>"
[    27.590] (II) UnloadModule: "mouse"

The file /dev/inputs/mice does not exist.

Note that our current link:kernel_confi_fragment sets:

# CONFIG_INPUT_MOUSE is not set
# CONFIG_INPUT_MOUSEDEV_PSAUX is not set

for gem5, so you might want to remove those lines to debug this.

13.4.2. X11 Buildroot ARM

On ARM, startx hangs at a message:

vgaarb: this pci device is not a vga device

and nothing shows on the screen, and:

grep EE /var/log/Xorg.0.log

says:

(EE) Failed to load module "modesetting" (module does not exist, 0)

A friend told me this but I haven’t tried it yet:

• xf86-video-modesetting is likely the missing ingredient, but it does not seem possible to activate it from Buildroot currently without patching things.

• xf86-video-fbdev should work as well, but we need to make sure fbdev is enabled, and maybe add some line to the Xorg.conf

14. Networking

14.1. Enable networking

We disable networking by default because it starts an userland process, and we want to keep the number of userland processes to a minimum to make the system more understandable as explained at: [resource-tradeoff-guidelines]

To enable networking on Buildroot, simply run:

ifup -a

That command goes over all (-a) the interfaces in /etc/network/interfaces and brings them up.

Then test it with:

wget google.com
cat index.html

Disable networking with:

ifdown -a

To enable networking by default after boot, use the methods documented at Run command at the end of BusyBox init.

14.2. ping

ping does not work within QEMU by default, e.g.:

ping google.com

hangs after printing the header:

PING google.com (216.58.204.46): 56 data bytes

Here Ciro describes how to get it working: https://unix.stackexchange.com/questions/473448/how-to-ping-from-the-qemu-guest-to-an-external-url

Further bibliography: https://superuser.com/questions/787400/qemu-user-mode-networking-doesnt-work

14.3. Guest host networking

In this section we discuss how to interact between the guest and the host through networking.

First ensure that you can access the external network since that is easier to get working, see: Section 14, “Networking”.

14.3.1. Host to guest networking

14.3.1.1. nc host to guest

With nc we can create the most minimal example possible as a sanity check.

On guest run:

nc -l -p 45455

Then on host run:

echo asdf | nc localhost 45455

asdf appears on the guest.

This uses:

• BusyBox' nc utility, which is enabled with CONFIG_NC=y

• nc from the netcat-openbsd package on an Ubuntu 18.04 host

Only this specific port works by default since we have forwarded it on the QEMU command line.

We us this exact procedure to connect to gdbserver.

14.3.1.2. ssh into guest

Not enabled by default due to the build / runtime overhead. To enable, build with:

./build-buildroot --config 'BR2_PACKAGE_OPENSSH=y'

Then inside the guest turn on sshd:

./sshd.sh

Source: rootfs_overlay/lkmc/sshd.sh

And finally on host:

ssh root@localhost -p 45456

Bibliography: https://unix.stackexchange.com/questions/124681/how-to-ssh-from-host-to-guest-using-qemu/307557#307557

14.3.1.3. gem5 host to guest networking

Could not do port forwarding from host to guest, and therefore could not use gdbserver: https://stackoverflow.com/questions/48941494/how-to-do-port-forwarding-from-guest-to-host-in-gem5

14.3.2. Guest to host networking

First Enable networking.

Then in the host, start a server:

python -m SimpleHTTPServer 8000

And then in the guest, find the IP we need to hit with:

ip rounte

which gives:

default via 10.0.2.2 dev eth0
10.0.2.0/24 dev eth0 scope link  src 10.0.2.15

so we use in the guest:

wget 10.0.2.2:8000

Bibliography:

• https://serverfault.com/questions/769874/how-to-forward-a-port-from-guest-to-host-in-qemu-kvm/951835#951835

• https://unix.stackexchange.com/questions/78953/qemu-how-to-ping-host-network/547698#547698

14.4. 9P

The 9p protocol allows the guest to mount a host directory.

Both QEMU and gem5 9P support 9P.

14.4.1. 9P vs NFS

All of 9P and NFS (and sshfs) allow sharing directories between guest and host.

Advantages of 9P

• requires sudo on the host to mount

• we could share a guest directory to the host, but this would require running a server on the guest, which adds simulation overhead

  Furthermore, this would be inconvenient, since what we usually want to do is to share host cross built files with the guest, and to do that we would have to copy the files over after the guest starts the server.

• QEMU implements 9P natively, which makes it very stable and convenient, and must mean it is a simpler protocol than NFS as one would expect.

  This is not the case for gem5 7bfb7f3a43f382eb49853f47b140bfd6caad0fb8 unfortunately, which relies on the diod host daemon, although it is not unfeasible that future versions could implement it natively as well.

Advantages of NFS:

• way more widely used and therefore stable and available, not to mention that it also works on real hardware.

• the name does not start with a digit, which is an invalid identifier in all programming languages known to man. Who in their right mind would call a software project as such? It does not even match the natural order of Plan 9; Plan then 9: P9!

14.4.2. 9P getting started

As usual, we have already set everything up for you. On host:

cd "$(./getvar p9_dir)"
uname -a > host

Guest:

cd /mnt/9p/data
cat host
uname -a > guest

Host:

cat guest

The main ingredients for this are:

• 9P settings in our kernel configs

• 9p entry on our rootfs_overlay/etc/fstab

  Alternatively, you could also mount your own with:

  mkdir /mnt/my9p
  mount -t 9p -o trans=virtio,version=9p2000.L host0 /mnt/my9p

  where mount tag host0 is set by the emulator (mount_tag flag on QEMU CLI), and can be found in the guest with: cat /sys/bus/virtio/drivers/9pnet_virtio/virtio0/mount_tag as documented at: https://www.kernel.org/doc/Documentation/filesystems/9p.txt.

• Launch QEMU with -virtfs as in your run script

  When we tried:

  security_model=mapped

  writes from guest failed due to user mismatch problems: https://serverfault.com/questions/342801/read-write-access-for-passthrough-9p-filesystems-with-libvirt-qemu

Bibliography:

• https://superuser.com/questions/628169/how-to-share-a-directory-with-the-host-without-networking-in-qemu

• https://wiki.qemu.org/Documentation/9psetup

14.4.3. gem5 9P

Is possible on aarch64 as shown at: https://gem5-review.googlesource.com/c/public/gem5/+/22831, and it is just a matter of exposing to X86 for those that want it.

Enable it by passing the --vio-9p option on the fs.py gem5 command line:

./run --arch aarch64 --emulator gem5 -- --vio-9p

Then on the guest:

mkdir -p /mnt/9p/gem5
mount -t 9p -o trans=virtio,version=9p2000.L,aname=/path/to/linux-kernel-module-cheat/out/run/gem5/aarch64/0/m5out/9p/share gem5 /mnt/9p/gem5
echo asdf > /mnt/9p/gem5/qwer

Yes, you have to pass the full path to the directory on the host. Yes, this is horrible.

The shared directory is:

out/run/gem5/aarch64/0/m5out/9p/share

so we can observe the file the guest wrote from the host with:

out/run/gem5/aarch64/0/m5out/9p/share/qwer

and vice versa:

echo zxvc > out/run/gem5/aarch64/0/m5out/9p/share/qwer

is now visible from the guest:

cat /mnt/9p/gem5/qwer

Checkpoint restore with an open mount will likely fail because gem5 uses an ugly external executable to implement diod. The protocol is not very complex, and QEMU implements it in-tree, which is what gem5 should do as well at some point.

Also checkpoint without --vio-9p and restore with --vio-9p did not work either, the mount fails.

However, this did work, on guest:

unmount /mnt/9p/gem5
m5 checkpoint

then restore with the detalied CPU of interest e.g.

./run --arch aarch64 --emulator gem5 -- --vio-9p --cpu-type DerivO3CPU --caches

Tested on gem5 b2847f43c91e27f43bd4ac08abd528efcf00f2fd, LKMC 52a5fdd7c1d6eadc5900fc76e128995d4849aada.

14.4.4. NFS

TODO: get working.

9P is better with emulation, but let’s just get this working for fun.

First make sure that this works: Section 14.3.2, “Guest to host networking”.

Then, build the kernel with NFS support:

./build-linux --config-fragment linux_config/nfs

Now on host:

sudo apt-get install nfs-kernel-server

Now edit /etc/exports to contain:

/tmp *(rw,sync,no_root_squash,no_subtree_check)

and restart the server:

sudo systemctl restart nfs-kernel-server

Now on guest:

mkdir /mnt/nfs
mount -t nfs 10.0.2.2:/tmp /mnt/nfs

TODO: failing with:

mount: mounting 10.0.2.2:/tmp on /mnt/nfs failed: No such device

And now the /tmp directory from host is not mounted on guest!

If you don’t want to start the NFS server after the next boot automatically so save resources, do:

systemctl disable nfs-kernel-server

15. Operating systems

https://en.wikipedia.org/wiki/Operating_system

• Linux kernel

• FreeBSD

• RTOS

• Xen

• U-Boot

16. Linux kernel

https://en.wikipedia.org/wiki/Linux_kernel

16.1. Linux kernel configuration

16.1.1. Modify kernel config

To modify a single option on top of our default kernel configs, do:

./build-linux --config 'CONFIG_FORTIFY_SOURCE=y'

Kernel modules depend on certain kernel configs, and therefore in general you might have to clean and rebuild the kernel modules after changing the kernel config:

./build-modules --clean
./build-modules

and then proceed as in Your first kernel module hack.

You might often get way without rebuilding the kernel modules however.

To use an extra kernel config fragment file on top of our defaults, do:

printf '
CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y
' > data/myconfig
./build-linux --config-fragment 'data/myconfig'

To use just your own exact .config instead of our defaults ones, use:

./build-linux --custom-config-file data/myconfig

There is also a shortcut --custom-config-file-gem5 to use the gem5 arm Linux kernel patches.

The following options can all be used together, sorted by decreasing config setting power precedence:

• --config

• --config-fragment

• --custom-config-file

To do a clean menu config yourself and use that for the build, do:

./build-linux --clean
./build-linux --custom-config-target menuconfig

But remember that every new build re-configures the kernel by default, so to keep your configs you will need to use on further builds:

./build-linux --no-configure

So what you likely want to do instead is to save that as a new defconfig and use it later as:

./build-linux --no-configure --no-modules-install savedefconfig
cp "$(./getvar linux_build_dir)/defconfig" data/myconfig
./build-linux --custom-config-file data/myconfig

You can also use other config generating targets such as defconfig with the same method as shown at: Section 16.1.3.1.1, “Linux kernel defconfig”.

16.1.2. Find the kernel config

Get the build config in guest:

zcat /proc/config.gz

or with our shortcut:

./conf.sh

or to conveniently grep for a specific option case insensitively:

./conf.sh ikconfig

Source: rootfs_overlay/lkmc/conf.sh.

This is enabled by:

CONFIG_IKCONFIG=y
CONFIG_IKCONFIG_PROC=y

From host:

cat "$(./getvar linux_config)"

Just for fun https://stackoverflow.com/questions/14958192/how-to-get-the-config-from-a-linux-kernel-image/14958263#14958263:

./linux/scripts/extract-ikconfig "$(./getvar vmlinux)"

although this can be useful when someone gives you a random image.

16.1.3. About our Linux kernel configs

By default, build-linux generates a .config that is a mixture of:

• a base config extracted from Buildroot’s minimal per machine .config, which has the minimal options needed to boot as explained at: Section 16.1.3.1, “About Buildroot’s kernel configs”.

• small overlays put top of that

To find out which kernel configs are being used exactly, simply run:

./build-linux --dry-run

and look for the merge_config.sh call. This script from the Linux kernel tree, as the name suggests, merges multiple configuration files into one as explained at: https://unix.stackexchange.com/questions/224887/how-to-script-make-menuconfig-to-automate-linux-kernel-build-configuration/450407#450407

For each arch, the base of our configs are named as:

linux_config/buildroot-<arch>

e.g.: linux_config/buildroot-x86_64.

These configs are extracted directly from a Buildroot build with update-buildroot-kernel-configs.

Note that Buildroot can sed override some of the configurations, e.g. it forces CONFIG_BLK_DEV_INITRD=y when BR2_TARGET_ROOTFS_CPIO is on. For this reason, those configs are not simply copy pasted from Buildroot files, but rather from a Buildroot kernel build, and then minimized with make savedefconfig: https://stackoverflow.com/questions/27899104/how-to-create-a-defconfig-file-from-a-config

On top of those, we add the following by default:

• linux_config/min: see: Section 16.1.3.1.2, “Linux kernel min config”

• linux_config/default: other optional configs that we enable by default because they increase visibility, or expose some cool feature, and don’t significantly increase build time nor add significant runtime overhead

  We have since observed that the kernel size itself is very bloated compared to defconfig as shown at: Section 16.1.3.1.1, “Linux kernel defconfig”.

16.1.3.1. About Buildroot’s kernel configs

To see Buildroot’s base configs, start from buildroot/configs/qemu_x86_64_defconfig.

That file contains BR2_LINUX_KERNEL_CUSTOM_CONFIG_FILE="board/qemu/x86_64/linux-4.15.config", which points to the base config file used: board/qemu/x86_64/linux-4.15.config.

arm, on the other hand, uses buildroot/configs/qemu_arm_vexpress_defconfig, which contains BR2_LINUX_KERNEL_DEFCONFIG="vexpress", and therefore just does a make vexpress_defconfig, and gets its config from the Linux kernel tree itself.

16.1.3.1.1. Linux kernel defconfig

To boot defconfig from disk on Linux and see a shell, all we need is these missing virtio options:

./build-linux \
  --linux-build-id defconfig \
  --custom-config-target defconfig \
  --config CONFIG_VIRTIO_PCI=y \
  --config CONFIG_VIRTIO_BLK=y \
;
./run --linux-build-id defconfig

Oh, and check this out:

du -h \
  "$(./getvar vmlinux)" \
  "$(./getvar --linux-build-id defconfig vmlinux)" \
;

Output:

360M    /path/to/linux-kernel-module-cheat/out/linux/default/x86_64/vmlinux
47M     /path/to/linux-kernel-module-cheat/out/linux/defconfig/x86_64/vmlinux

Brutal. Where did we go wrong?

The extra virtio options are not needed if we use initrd:

./build-linux \
  --linux-build-id defconfig \
  --custom-config-target defconfig \
;
./run --initrd --linux-build-id defconfig

On aarch64, we can boot from initrd with:

./build-linux \
  --arch aarch64 \
  --linux-build-id defconfig \
  --custom-config-target defconfig \
;
./run \
  --arch aarch64 \
  --initrd \
  --linux-build-id defconfig \
  --memory 2G \
;

We need the 2G of memory because the CPIO is 600MiB due to a humongous amount of loadable kernel modules!

In aarch64, the size situation is inverted from x86_64, and this can be seen on the vmlinux size as well:

118M    /path/to/linux-kernel-module-cheat/out/linux/default/aarch64/vmlinux
240M    /path/to/linux-kernel-module-cheat/out/linux/defconfig/aarch64/vmlinux

So it seems that the ARM devs decided rather than creating a minimal config that boots QEMU, to try and make a single config that boots every board in existence. Terrible!

Bibliography: https://unix.stackexchange.com/questions/29439/compiling-the-kernel-with-default-configurations/204512#204512

Tested on 1e2b7f1e5e9e3073863dc17e25b2455c8ebdeadd + 1.

16.1.3.1.2. Linux kernel min config

linux_config/min contains minimal tweaks required to boot gem5 or for using our slightly different QEMU command line options than Buildroot on all archs.

It is one of the default config fragments we use, as explained at: Section 16.1.3, “About our Linux kernel configs”>.

Having the same config working for both QEMU and gem5 (oh, the hours of bisection) means that you can deal with functional matters in QEMU, which runs much faster, and switch to gem5 only for performance issues.

We can build just with min on top of the base config with:

./build-linux \
  --arch aarch64 \
  --config-fragment linux_config/min \
  --custom-config-file linux_config/buildroot-aarch64 \
  --linux-build-id min \
;

vmlinux had a very similar size to the default. It seems that linux_config/buildroot-aarch64 contains or implies most linux_config/default options already? TODO: that seems odd, really?

Tested on 649d06d6758cefd080d04dc47fd6a5a26a620874 + 1.

16.1.3.2. Notable alternate gem5 kernel configs

Other configs which we had previously tested at 4e0d9af81fcce2ce4e777cb82a1990d7c2ca7c1e are:

• arm and aarch64 configs present in the official ARM gem5 Linux kernel fork as described at: Section 23.9, “gem5 arm Linux kernel patches”. Some of the configs present there are added by the patches.

• Jason’s magic x86_64 config: http://web.archive.org/web/20171229121642/http://www.lowepower.com/jason/files/config which is referenced at: http://web.archive.org/web/20171229121525/http://www.lowepower.com/jason/setting-up-gem5-full-system.html. QEMU boots with that by removing # CONFIG_VIRTIO_PCI is not set.

16.2. Kernel version

16.2.1. Find the kernel version

We try to use the latest possible kernel major release version.

In QEMU:

cat /proc/version

or in the source:

cd "$(./getvar linux_source_dir)"
git log | grep -E '    Linux [0-9]+\.' | head

16.2.2. Update the Linux kernel

During update all you kernel modules may break since the kernel API is not stable.

They are usually trivial breaks of things moving around headers or to sub-structs.

The userland, however, should simply not break, as Linus enforces strict backwards compatibility of userland interfaces.

This backwards compatibility is just awesome, it makes getting and running the latest master painless.

This also makes this repo the perfect setup to develop the Linux kernel.

In case something breaks while updating the Linux kernel, you can try to bisect it to understand the root cause, see: [bisection].

16.2.2.1. Update the Linux kernel LKMC procedure

First, use use the branching procedure described at: [update-a-forked-submodule]

Because the kernel is so central to this repository, almost all tests must be re-run, so basically just follow the full testing procedure described at: [test-this-repo]. The only tests that can be skipped are essentially the [baremetal] tests.

Before comitting, don’t forget to update:

• the linux_kernel_version constant in common.py

• the tagline of this repository on:

  • this README

  • the GitHub project description

16.2.3. Downgrade the Linux kernel

The kernel is not forward compatible, however, so downgrading the Linux kernel requires downgrading the userland too to the latest Buildroot branch that supports it.

The default Linux kernel version is bumped in Buildroot with commit messages of type:

linux: bump default to version 4.9.6

So you can try:

git log --grep 'linux: bump default to version'

Those commits change BR2_LINUX_KERNEL_LATEST_VERSION in /linux/Config.in.

You should then look up if there is a branch that supports that kernel. Staying on branches is a good idea as they will get backports, in particular ones that fix the build as newer host versions come out.

Finally, after downgrading Buildroot, if something does not work, you might also have to make some changes to how this repo uses Buildroot, as the Buildroot configuration options might have changed.

We don’t expect those changes to be very difficult. A good way to approach the task is to:

• do a dry run build to get the equivalent Bash commands used:

  ./build-buildroot --dry-run

• build the Buildroot documentation for the version you are going to use, and check if all Buildroot build commands make sense there

Then, if you spot an option that is wrong, some grepping in this repo should quickly point you to the code you need to modify.

It also possible that you will need to apply some patches from newer Buildroot versions for it to build, due to incompatibilities with the host Ubuntu packages and that Buildroot version. Just read the error message, and try:

• git log master — packages/<pkg>

• Google the error message for mailing list hits

Successful port reports:

• v3.18: https://github.com/cirosantilli/linux-kernel-module-cheat/issues/39#issuecomment-438525481

16.3. Kernel command line parameters

Bootloaders can pass a string as input to the Linux kernel when it is booting to control its behaviour, much like the execve system call does to userland processes.

This allows us to control the behaviour of the kernel without rebuilding anything.

With QEMU, QEMU itself acts as the bootloader, and provides the -append option and we expose it through ./run --kernel-cli, e.g.:

./run --kernel-cli 'foo bar'

Then inside the host, you can check which options were given with:

cat /proc/cmdline

They are also printed at the beginning of the boot message:

dmesg | grep "Command line"

See also:

• https://unix.stackexchange.com/questions/48601/how-to-display-the-linux-kernel-command-line-parameters-given-for-the-current-bo

• https://askubuntu.com/questions/32654/how-do-i-find-the-boot-parameters-used-by-the-running-kernel

The arguments are documented in the kernel documentation: https://www.kernel.org/doc/html/v4.14/admin-guide/kernel-parameters.html

When dealing with real boards, extra command line options are provided on some magic bootloader configuration file, e.g.:

• GRUB configuration files: https://askubuntu.com/questions/19486/how-do-i-add-a-kernel-boot-parameter

• Raspberry pi /boot/cmdline.txt on a magic partition: https://raspberrypi.stackexchange.com/questions/14839/how-to-change-the-kernel-commandline-for-archlinuxarm-on-raspberry-pi-effectly

16.3.1. Kernel command line parameters escaping

Double quotes can be used to escape spaces as in opt="a b", but double quotes themselves cannot be escaped, e.g. opt"a\"b"

This even lead us to use base64 encoding with --eval!

16.3.2. Kernel command line parameters definition points

There are two methods:

• __setup as in:

  __setup("console=", console_setup);

• core_param as in:

  core_param(panic, panic_timeout, int, 0644);

core_param suggests how they are different:

/**
 * core_param - define a historical core kernel parameter.

...

 * core_param is just like module_param(), but cannot be modular and
 * doesn't add a prefix (such as "printk.").  This is for compatibility
 * with __setup(), and it makes sense as truly core parameters aren't
 * tied to the particular file they're in.
 */

16.3.3. rw

By default, the Linux kernel mounts the root filesystem as readonly. TODO rationale?

This cannot be observed in the default BusyBox init, because by default our rootfs_overlay/etc/inittab does:

/bin/mount -o remount,rw /

Analogously, Ubuntu 18.04 does in its fstab something like:

UUID=/dev/sda1 / ext4 errors=remount-ro 0 1

which uses default mount rw flags.

We have however removed those setups init setups to keep things more minimal, and replaced them with the rw kernel boot parameter makes the root mounted as writable.

To observe the default readonly behaviour, hack the run script to remove replace init, and then run on a raw shell:

./run --kernel-cli 'init=/bin/sh'

Now try to do:

touch a

which fails with:

touch: a: Read-only file system

We can also observe the read-onlyness with:

mount -t proc /proc
mount

which contains:

/dev/root on / type ext2 (ro,relatime,block_validity,barrier,user_xattr)

and so it is Read Only as shown by ro.

16.3.4. norandmaps

Disable userland address space randomization. Test it out by running [rand-check-out] twice:

./run --eval-after './linux/rand_check.out;./linux/poweroff.out'
./run --eval-after './linux/rand_check.out;./linux/poweroff.out'

If we remove it from our run script by hacking it up, the addresses shown by linux/rand_check.out vary across boots.

Equivalent to:

echo 0 > /proc/sys/kernel/randomize_va_space

16.4. printk

printk is the most simple and widely used way of getting information from the kernel, so you should familiarize yourself with its basic configuration.

We use printk a lot in our kernel modules, and it shows on the terminal by default, along with stdout and what you type.

Hide all printk messages:

dmesg -n 1

or equivalently:

echo 1 > /proc/sys/kernel/printk

See also: https://superuser.com/questions/351387/how-to-stop-kernel-messages-from-flooding-my-console

Do it with a Kernel command line parameters to affect the boot itself:

./run --kernel-cli 'loglevel=5'

and now only boot warning messages or worse show, which is useful to identify problems.

Our default printk format is:

<LEVEL>[TIMESTAMP] MESSAGE

e.g.:

<6>[    2.979121] Freeing unused kernel memory: 2024K

where:

• LEVEL: higher means less serious

• TIMESTAMP: seconds since boot

This format is selected by the following boot options:

• console_msg_format=syslog: add the <LEVEL> part. Added in v4.16.

• printk.time=y: add the [TIMESTAMP] part

The debug highest level is a bit more magic, see: Section 16.4.3, “pr_debug” for more info.

16.4.1. /proc/sys/kernel/printk

The current printk level can be obtained with:

cat /proc/sys/kernel/printk

As of 87e846fc1f9c57840e143513ebd69c638bd37aa8 this prints:

7       4       1       7

which contains:

• 7: current log level, modifiable by previously mentioned methods

• 4: documented as: "printk’s without a loglevel use this": TODO what does that mean, how to call printk without a log level?

• 1: minimum log level that still prints something (0 prints nothing)

• 7: default log level

We start at the boot time default after boot by default, as can be seen from:

insmod myprintk.ko

which outputs something like:

<1>[   12.494429] pr_alert
<2>[   12.494666] pr_crit
<3>[   12.494823] pr_err
<4>[   12.494911] pr_warning
<5>[   12.495170] pr_notice
<6>[   12.495327] pr_info

Source: kernel_modules/myprintk.c

This proc entry is defined at: https://github.com/torvalds/linux/blob/v5.1/kernel/sysctl.c#L839

#if defined CONFIG_PRINTK
	{
		.procname	= "printk",
		.data		= &console_loglevel,
		.maxlen		= 4*sizeof(int),
		.mode		= 0644,
		.proc_handler	= proc_dointvec,
	},

which teaches us that printk can be completely disabled at compile time:

config PRINTK
	default y
	bool "Enable support for printk" if EXPERT
	select IRQ_WORK
	help
	  This option enables normal printk support. Removing it
	  eliminates most of the message strings from the kernel image
	  and makes the kernel more or less silent. As this makes it
	  very difficult to diagnose system problems, saying N here is
	  strongly discouraged.

console_loglevel is defined at:

#define console_loglevel (console_printk[0])

and console_printk is an array with 4 ints:

int console_printk[4] = {
	CONSOLE_LOGLEVEL_DEFAULT,	/* console_loglevel */
	MESSAGE_LOGLEVEL_DEFAULT,	/* default_message_loglevel */
	CONSOLE_LOGLEVEL_MIN,		/* minimum_console_loglevel */
	CONSOLE_LOGLEVEL_DEFAULT,	/* default_console_loglevel */
};

and then we see that the default is configurable with CONFIG_CONSOLE_LOGLEVEL_DEFAULT:

/*
 * Default used to be hard-coded at 7, quiet used to be hardcoded at 4,
 * we're now allowing both to be set from kernel config.
 */
#define CONSOLE_LOGLEVEL_DEFAULT CONFIG_CONSOLE_LOGLEVEL_DEFAULT
#define CONSOLE_LOGLEVEL_QUIET	 CONFIG_CONSOLE_LOGLEVEL_QUIET

The message loglevel default is explained at:

/* printk's without a loglevel use this.. */
#define MESSAGE_LOGLEVEL_DEFAULT CONFIG_MESSAGE_LOGLEVEL_DEFAULT

The min is just hardcoded to one as you would expect, with some amazing kernel comedy around it:

/* We show everything that is MORE important than this.. */
#define CONSOLE_LOGLEVEL_SILENT  0 /* Mum's the word */
#define CONSOLE_LOGLEVEL_MIN	 1 /* Minimum loglevel we let people use */
#define CONSOLE_LOGLEVEL_DEBUG	10 /* issue debug messages */
#define CONSOLE_LOGLEVEL_MOTORMOUTH 15	/* You can't shut this one up */

We then also learn about the useless quiet and debug kernel parameters at:

config CONSOLE_LOGLEVEL_QUIET
	int "quiet console loglevel (1-15)"
	range 1 15
	default "4"
	help
	  loglevel to use when "quiet" is passed on the kernel commandline.

	  When "quiet" is passed on the kernel commandline this loglevel
	  will be used as the loglevel. IOW passing "quiet" will be the
	  equivalent of passing "loglevel=<CONSOLE_LOGLEVEL_QUIET>"

which explains the useless reason why that number is special. This is implemented at:

static int __init debug_kernel(char *str)
{
	console_loglevel = CONSOLE_LOGLEVEL_DEBUG;
	return 0;
}

static int __init quiet_kernel(char *str)
{
	console_loglevel = CONSOLE_LOGLEVEL_QUIET;
	return 0;
}

early_param("debug", debug_kernel);
early_param("quiet", quiet_kernel);

16.4.2. ignore_loglevel

./run --kernel-cli 'ignore_loglevel'

enables all log levels, and is basically the same as:

./run --kernel-cli 'loglevel=8'

except that you don’t need to know what is the maximum level.

16.4.3. pr_debug

https://stackoverflow.com/questions/28936199/why-is-pr-debug-of-the-linux-kernel-not-giving-any-output/49835405#49835405

Debug messages are not printable by default without recompiling.

But the awesome CONFIG_DYNAMIC_DEBUG=y option which we enable by default allows us to do:

echo 8 > /proc/sys/kernel/printk
echo 'file kernel/module.c +p' > /sys/kernel/debug/dynamic_debug/control
./linux/myinsmod.out hello.ko

and we have a shortcut at:

./pr_debug.sh

Source: rootfs_overlay/lkmc/pr_debug.sh.

Syntax: https://www.kernel.org/doc/html/v4.11/admin-guide/dynamic-debug-howto.html

Wildcards are also accepted, e.g. enable all messages from all files:

echo 'file * +p' > /sys/kernel/debug/dynamic_debug/control

TODO: why is this not working:

echo 'func sys_init_module +p' > /sys/kernel/debug/dynamic_debug/control

Enable messages in specific modules:

echo 8 > /proc/sys/kernel/printk
echo 'module myprintk +p' > /sys/kernel/debug/dynamic_debug/control
insmod myprintk.ko

Source: kernel_modules/myprintk.c

This outputs the pr_debug message:

printk debug

but TODO: it also shows debug messages even without enabling them explicitly:

echo 8 > /proc/sys/kernel/printk
insmod myprintk.ko

and it shows as enabled:

# grep myprintk /sys/kernel/debug/dynamic_debug/control
/root/linux-kernel-module-cheat/out/kernel_modules/x86_64/kernel_modules/panic.c:12 [myprintk]myinit =p "pr_debug\012"

Enable pr_debug for boot messages as well, before we can reach userland and write to /proc:

./run --kernel-cli 'dyndbg="file * +p" loglevel=8'

Get ready for the noisiest boot ever, I think it overflows the printk buffer and funny things happen.

16.4.3.1. pr_debug != printk(KERN_DEBUG

When CONFIG_DYNAMIC_DEBUG is set, printk(KERN_DEBUG is not the exact same as pr_debug( since printk(KERN_DEBUG messages are visible with:

./run --kernel-cli 'initcall_debug logleve=8'

which outputs lines of type:

<7>[    1.756680] calling  clk_disable_unused+0x0/0x130 @ 1
<7>[    1.757003] initcall clk_disable_unused+0x0/0x130 returned 0 after 111 usecs

which are printk(KERN_DEBUG inside init/main.c in v4.16.

Mentioned at: https://stackoverflow.com/questions/37272109/how-to-get-details-of-all-modules-drivers-got-initialized-probed-during-kernel-b

This likely comes from the ifdef split at init/main.c:

/* If you are writing a driver, please use dev_dbg instead */
#if defined(CONFIG_DYNAMIC_DEBUG)
#include <linux/dynamic_debug.h>

/* dynamic_pr_debug() uses pr_fmt() internally so we don't need it here */
#define pr_debug(fmt, ...) \
    dynamic_pr_debug(fmt, ##__VA_ARGS__)
#elif defined(DEBUG)
#define pr_debug(fmt, ...) \
    printk(KERN_DEBUG pr_fmt(fmt), ##__VA_ARGS__)
#else
#define pr_debug(fmt, ...) \
    no_printk(KERN_DEBUG pr_fmt(fmt), ##__VA_ARGS__)
#endif

16.5. Kernel module APIs

16.5.1. Kernel module parameters

The Linux kernel allows passing module parameters at insertion time through the init_module and finit_module system calls.

The insmod tool exposes that as:

insmod params.ko i=3 j=4

Parameters are declared in the module as:

static u32 i = 0;
module_param(i, int, S_IRUSR | S_IWUSR);
MODULE_PARM_DESC(i, "my favorite int");

Automated test:

./params.sh
echo $?

Outcome: the test passes:

0

Sources:

• kernel_modules/params.c

• rootfs_overlay/lkmc/params.sh

As shown in the example, module parameters can also be read and modified at runtime from sysfs.

We can obtain the help text of the parameters with:

modinfo params.ko

The output contains:

parm:           j:my second favorite int
parm:           i:my favorite int

16.5.1.1. modprobe.conf

modprobe insertion can also set default parameters via the /etc/modprobe.conf file:

modprobe params
cat /sys/kernel/debug/lkmc_params

Output:

12 34

This is specially important when loading modules with Kernel module dependencies or else we would have no opportunity of passing those.

modprobe.conf doesn’t actually insmod anything for us: https://superuser.com/questions/397842/automatically-load-kernel-module-at-boot-angstrom/1267464#1267464

16.5.2. Kernel module dependencies

One module can depend on symbols of another module that are exported with EXPORT_SYMBOL:

./dep.sh
echo $?

Outcome: the test passes:

0

Sources:

• kernel_modules/dep.c

• kernel_modules/dep2.c

• rootfs_overlay/lkmc/dep.sh

The kernel deduces dependencies based on the EXPORT_SYMBOL that each module uses.

Symbols exported by EXPORT_SYMBOL can be seen with:

insmod dep.ko
grep lkmc_dep /proc/kallsyms

sample output:

ffffffffc0001030 r __ksymtab_lkmc_dep   [dep]
ffffffffc000104d r __kstrtab_lkmc_dep   [dep]
ffffffffc0002300 B lkmc_dep     [dep]

This requires CONFIG_KALLSYMS_ALL=y.

Dependency information is stored by the kernel module build system in the .ko files' MODULE_INFO, e.g.:

modinfo dep2.ko

contains:

depends:        dep

We can double check with:

strings 3 dep2.ko | grep -E 'depends'

The output contains:

depends=dep

Module dependencies are also stored at:

cd /lib/module/*
grep dep modules.dep

Output:

extra/dep2.ko: extra/dep.ko
extra/dep.ko:

TODO: what for, and at which point point does Buildroot / BusyBox generate that file?

16.5.2.1. Kernel module dependencies with modprobe

Unlike insmod, modprobe deals with kernel module dependencies for us.

First get [kernel-modules-buildroot-package] working.

Then, for example:

modprobe buildroot_dep2

outputs to dmesg:

42

and then:

lsmod

outputs:

Module                  Size  Used by    Tainted: G
buildroot_dep2         16384  0
buildroot_dep          16384  1 buildroot_dep2

Sources:

• buildroot_packages/kernel_modules/buildroot_dep.c

• buildroot_packages/kernel_modules/buildroot_dep2.c

Removal also removes required modules that have zero usage count:

modprobe -r buildroot_dep2

modprobe uses information from the modules.dep file to decide the required dependencies. That file contains:

extra/buildroot_dep2.ko: extra/buildroot_dep.ko

Bibliography:

• https://askubuntu.com/questions/20070/whats-the-difference-between-insmod-and-modprobe

• https://stackoverflow.com/questions/22891705/whats-the-difference-between-insmod-and-modprobe

16.5.3. MODULE_INFO

Module metadata is stored on module files at compile time. Some of the fields can be retrieved through the THIS_MODULE struct module:

insmod module_info.ko

Dmesg output:

name = module_info
version = 1.0

Source: kernel_modules/module_info.c

Some of those are also present on sysfs:

cat /sys/module/module_info/version

Output:

1.0

And we can also observe them with the modinfo command line utility:

modinfo module_info.ko

sample output:

filename:       module_info.ko
license:        GPL
version:        1.0
srcversion:     AF3DE8A8CFCDEB6B00E35B6
depends:
vermagic:       4.17.0 SMP mod_unload modversions

Module information is stored in a special .modinfo section of the ELF file:

./run-toolchain readelf -- -SW "$(./getvar kernel_modules_build_subdir)/module_info.ko"

contains:

  [ 5] .modinfo          PROGBITS        0000000000000000 0000d8 000096 00   A  0   0  8

and:

./run-toolchain readelf -- -x .modinfo "$(./getvar kernel_modules_build_subdir)/module_info.ko"

gives:

  0x00000000 6c696365 6e73653d 47504c00 76657273 license=GPL.vers
  0x00000010 696f6e3d 312e3000 61736466 3d717765 ion=1.0.asdf=qwe
  0x00000020 72000000 00000000 73726376 65727369 r.......srcversi
  0x00000030 6f6e3d41 46334445 38413843 46434445 on=AF3DE8A8CFCDE
  0x00000040 42364230 30453335 42360000 00000000 B6B00E35B6......
  0x00000050 64657065 6e64733d 006e616d 653d6d6f depends=.name=mo
  0x00000060 64756c65 5f696e66 6f007665 726d6167 dule_info.vermag
  0x00000070 69633d34 2e31372e 3020534d 50206d6f ic=4.17.0 SMP mo
  0x00000080 645f756e 6c6f6164 206d6f64 76657273 d_unload modvers
  0x00000090 696f6e73 2000                       ions .

I think a dedicated section is used to allow the Linux kernel and command line tools to easily parse that information from the ELF file as we’ve done with readelf.

Bibliography:

• https://stackoverflow.com/questions/19467150/significance-of-this-module-in-linux-driver/49812248#49812248

• https://stackoverflow.com/questions/4839024/how-to-find-the-version-of-a-compiled-kernel-module/42556565#42556565

• https://unix.stackexchange.com/questions/238167/how-to-understand-the-modinfo-output

16.5.4. vermagic

kernel_modules/vermagic.c

As of kernel v5.8, you can’t use VERMAGIC_STRING string from modules anymore as per: https://github.com/cirosantilli/linux/commit/51161bfc66a68d21f13d15a689b3ea7980457790. So instead we just showcase init_utsname.

Sample insmod output as of LKMC fa8c2ee521ea83a74a2300e7a3be9f9ab86e2cb6 + 1 aarch64:

<6>[   25.180697] sysname    = Linux
<6>[   25.180697] nodename   = buildroot
<6>[   25.180697] release    = 5.9.2
<6>[   25.180697] version    = #1 SMP Thu Jan 1 00:00:00 UTC 1970
<6>[   25.180697] machine    = aarch64
<6>[   25.180697] domainname = (none)

Vermagic is a magic string present in the kernel and previously visible in MODULE_INFO on kernel modules. It is used to verify that the kernel module was compiled against a compatible kernel version and relevant configuration:

insmod vermagic.ko

Possible dmesg output:

VERMAGIC_STRING = 4.17.0 SMP mod_unload modversions

If we artificially create a mismatch with MODULE_INFO(vermagic, the insmod fails with:

insmod: can't insert 'vermagic_fail.ko': invalid module format

and dmesg says the expected and found vermagic found:

vermagic_fail: version magic 'asdfqwer' should be '4.17.0 SMP mod_unload modversions '

Source: kernel_modules/vermagic_fail.c

The kernel’s vermagic is defined based on compile time configurations at include/linux/vermagic.h:

#define VERMAGIC_STRING                                                 \
        UTS_RELEASE " "                                                 \
        MODULE_VERMAGIC_SMP MODULE_VERMAGIC_PREEMPT                     \
        MODULE_VERMAGIC_MODULE_UNLOAD MODULE_VERMAGIC_MODVERSIONS       \
        MODULE_ARCH_VERMAGIC                                            \
        MODULE_RANDSTRUCT_PLUGIN

The SMP part of the string for example is defined on the same file based on the value of CONFIG_SMP:

#ifdef CONFIG_SMP
#define MODULE_VERMAGIC_SMP "SMP "
#else
#define MODULE_VERMAGIC_SMP ""

TODO how to get the vermagic from running kernel from userland? https://lists.kernelnewbies.org/pipermail/kernelnewbies/2012-October/006306.html

kmod modprobe has a flag to skip the vermagic check:

--force-modversion

This option just strips modversion information from the module before loading, so it is not a kernel feature.

16.5.5. init_module

init_module and cleanup_module are an older alternative to the module_init and module_exit macros:

insmod init_module.ko
rmmod init_module

Dmesg output:

init_module
cleanup_module

Source: kernel_modules/init_module.c

TODO why were module_init and module_exit created? https://stackoverflow.com/questions/3218320/what-is-the-difference-between-module-init-and-init-module-in-a-linux-kernel-mod

16.5.6. Floating point in kernel modules

It is generally hard / impossible to use floating point operations in the kernel. TODO understand details.

A quick (x86-only for now because lazy) example is shown at: kernel_modules/float.c

Usage:

insmod float.ko myfloat=1 enable_fpu=1

We have to call: kernel_fpu_begin() before starting FPU operations, and kernel_fpu_end() when we are done. This particular example however did not blow up without it at lkmc 7f917af66b17373505f6c21d75af9331d624b3a9 + 1:

insmod float.ko myfloat=1 enable_fpu=0

The v5.1 documentation under arch/x86/include/asm/fpu/api.h reads:

 * Use kernel_fpu_begin/end() if you intend to use FPU in kernel context. It
 * disables preemption so be careful if you intend to use it for long periods
 * of time.

The example sets in the kernel_modules/Makefile:

CFLAGS_REMOVE_float.o += -mno-sse -mno-sse2

to avoid:

error: SSE register return with SSE disabled

We found those flags with ./build-modules --verbose.

Bibliography:

• https://stackoverflow.com/questions/13886338/use-of-floating-point-in-the-linux-kernel

• https://stackoverflow.com/questions/15883947/why-am-i-able-to-perform-floating-point-operations-inside-a-linux-kernel-module/47056242

• https://stackoverflow.com/questions/1556142/sse-register-return-with-sse-disabled

16.6. Kernel panic and oops

To test out kernel panics and oops in controlled circumstances, try out the modules:

insmod panic.ko
insmod oops.ko

Source:

• kernel_modules/panic.c

• kernel_modules/oops.c

A panic can also be generated with:

echo c > /proc/sysrq-trigger

Panic vs oops: https://unix.stackexchange.com/questions/91854/whats-the-difference-between-a-kernel-oops-and-a-kernel-panic

How to generate them:

• https://unix.stackexchange.com/questions/66197/how-to-cause-kernel-panic-with-a-single-command

• https://stackoverflow.com/questions/23484147/generate-kernel-oops-or-crash-in-the-code

When a panic happens, Shift-PgUp does not work as it normally does, and it is hard to get the logs if on are on QEMU graphic mode:

• https://superuser.com/questions/848412/scrolling-up-the-failed-screen-with-kernel-panic

• https://superuser.com/questions/269228/write-qemu-booting-virtual-machine-output-to-a-file

• http://www.reactos.org/wiki/QEMU#Redirect_to_a_file

16.6.1. Kernel panic

On panic, the kernel dies, and so does our terminal.

The panic trace looks like:

panic: loading out-of-tree module taints kernel.
panic myinit
Kernel panic - not syncing: hello panic
CPU: 0 PID: 53 Comm: insmod Tainted: G           O     4.16.0 #6
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.0-0-g63451fca13-prebuilt.qemu-project.org 04/01/2014
Call Trace:
 dump_stack+0x7d/0xba
 ? 0xffffffffc0000000
 panic+0xda/0x213
 ? printk+0x43/0x4b
 ? 0xffffffffc0000000
 myinit+0x1d/0x20 [panic]
 do_one_initcall+0x3e/0x170
 do_init_module+0x5b/0x210
 load_module+0x2035/0x29d0
 ? kernel_read_file+0x7d/0x140
 ? SyS_finit_module+0xa8/0xb0
 SyS_finit_module+0xa8/0xb0
 do_syscall_64+0x6f/0x310
 ? trace_hardirqs_off_thunk+0x1a/0x32
 entry_SYSCALL_64_after_hwframe+0x42/0xb7
RIP: 0033:0x7ffff7b36206
RSP: 002b:00007fffffffeb78 EFLAGS: 00000206 ORIG_RAX: 0000000000000139
RAX: ffffffffffffffda RBX: 000000000000005c RCX: 00007ffff7b36206
RDX: 0000000000000000 RSI: 000000000069e010 RDI: 0000000000000003
RBP: 000000000069e010 R08: 00007ffff7ddd320 R09: 0000000000000000
R10: 00007ffff7ddd320 R11: 0000000000000206 R12: 0000000000000003
R13: 00007fffffffef4a R14: 0000000000000000 R15: 0000000000000000
Kernel Offset: disabled
---[ end Kernel panic - not syncing: hello panic

Notice how our panic message hello panic is visible at:

Kernel panic - not syncing: hello panic

16.6.1.1. Kernel module stack trace to source line

The log shows which module each symbol belongs to if any, e.g.:

myinit+0x1d/0x20 [panic]

says that the function myinit is in the module panic.

To find the line that panicked, do:

./run-gdb

and then:

info line *(myinit+0x1d)

which gives us the correct line:

Line 7 of "/root/linux-kernel-module-cheat/out/kernel_modules/x86_64/kernel_modules/panic.c" starts at address 0xbf00001c <myinit+28> and ends at 0xbf00002c <myexit>.

as explained at: https://stackoverflow.com/questions/8545931/using-gdb-to-convert-addresses-to-lines/27576029#27576029

The exact same thing can be done post mortem with:

./run-toolchain gdb -- \
  -batch \
  -ex 'info line *(myinit+0x1d)' \
  "$(./getvar kernel_modules_build_subdir)/panic.ko" \
;

Related:

• https://stackoverflow.com/questions/6151538/addr2line-on-kernel-module

• https://stackoverflow.com/questions/13468286/how-to-read-understand-analyze-and-debug-a-linux-kernel-panic

16.6.1.2. BUG_ON

Basically just calls panic("BUG!") for most archs.

16.6.1.3. Exit emulator on panic

For testing purposes, it is very useful to quit the emulator automatically with exit status non zero in case of kernel panic, instead of just hanging forever.

16.6.1.3.1. Exit QEMU on panic

Enabled by default with:

• panic=-1 command line option which reboots the kernel immediately on panic, see: Section 16.6.1.4, “Reboot on panic”

• QEMU -no-reboot, which makes QEMU exit when the guest tries to reboot

Also asked at https://unix.stackexchange.com/questions/443017/can-i-make-qemu-exit-with-failure-on-kernel-panic which also mentions the x86_64 -device pvpanic, but I don’t see much advantage to it.

TODO neither method exits with exit status different from 0, so for now we are just grepping the logs for panic messages, which sucks.

One possibility that gets close would be to use GDB step debug to break at the panic function, and then send a QEMU monitor from GDB quit command if that happens, but I don’t see a way to exit with non-zero status to indicate error.

16.6.1.3.2. Exit gem5 on panic

gem5 9048ef0ffbf21bedb803b785fb68f83e95c04db8 (January 2019) can detect panics automatically if the option system.panic_on_panic is on.

It parses kernel symbols and detecting when the PC reaches the address of the panic function. gem5 then prints to stdout:

Kernel panic in simulated kernel

and exits with status -6.

At gem5 ff52563a214c71fcd1e21e9f00ad839612032e3b (July 2018) behaviour was different, and just exited 0: https://www.mail-archive.com/gem5-users@gem5.org/msg15870.html TODO find fixing commit.

We enable the system.panic_on_panic option by default on arm and aarch64, which makes gem5 exit immediately in case of panic, which is awesome!

If we don’t set system.panic_on_panic, then gem5 just hangs on an infinite guest loop.

TODO: why doesn’t gem5 x86 ff52563a214c71fcd1e21e9f00ad839612032e3b support system.panic_on_panic as well? Trying to set system.panic_on_panic there fails with:

tried to set or access non-existentobject parameter: panic_on_panic

However, at that commit panic on x86 makes gem5 crash with:

panic: i8042 "System reset" command not implemented.

which is a good side effect of an unimplemented hardware feature, since the simulation actually stops.

The implementation of panic detection happens at: https://github.com/gem5/gem5/blob/1da285dfcc31b904afc27e440544d006aae25b38/src/arch/arm/linux/system.cc#L73

        kernelPanicEvent = addKernelFuncEventOrPanic<Linux::KernelPanicEvent>(
            "panic", "Kernel panic in simulated kernel", dmesg_output);

Here we see that the symbol "panic" for the panic() function is the one being tracked.

Related thread: https://stackoverflow.com/questions/56032347/is-there-a-way-to-identify-if-gem5-run-got-over-successfully

16.6.1.4. Reboot on panic

Make the kernel reboot after n seconds after panic:

echo 1 > /proc/sys/kernel/panic

Can also be controlled with the panic= kernel boot parameter.

0 to disable, -1 to reboot immediately.

Bibliography:

• https://github.com/torvalds/linux/blob/v4.17/Documentation/admin-guide/kernel-parameters.txt#L2931

• https://unix.stackexchange.com/questions/29567/how-to-configure-the-linux-kernel-to-reboot-on-panic/29569#29569

16.6.1.5. Panic trace show addresses instead of symbols

If CONFIG_KALLSYMS=n, then addresses are shown on traces instead of symbol plus offset.

In v4.16 it does not seem possible to configure that at runtime. GDB step debugging with:

./run --eval-after 'insmod dump_stack.ko' --gdb-wait --tmux-args dump_stack

shows that traces are printed at arch/x86/kernel/dumpstack.c:

static void printk_stack_address(unsigned long address, int reliable,
                 char *log_lvl)
{
    touch_nmi_watchdog();
    printk("%s %s%pB\n", log_lvl, reliable ? "" : "? ", (void *)address);
}

and %pB is documented at Documentation/core-api/printk-formats.rst:

If KALLSYMS are disabled then the symbol address is printed instead.

I wasn’t able do disable CONFIG_KALLSYMS to test this this out however, it is being selected by some other option? But I then used make menuconfig to see which options select it, and they were all off…​

16.6.2. Kernel oops

On oops, the shell still lives after.

However we:

• leave the normal control flow, and oops after never gets printed: an interrupt is serviced

• cannot rmmod oops afterwards

It is possible to make oops lead to panics always with:

echo 1 > /proc/sys/kernel/panic_on_oops
insmod oops.ko

An oops stack trace looks like:

BUG: unable to handle kernel NULL pointer dereference at 0000000000000000
IP: myinit+0x18/0x30 [oops]
PGD dccf067 P4D dccf067 PUD dcc1067 PMD 0
Oops: 0002 [#1] SMP NOPTI
Modules linked in: oops(O+)
CPU: 0 PID: 53 Comm: insmod Tainted: G           O     4.16.0 #6
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS rel-1.11.0-0-g63451fca13-prebuilt.qemu-project.org 04/01/2014
RIP: 0010:myinit+0x18/0x30 [oops]
RSP: 0018:ffffc900000d3cb0 EFLAGS: 00000282
RAX: 000000000000000b RBX: ffffffffc0000000 RCX: ffffffff81e3e3a8
RDX: 0000000000000001 RSI: 0000000000000086 RDI: ffffffffc0001033
RBP: ffffc900000d3e30 R08: 69796d2073706f6f R09: 000000000000013b
R10: ffffea0000373280 R11: ffffffff822d8b2d R12: 0000000000000000
R13: ffffffffc0002050 R14: ffffffffc0002000 R15: ffff88000dc934c8
FS:  00007ffff7ff66a0(0000) GS:ffff88000fc00000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 000000000dcd2000 CR4: 00000000000006f0
Call Trace:
 do_one_initcall+0x3e/0x170
 do_init_module+0x5b/0x210
 load_module+0x2035/0x29d0
 ? SyS_finit_module+0xa8/0xb0
 SyS_finit_module+0xa8/0xb0
 do_syscall_64+0x6f/0x310
 ? trace_hardirqs_off_thunk+0x1a/0x32
 entry_SYSCALL_64_after_hwframe+0x42/0xb7
RIP: 0033:0x7ffff7b36206
RSP: 002b:00007fffffffeb78 EFLAGS: 00000206 ORIG_RAX: 0000000000000139
RAX: ffffffffffffffda RBX: 000000000000005c RCX: 00007ffff7b36206
RDX: 0000000000000000 RSI: 000000000069e010 RDI: 0000000000000003
RBP: 000000000069e010 R08: 00007ffff7ddd320 R09: 0000000000000000
R10: 00007ffff7ddd320 R11: 0000000000000206 R12: 0000000000000003
R13: 00007fffffffef4b R14: 0000000000000000 R15: 0000000000000000
Code: <c7> 04 25 00 00 00 00 00 00 00 00 e8 b2 33 09 c1 31 c0 c3 0f 1f 44
RIP: myinit+0x18/0x30 [oops] RSP: ffffc900000d3cb0
CR2: 0000000000000000
---[ end trace 3cdb4e9d9842b503 ]---

To find the line that oopsed, look at the RIP register:

RIP: 0010:myinit+0x18/0x30 [oops]

and then on GDB:

./run-gdb

run

info line *(myinit+0x18)

which gives us the correct line:

Line 7 of "/root/linux-kernel-module-cheat/out/kernel_modules/x86_64/kernel_modules/panic.c" starts at address 0xbf00001c <myinit+28> and ends at 0xbf00002c <myexit>.

This-did not work on arm due to GDB step debug kernel module insmodded by init on ARM so we need to either:

• GDB module_init

• Kernel module stack trace to source line post-mortem method

16.6.3. dump_stack

The dump_stack function produces a stack trace much like panic and oops, but causes no problems and we return to the normal control flow, and can cleanly remove the module afterwards:

insmod dump_stack.ko

Source: kernel_modules/dump_stack.c

16.6.4. WARN_ON

The WARN_ON macro basically just calls dump_stack.

One extra side effect is that we can make it also panic with:

echo 1 > /proc/sys/kernel/panic_on_warn
insmod warn_on.ko

Source: kernel_modules/warn_on.c

Can also be activated with the panic_on_warn boot parameter.

16.6.5. not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

Let’s learn how to diagnose problems with the root filesystem not being found. TODO add a sample panic error message for each error type:

• https://askubuntu.com/questions/41930/kernel-panic-not-syncing-vfs-unable-to-mount-root-fs-on-unknown-block0-0/1048477#1048477

This is the diagnosis procedure.

First, if we remove the following options from the our kernel build:

CONFIG_VIRTIO_BLK=y
CONFIG_VIRTIO_PCI=y

we get a message like this:

<4>[    0.541708] VFS: Cannot open root device "vda" or unknown-block(0,0): error -6
<4>[    0.542035] Please append a correct "root=" boot option; here are the available partitions:
<0>[    0.542562] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)

From the message, we notice that the kernel sees a disk of some sort (vda means a virtio disk), but it could not open it.

This means that the kernel cannot properly read any bytes from the disk.

And afterwards, it has an useless message here are the available partitions:, but of course we have no available partitions, the list is empty, because the kernel cannot even read bytes from the disk, so it definitely cannot understand its filesystems.

This can indicate basically two things:

• on real hardware, it could mean that the hardware is broken. Kind of hard on emulators ;-)

• you didn’t configure the kernel with the option that enables it to read from that kind of disk.

  In our case, disks are virtio devices that QEMU exposes to the guest kernel. This is why removing the options:

  CONFIG_VIRTIO_BLK=y
  CONFIG_VIRTIO_PCI=y

  led to this error.

Now, let’s restore the previously removed virtio options, and instead remove:

CONFIG_EXT4_FS=y

This time, the kernel will be able to read bytes from the device. But it won’t be able to read files from the filesystem, because our filesystem is in ext4 format.

Therefore, this time the error message looks like this:

<4>[    0.585296] List of all partitions:
<4>[    0.585913] fe00          524288 vda
<4>[    0.586123]  driver: virtio_blk
<4>[    0.586471] No filesystem could mount root, tried:
<4>[    0.586497]  squashfs
<4>[    0.586724]
<0>[    0.587360] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(254,0)

In this case, we see that the kernel did manage to read from the vda disk! It even told us how: by using the driver: virtio_blk.

However, it then went through the list of all filesystem types it knows how to read files from, in our case just squashf, and none of those worked, because our partition is an ext4 partition.

Finally, the last possible error is that we simply passed the wrong root= kernel CLI option. For example, if we hack our command to pass:

root=/dev/vda2

which does not even exist since /dev/vda is a raw non-partitioned ext4 image, then boot fails with a message:

<4>[    0.608475] Please append a correct "root=" boot option; here are the available partitions:
<4>[    0.609563] fe00          524288 vda
<4>[    0.609723]  driver: virtio_blk
<0>[    0.610433] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(254,2)

This one is easy, because the kernel tells us clearly which partitions it would have been able to understand. In our case /dev/vda.

Once all those problems are solved, in the working setup, we finally see something like:

<6>[    0.636129] EXT4-fs (vda): mounted filesystem with ordered data mode. Opts: (null)
<6>[    0.636700] VFS: Mounted root (ext4 filesystem) on device 254:0.

Tested on LKMC 863a373a30cd3c7982e3e453c4153f85133b17a9, Linux kernel 5.4.3.

Bibliography:

• https://askubuntu.com/questions/41930/kernel-panic-not-syncing-vfs-unable-to-mount-root-fs-on-unknown-block0-0/1048477#1048477

• https://unix.stackexchange.com/questions/414655/not-syncing-vfs-unable-to-mount-root-fs-on-unknown-block0-0/603197#603197

• https://stackoverflow.com/questions/63277677/i-meet-a-problem-when-i-encountered-in-the-fs-mode-of-running-gem5/63278487#63278487 summary only

16.7. Pseudo filesystems

Pseudo filesystems are filesystems that don’t represent actual files in a hard disk, but rather allow us to do special operations on filesystem-related system calls.

What each pseudo-file does for each related system call does is defined by its File operations.

Bibliography:

• https://superuser.com/questions/1198292/what-is-a-pseudo-file-system-in-linux

• https://en.wikipedia.org/wiki/Synthetic_file_system

16.7.1. debugfs

Debugfs is the simplest pseudo filesystem to play around with:

./debugfs.sh
echo $?

Outcome: the test passes:

0

Sources:

• kernel_modules/debugfs.c

• rootfs_overlay/lkmc/debugfs.sh

Debugfs is made specifically to help test kernel stuff. Just mount, set File operations, and we are done.

For this reason, it is the filesystem that we use whenever possible in our tests.

debugfs.sh explicitly mounts a debugfs at a custom location, but the most common mount point is /sys/kernel/debug.

This mount not done automatically by the kernel however: we, like most distros, do it from userland with our fstab.

Debugfs support requires the kernel to be compiled with CONFIG_DEBUG_FS=y.

Only the more basic file operations can be implemented in debugfs, e.g. mmap never gets called:

• https://patchwork.kernel.org/patch/9252557/

• https://github.com/torvalds/linux/blob/v4.9/fs/debugfs/file.c#L212

Bibliography: https://github.com/chadversary/debugfs-tutorial

16.7.2. procfs

Procfs is just another fops entry point:

./procfs.sh
echo $?

Outcome: the test passes:

0

Procfs is a little less convenient than debugfs, but is more used in serious applications.

Procfs can run all system calls, including ones that debugfs can’t, e.g. mmap.

Sources:

• kernel_modules/procfs.c

• rootfs_overlay/lkmc/procfs.sh

Bibliography:

• https://superuser.com/questions/619955/how-does-proc-work/1442571#1442571

• https://stackoverflow.com/questions/8516021/proc-create-example-for-kernel-module/18924359#18924359

16.7.2.1. /proc/version

Its data is shared with uname(), which is a POSIX C function and has a Linux syscall to back it up.

Where the data comes from and how to modify it:

• https://unix.stackexchange.com/questions/136959/where-does-uname-get-its-information-from/485962#485962

• https://stackoverflow.com/questions/23424174/how-to-customize-or-remove-extra-linux-kernel-version-details-shown-at-boot

In this repo, leaking host information, and to make builds more reproducible, we are setting:

• user and date to dummy values with KBUILD_BUILD_USER and KBUILD_BUILD_TIMESTAMP

• hostname to the kernel git commit with KBUILD_BUILD_HOST and KBUILD_BUILD_VERSION

A sample result is:

Linux version 4.19.0-dirty (lkmc@84df9525b0c27f3ebc2ebb1864fa62a97fdedb7d) (gcc version 6.4.0 (Buildroot 2018.05-00002-gbc60382b8f)) #1 SMP Thu Jan 1 00:00:00 UTC 1970

16.7.3. sysfs

Sysfs is more restricted than procfs, as it does not take an arbitrary file_operations:

./sysfs.sh
echo $?

Outcome: the test passes:

0

Sources:

• kernel_modules/sysfs.c

• rootfs_overlay/lkmc/sysfs.sh

Vs procfs:

• https://unix.stackexchange.com/questions/4884/what-is-the-difference-between-procfs-and-sysfs/382315#382315

• https://stackoverflow.com/questions/37237835/how-to-attach-file-operations-to-sysfs-attribute-in-platform-driver

• https://serverfault.com/questions/65261/linux-proc-sys-kernel-vs-sys-kernel

You basically can only do open, close, read, write, and lseek on sysfs files.

It is similar to a seq_file file operation, except that write is also implemented.

TODO: what are those kobject structs? Make a more complex example that shows what they can do.

Bibliography:

• https://github.com/t3rm1n4l/kern-dev-tutorial/blob/1f036ef40fc4378f5c8d2842e55bcea7c6f8894a/05-sysfs/sysfs.c

• https://www.kernel.org/doc/Documentation/kobject.txt

• https://www.quora.com/What-are-kernel-objects-Kobj

• http://www.makelinux.net/ldd3/chp-14-sect-1

• https://www.win.tue.nl/~aeb/linux/lk/lk-13.html

16.7.4. Character devices

Character devices can have arbitrary File operations associated to them:

./character_device.sh
echo $?

Outcome: the test passes:

0

Sources:

• rootfs_overlay/lkmc/character_device.sh

• rootfs_overlay/lkmc/mknoddev.sh

• kernel_modules/character_device.c

Unlike procfs entires, character device files are created with userland mknod or mknodat syscalls:

mknod </dev/path_to_dev> c <major> <minor>

Intuitively, for physical devices like keyboards, the major number maps to which driver, and the minor number maps to which device it is.

A single driver can drive multiple compatible devices.

The major and minor numbers can be observed with:

ls -l /dev/urandom

Output:

crw-rw-rw-    1 root     root        1,   9 Jun 29 05:45 /dev/urandom

which means:

• c (first letter): this is a character device. Would be b for a block device.

• 1, 9: the major number is 1, and the minor 9

To avoid device number conflicts when registering the driver we:

• ask the kernel to allocate a free major number for us with: register_chrdev(0

• find ouf which number was assigned by grepping /proc/devices for the kernel module name

Bibliography: https://unix.stackexchange.com/questions/37829/understanding-character-device-or-character-special-files/371758#371758

16.7.4.1. Automatically create character device file on insmod

And also destroy it on rmmod:

./character_device_create.sh
echo $?

Outcome: the test passes:

0

Sources:

• kernel_modules/character_device_create.c

• rootfs_overlay/lkmc/character_device_create.sh

Bibliography: https://stackoverflow.com/questions/5970595/how-to-create-a-device-node-from-the-init-module-code-of-a-linux-kernel-module/45531867#45531867

16.8. Pseudo files

16.8.1. File operations

File operations are the main method of userland driver communication.

struct file_operations determines what the kernel will do on filesystem system calls of Pseudo filesystems.

This example illustrates the most basic system calls: open, read, write, close and lseek:

./fops.sh
echo $?

Outcome: the test passes:

0

Sources:

• kernel_modules/fops.c

• rootfs_overlay/lkmc/fops.sh

Then give this a try:

sh -x ./fops.sh

We have put printks on each fop, so this allows you to see which system calls are being made for each command.

No, there no official documentation: https://stackoverflow.com/questions/15213932/what-are-the-struct-file-operations-arguments

16.8.2. seq_file

Writing trivial read File operations is repetitive and error prone. The seq_file API makes the process much easier for those trivial cases:

./seq_file.sh
echo $?

Outcome: the test passes:

0

Sources:

• kernel_modules/seq_file.c

• rootfs_overlay/lkmc/seq_file.sh

In this example we create a debugfs file that behaves just like a file that contains:

0
1
2

However, we only store a single integer in memory and calculate the file on the fly in an iterator fashion.

seq_file does not provide write: https://stackoverflow.com/questions/30710517/how-to-implement-a-writable-proc-file-by-using-seq-file-in-a-driver-module

Bibliography:

• Documentation/filesystems/seq_file.txt

• https://stackoverflow.com/questions/25399112/how-to-use-a-seq-file-in-linux-modules

16.8.2.1. seq_file single_open

If you have the entire read output upfront, single_open is an even more convenient version of seq_file:

./seq_file.sh
echo $?

Outcome: the test passes:

0

Sources:

• kernel_modules/seq_file_single_open.c

• rootfs_overlay/lkmc/seq_file_single_open.sh

This example produces a debugfs file that behaves like a file that contains:

ab
cd

16.8.3. poll

The poll system call allows an user process to do a non-busy wait on a kernel event.

Sources:

• kernel_modules/poll.c

• rootfs_overlay/lkmc/poll.sh

Example:

./poll.sh

Outcome: jiffies gets printed to stdout every second from userland, e.g.:

poll
<6>[    4.275305] poll
<6>[    4.275580] return POLLIN
revents = 1
POLLIN n=10 buf=4294893337
poll
<6>[    4.276627] poll
<6>[    4.276911] return 0
<6>[    5.271193] wake_up
<6>[    5.272326] poll
<6>[    5.273207] return POLLIN
revents = 1
POLLIN n=10 buf=4294893588
poll
<6>[    5.276367] poll
<6>[    5.276618] return 0
<6>[    6.275178] wake_up
<6>[    6.276370] poll
<6>[    6.277269] return POLLIN
revents = 1
POLLIN n=10 buf=4294893839

Force the poll file_operation to return 0 to see what happens more clearly:

./poll.sh pol0=1

Sample output:

poll
<6>[   85.674801] poll
<6>[   85.675788] return 0
<6>[   86.675182] wake_up
<6>[   86.676431] poll
<6>[   86.677373] return 0
<6>[   87.679198] wake_up
<6>[   87.680515] poll
<6>[   87.681564] return 0
<6>[   88.683198] wake_up

From this we see that control is not returned to userland: the kernel just keeps calling the poll file_operation again and again.

Typically, we are waiting for some hardware to make some piece of data available available to the kernel.

The hardware notifies the kernel that the data is ready with an interrupt.

To simplify this example, we just fake the hardware interrupts with a kthread that sleeps for a second in an infinite loop.

Bibliography:

• https://stackoverflow.com/questions/30035776/how-to-add-poll-function-to-the-kernel-module-code/44645336#44645336

• https://stackoverflow.com/questions/30234496/why-do-we-need-to-call-poll-wait-in-poll/44645480#44645480

16.8.4. ioctl

The ioctl system call is the best way to pass an arbitrary number of parameters to the kernel in a single go:

./ioctl.sh
echo $?

Outcome: the test passes:

0

Sources:

• kernel_modules/ioctl.c

• lkmc/ioctl.h

• userland/kernel_modules/ioctl.c

• rootfs_overlay/lkmc/ioctl.sh

ioctl is one of the most important methods of communication with real device drivers, which often take several fields as input.

ioctl takes as input:

• an integer request : it usually identifies what type of operation we want to do on this call

• an untyped pointer to memory: can be anything, but is typically a pointer to a struct

  The type of the struct often depends on the request input

  This struct is defined on a uapi-style C header that is used both to compile the kernel module and the userland executable.

  The fields of this struct can be thought of as arbitrary input parameters.

And the output is:

• an integer return value. man ioctl documents:

  Usually, on success zero is returned. A few ioctl() requests use the return value as an output parameter and return a nonnegative value on success. On error, -1 is returned, and errno is set appropriately.

• the input pointer data may be overwritten to contain arbitrary output

Bibliography:

• https://stackoverflow.com/questions/2264384/how-do-i-use-ioctl-to-manipulate-my-kernel-module/44613896#44613896

• https://askubuntu.com/questions/54239/problem-with-ioctl-in-a-simple-kernel-module/926675#926675

16.8.5. mmap

The mmap system call allows us to share memory between user and kernel space without copying:

./mmap.sh
echo $?

Outcome: the test passes:

0

Sources:

• kernel_modules/mmap.c

• userland/kernel_modules/mmap.c

• rootfs_overlay/lkmc/mmap.sh

In this example, we make a tiny 4 byte kernel buffer available to user-space, and we then modify it on userspace, and check that the kernel can see the modification.

mmap, like most more complex File operations, does not work with debugfs as of 4.9, so we use a procfs file for it.

Example adapted from: https://coherentmusings.wordpress.com/2014/06/10/implementing-mmap-for-transferring-data-from-user-space-to-kernel-space/

Bibliography:

• https://stackoverflow.com/questions/10760479/mmap-kernel-buffer-to-user-space/10770582#10770582

• https://stackoverflow.com/questions/1623008/allocating-memory-for-user-space-from-kernel-thread

• https://stackoverflow.com/questions/6967933/mmap-mapping-in-user-space-a-kernel-buffer-allocated-with-kmalloc

• https://github.com/jeremytrimble/ezdma

• https://github.com/simonjhall/dma

• https://github.com/ikwzm/udmabuf

16.8.6. Anonymous inode

Anonymous inodes allow getting multiple file descriptors from a single filesystem entry, which reduces namespace pollution compared to creating multiple device files:

./anonymous_inode.sh
echo $?

Outcome: the test passes:

0

Sources:

• kernel_modules/anonymous_inode.c

• lkmc/anonymous_inode.h

• userland/kernel_modules/anonymous_inode.c

• rootfs_overlay/lkmc/anonymous_inode.sh

This example gets an anonymous inode via ioctl from a debugfs entry by using anon_inode_getfd.

Reads to that inode return the sequence: 1, 10, 100, …​ 10000000, 1, 100, …​

Bibliography: https://stackoverflow.com/questions/4508998/what-is-an-anonymous-inode-in-linux/44388030#44388030

16.8.7. netlink sockets

Netlink sockets offer a socket API for kernel / userland communication:

./netlink.sh
echo $?

Outcome: the test passes:

0

Sources:

• kernel_modules/netlink.c

• lkmc/netlink.h

• userland/kernel_modules/netlink.c

• rootfs_overlay/lkmc/netlink.sh

Launch multiple user requests in parallel to stress our socket:

insmod netlink.ko sleep=1
for i in `seq 16`; do ./netlink.out & done

TODO: what is the advantage over read, write and poll? https://stackoverflow.com/questions/16727212/how-netlink-socket-in-linux-kernel-is-different-from-normal-polling-done-by-appl

Bibliography:

• https://stackoverflow.com/questions/3299386/how-to-use-netlink-socket-to-communicate-with-a-kernel-module

• https://en.wikipedia.org/wiki/Netlink

16.9. kthread

Kernel threads are managed exactly like userland threads; they also have a backing task_struct, and are scheduled with the same mechanism:

insmod kthread.ko

Source: kernel_modules/kthread.c

Outcome: dmesg counts from 0 to 9 once every second infinitely many times:

0
1
2
...
8
9
0
1
2
...

The count stops when we rmmod:

rmmod kthread

The sleep is done with usleep_range, see: Section 16.9.2, “sleep”.

Bibliography:

• https://stackoverflow.com/questions/10177641/proper-way-of-handling-threads-in-kernel

• https://stackoverflow.com/questions/4084708/how-to-wait-for-a-linux-kernel-thread-kthreadto-exit

16.9.1. kthreads

Let’s launch two threads and see if they actually run in parallel:

insmod kthreads.ko

Source: kernel_modules/kthreads.c

Outcome: two threads count to dmesg from 0 to 9 in parallel.

Each line has output of form:

<thread_id> <count>

Possible very likely outcome:

1 0
2 0
1 1
2 1
1 2
2 2
1 3
2 3

The threads almost always interleaved nicely, thus confirming that they are actually running in parallel.

16.9.2. sleep

Count to dmesg every one second from 0 up to n - 1:

insmod sleep.ko n=5

Source: kernel_modules/sleep.c

The sleep is done with a call to usleep_range directly inside module_init for simplicity.

Bibliography:

• https://stackoverflow.com/questions/15994603/how-to-sleep-in-the-linux-kernel/44153288#44153288

• https://github.com/torvalds/linux/blob/v4.17/Documentation/timers/timers-howto.txt

16.9.3. Workqueues

A more convenient front-end for kthread:

insmod workqueue_cheat.ko

Outcome: count from 0 to 9 infinitely many times

Stop counting:

rmmod workqueue_cheat

Source: kernel_modules/workqueue_cheat.c

The workqueue thread is killed after the worker function returns.

We can’t call the module just workqueue.c because there is already a built-in with that name: https://unix.stackexchange.com/questions/364956/how-can-insmod-fail-with-kernel-module-is-already-loaded-even-is-lsmod-does-not

Bibliography: https://github.com/torvalds/linux/blob/v4.17/Documentation/core-api/workqueue.rst

16.9.3.1. Workqueue from workqueue

Count from 0 to 9 every second infinitely many times by scheduling a new work item from a work item:

insmod work_from_work.ko

Stop:

rmmod work_from_work

The sleep is done indirectly through: queue_delayed_work, which waits the specified time before scheduling the work.

Source: kernel_modules/work_from_work.c

16.9.4. schedule

Let’s block the entire kernel! Yay:

./run --eval-after 'dmesg -n 1;insmod schedule.ko schedule=0'

Outcome: the system hangs, the only way out is to kill the VM.

Source: kernel_modules/schedule.c

kthreads only allow interrupting if you call schedule(), and the schedule=0 kernel module parameter turns it off.

Sleep functions like usleep_range also end up calling schedule.

If we allow schedule() to be called, then the system becomes responsive:

./run --eval-after 'dmesg -n 1;insmod schedule.ko schedule=1'

and we can observe the counting with:

dmesg -w

The system also responds if we add another core:

./run --cpus 2 --eval-after 'dmesg -n 1;insmod schedule.ko schedule=0'

16.9.5. Wait queues

Wait queues are a way to make a thread sleep until an event happens on the queue:

insmod wait_queue.c

Dmesg output:

0 0
1 0
2 0
# Wait one second.
0 1
1 1
2 1
# Wait one second.
0 2
1 2
2 2
...

Stop the count:

rmmod wait_queue

Source: kernel_modules/wait_queue.c

This example launches three threads:

• one thread generates events every with wake_up

• the other two threads wait for that with wait_event, and print a dmesg when it happens.

  The wait_event macro works a bit like:

  while (!cond)
      sleep_until_event

16.10. Timers

Count from 0 to 9 infinitely many times in 1 second intervals using timers:

insmod timer.ko

Stop counting:

rmmod timer

Source: kernel_modules/timer.c

Timers are callbacks that run when an interrupt happens, from the interrupt context itself.

Therefore they produce more accurate timing than thread scheduling, which is more complex, but you can’t do too much work inside of them.

Bibliography:

• https://stackoverflow.com/questions/10812858/timers-in-linux-device-drivers

• https://gist.github.com/yagihiro/310149

16.11. IRQ

16.11.1. irq.ko

Brute force monitor every shared interrupt that will accept us:

./run --eval-after 'insmod irq.ko' --graphic

Source: kernel_modules/irq.c.

Now try the following:

• press a keyboard key and then release it after a few seconds

• press a mouse key, and release it after a few seconds

• move the mouse around

Outcome: dmesg shows which IRQ was fired for each action through messages of type:

handler irq = 1 dev = 250

dev is the character device for the module and never changes, as can be confirmed by:

grep lkmc_irq /proc/devices

The IRQs that we observe are:

• 1 for keyboard press and release.

  If you hold the key down for a while, it starts firing at a constant rate. So this happens at the hardware level!

• 12 mouse actions

This only works if for IRQs for which the other handlers are registered as IRQF_SHARED.

We can see which ones are those, either via dmesg messages of type:

genirq: Flags mismatch irq 0. 00000080 (myirqhandler0) vs. 00015a00 (timer)
request_irq irq = 0 ret = -16
request_irq irq = 1 ret = 0

which indicate that 0 is not, but 1 is, or with:

cat /proc/interrupts

which shows:

  0:         31   IO-APIC   2-edge      timer
  1:          9   IO-APIC   1-edge      i8042, myirqhandler0

so only 1 has myirqhandler0 attached but not 0.

The QEMU monitor also has some interrupt statistics for x86_64:

./qemu-monitor info irq

TODO: properly understand how each IRQ maps to what number.

16.11.2. dummy-irq

The Linux kernel v4.16 mainline also has a dummy-irq module at drivers/misc/dummy-irq.c for monitoring a single IRQ.

We build it by default with:

CONFIG_DUMMY_IRQ=m

And then you can do

./run --graphic

and in guest:

modprobe dummy-irq irq=1

Outcome: when you click a key on the keyboard, dmesg shows:

dummy-irq: interrupt occurred on IRQ 1

However, this module is intended to fire only once as can be seen from its source:

    static int count = 0;

    if (count == 0) {
        printk(KERN_INFO "dummy-irq: interrupt occurred on IRQ %d\n",
            irq);
        count++;
    }

and furthermore interrupt 1 and 12 happen immediately TODO why, were they somehow pending?

16.11.3. /proc/interrupts

In the guest with QEMU graphic mode:

watch -n 1 cat /proc/interrupts

Then see how clicking the mouse and keyboard affect the interrupt counts.

This confirms that:

• 1: keyboard

• 12: mouse click and drags

The module also shows which handlers are registered for each IRQ, as we have observed at irq.ko

When in text mode, we can also observe interrupt line 4 with handler ttyS0 increase continuously as IO goes through the UART.

16.12. Kernel utility functions

https://github.com/torvalds/linux/blob/v4.17/Documentation/core-api/kernel-api.rst

16.12.1. kstrto

Convert a string to an integer:

./kstrto.sh
echo $?

Outcome: the test passes:

0

Sources:

• kernel_modules/kstrto.c

• rootfs_overlay/lkmc/kstrto.sh

Bibliography: https://stackoverflow.com/questions/6139493/how-convert-char-to-int-in-linux-kernel/49811658#49811658

16.12.2. virt_to_phys

Convert a virtual address to physical:

insmod virt_to_phys.ko
cat /sys/kernel/debug/lkmc_virt_to_phys

Source: kernel_modules/virt_to_phys.c

Sample output:

*kmalloc_ptr = 0x12345678
kmalloc_ptr = ffff88000e169ae8
virt_to_phys(kmalloc_ptr) = 0xe169ae8
static_var = 0x12345678
&static_var = ffffffffc0002308
virt_to_phys(&static_var) = 0x40002308

We can confirm that the kmalloc_ptr translation worked with:

./qemu-monitor 'xp 0xe169ae8'

which reads four bytes from a given physical address, and gives the expected:

000000000e169ae8: 0x12345678

TODO it only works for kmalloc however, for the static variable:

./qemu-monitor 'xp 0x40002308'

it gave a wrong value of 00000000.

Bibliography:

• https://stackoverflow.com/questions/5748492/is-there-any-api-for-determining-the-physical-address-from-virtual-address-in-li/45128487#45128487

• https://stackoverflow.com/questions/39134990/mmap-of-dev-mem-fails-with-invalid-argument-for-virt-to-phys-address-but-addre/45127582#45127582

• https://stackoverflow.com/questions/43325205/can-we-use-virt-to-phys-for-user-space-memory-in-kernel-module

16.12.2.1. Userland physical address experiments

Only tested in x86_64.

The Linux kernel exposes physical addresses to userland through:

• /proc/<pid>/maps

• /proc/<pid>/pagemap

• /dev/mem

In this section we will play with them.

The following files contain examples to access that data and test it out:

• lkmc/pagemap.h

• rootfs_overlay/lkmc/virt_to_phys.sh

• userland/linux/virt_to_phys_user.c

• userland/posix/virt_to_phys_test.c

First get a virtual address to play with:

./posix/virt_to_phys_test.out &

Source: userland/posix/virt_to_phys_test.c

Sample output:

vaddr 0x600800
pid 110

The program:

• allocates a volatile variable and sets is value to 0x12345678

• prints the virtual address of the variable, and the program PID

• runs a while loop until until the value of the variable gets mysteriously changed somehow, e.g. by nasty tinkerers like us

Then, translate the virtual address to physical using /proc/<pid>/maps and /proc/<pid>/pagemap:

./linux/virt_to_phys_user.out 110 0x600800

Sample output physical address:

0x7c7b800

Now we can verify that linux/virt_to_phys_user.out gave the correct physical address in the following ways:

• QEMU xp

• /dev/mem

Bibliography:

• https://stackoverflow.com/questions/17021214/decode-proc-pid-pagemap-entry/45126141#45126141

• https://stackoverflow.com/questions/6284810/proc-pid-pagemaps-and-proc-pid-maps-linux/45500208#45500208

16.12.2.1.1. QEMU xp

The xp QEMU monitor command reads memory at a given physical address.

First launch linux/virt_to_phys_user.out as described at Userland physical address experiments.

On a second terminal, use QEMU to read the physical address:

./qemu-monitor 'xp 0x7c7b800'

Output:

0000000007c7b800: 0x12345678

Yes!!! We read the correct value from the physical address.

We could not find however to write to memory from the QEMU monitor, boring.

16.12.2.1.2. /dev/mem

/dev/mem exposes access to physical addresses, and we use it through the convenient devmem BusyBox utility.

First launch linux/virt_to_phys_user.out as described at Userland physical address experiments.

Next, read from the physical address:

devmem 0x7c7b800

Possible output:

Memory mapped at address 0x7ff7dbe01000.
Value at address 0X7C7B800 (0x7ff7dbe01800): 0x12345678

which shows that the physical memory contains the expected value 0x12345678.

0x7ff7dbe01000 is a new virtual address that devmem maps to the physical address to be able to read from it.

Modify the physical memory:

devmem 0x7c7b800 w 0x9abcdef0

After one second, we see on the screen:

i 9abcdef0
[1]+  Done                       ./posix/virt_to_phys_test.out

so the value changed, and the while loop exited!

This example requires:

• CONFIG_STRICT_DEVMEM=n, otherwise devmem fails with:

  devmem: mmap: Operation not permitted

• nopat kernel parameter

which we set by default.

Bibliography: https://stackoverflow.com/questions/11891979/how-to-access-mmaped-dev-mem-without-crashing-the-linux-kernel

16.12.2.1.3. pagemap_dump.out

Dump the physical address of all pages mapped to a given process using /proc/<pid>/maps and /proc/<pid>/pagemap.

First launch linux/virt_to_phys_user.out as described at Userland physical address experiments. Suppose that the output was:

# ./posix/virt_to_phys_test.out &
vaddr 0x601048
pid 63
# ./linux/virt_to_phys_user.out 63 0x601048
0x1a61048

Now obtain the page map for the process:

./linux/pagemap_dump.out 63

Sample output excerpt:

vaddr pfn soft-dirty file/shared swapped present library
400000 1ede 0 1 0 1 ./posix/virt_to_phys_test.out
600000 1a6f 0 0 0 1 ./posix/virt_to_phys_test.out
601000 1a61 0 0 0 1 ./posix/virt_to_phys_test.out
602000 2208 0 0 0 1 [heap]
603000 220b 0 0 0 1 [heap]
7ffff78ec000 1fd4 0 1 0 1 /lib/libuClibc-1.0.30.so

Source:

• userland/linux/pagemap_dump.c

• lkmc/pagemap.h

Adapted from: https://github.com/dwks/pagemap/blob/8a25747bc79d6080c8b94eac80807a4dceeda57a/pagemap2.c

Meaning of the flags:

• vaddr: first virtual address of a page the belongs to the process. Notably:

  ./run-toolchain readelf -- -l "$(./getvar userland_build_dir)/posix/virt_to_phys_test.out"

  contains:

    Type           Offset             VirtAddr           PhysAddr
                   FileSiz            MemSiz              Flags  Align
  ...
    LOAD           0x0000000000000000 0x0000000000400000 0x0000000000400000
                   0x000000000000075c 0x000000000000075c  R E    0x200000
    LOAD           0x0000000000000e98 0x0000000000600e98 0x0000000000600e98
                   0x00000000000001b4 0x0000000000000218  RW     0x200000
  
   Section to Segment mapping:
    Segment Sections...
  ...
     02     .interp .hash .dynsym .dynstr .rela.plt .init .plt .text .fini .rodata .eh_frame_hdr .eh_frame
     03     .ctors .dtors .jcr .dynamic .got.plt .data .bss

  from which we deduce that:

  • 400000 is the text segment

  • 600000 is the data segment

• pfn: add three zeroes to it, and you have the physical address.

  Three zeroes is 12 bits which is 4kB, which is the size of a page.

  For example, the virtual address 0x601000 has pfn of 0x1a61, which means that its physical address is 0x1a61000

  This is consistent with what linux/virt_to_phys_user.out told us: the virtual address 0x601048 has physical address 0x1a61048.

  048 corresponds to the three last zeroes, and is the offset within the page.

  Also, this value falls inside 0x601000, which as previously analyzed is the data section, which is the normal location for global variables such as ours.

• soft-dirty: TODO

• file/shared: TODO. 1 seems to indicate that the page can be shared across processes, possibly for read-only pages? E.g. the text segment has 1, but the data has 0.

• swapped: TODO swapped to disk?

• present: TODO vs swapped?

• library: which executable owns that page

This program works in two steps:

• parse the human readable lines lines from /proc/<pid>/maps. This files contains lines of form:

  7ffff7b6d000-7ffff7bdd000 r-xp 00000000 fe:00 658                        /lib/libuClibc-1.0.22.so

  which tells us that:

  • 7f8af99f8000-7f8af99ff000 is a virtual address range that belong to the process, possibly containing multiple pages.

  • /lib/libuClibc-1.0.22.so is the name of the library that owns that memory

• loop over each page of each address range, and ask /proc/<pid>/pagemap for more information about that page, including the physical address

16.13. Linux kernel tracing

Good overviews:

• http://www.brendangregg.com/blog/2015-07-08/choosing-a-linux-tracer.html by Brendan Greg, AKA the master of tracing. Also: https://github.com/brendangregg/perf-tools

• https://jvns.ca/blog/2017/07/05/linux-tracing-systems/

I hope to have examples of all methods some day, since I’m obsessed with visibility.

16.13.1. CONFIG_PROC_EVENTS

Logs proc events such as process creation to a netlink socket.

We then have a userland program that listens to the events and prints them out:

# ./linux/proc_events.out &
# set mcast listen ok
# sleep 2 & sleep 1
fork: parent tid=48 pid=48 -> child tid=79 pid=79
fork: parent tid=48 pid=48 -> child tid=80 pid=80
exec: tid=80 pid=80
exec: tid=79 pid=79
# exit: tid=80 pid=80 exit_code=0
exit: tid=79 pid=79 exit_code=0
echo a
a
#

Source: userland/linux/proc_events.c

TODO: why exit: tid=79 shows after exit: tid=80?

Note how echo a is a Bash built-in, and therefore does not spawn a new process.

TODO: why does this produce no output?

./linux/proc_events.out >f &

• https://stackoverflow.com/questions/6075013/detect-launching-of-programs-on-linux-platform/8255487#8255487

• https://serverfault.com/questions/199654/does-anyone-know-a-simple-way-to-monitor-root-process-spawn

• https://unix.stackexchange.com/questions/260162/how-to-track-newly-created-processes

TODO can you get process data such as UID and process arguments? It seems not since exec_proc_event contains so little data: https://github.com/torvalds/linux/blob/v4.16/include/uapi/linux/cn_proc.h#L80 We could try to immediately read it from /proc, but there is a risk that the process finished and another one took its PID, so it wouldn’t be reliable.

• https://unix.stackexchange.com/questions/163681/print-pids-and-names-of-processes-as-they-are-created/163689 requests process name

• https://serverfault.com/questions/199654/does-anyone-know-a-simple-way-to-monitor-root-process-spawn requests UID

16.13.1.1. CONFIG_PROC_EVENTS aarch64

0111ca406bdfa6fd65a2605d353583b4c4051781 was failing with:

>>> kernel_modules 1.0 Building
/usr/bin/make -j8 -C '/linux-kernel-module-cheat//out/aarch64/buildroot/build/kernel_modules-1.0/user' BR2_PACKAGE_OPENBLAS="" CC="/linux-kernel-module-cheat//out/aarch64/buildroot/host/bin/aarch64-buildroot-linux-uclibc-gcc" LD="/linux-kernel-module-cheat//out/aarch64/buildroot/host/bin/aarch64-buildroot-linux-uclibc-ld"
/linux-kernel-module-cheat//out/aarch64/buildroot/host/bin/aarch64-buildroot-linux-uclibc-gcc  -ggdb3 -fopenmp -O0 -std=c99 -Wall -Werror -Wextra -o 'proc_events.out' 'proc_events.c'
In file included from /linux-kernel-module-cheat//out/aarch64/buildroot/host/aarch64-buildroot-linux-uclibc/sysroot/usr/include/signal.h:329:0,
                 from proc_events.c:12:
/linux-kernel-module-cheat//out/aarch64/buildroot/host/aarch64-buildroot-linux-uclibc/sysroot/usr/include/sys/ucontext.h:50:16: error: field ‘uc_mcontext’ has incomplete type
     mcontext_t uc_mcontext;
                ^~~~~~~~~~~

so we commented it out.

Related threads:

• https://mailman.uclibc-ng.org/pipermail/devel/2018-January/001624.html

• https://github.com/DynamoRIO/dynamorio/issues/2356

If we try to naively update uclibc to 1.0.29 with buildroot_override, which contains the above mentioned patch, clean aarch64 test build fails with:

../utils/ldd.c: In function 'elf_find_dynamic':
../utils/ldd.c:238:12: warning: cast to pointer from integer of different size [-Wint-to-pointer-cast]
     return (void *)byteswap_to_host(dynp->d_un.d_val);
            ^
/tmp/user/20321/cciGScKB.o: In function `process_line_callback':
msgmerge.c:(.text+0x22): undefined reference to `escape'
/tmp/user/20321/cciGScKB.o: In function `process':
msgmerge.c:(.text+0xf6): undefined reference to `poparser_init'
msgmerge.c:(.text+0x11e): undefined reference to `poparser_feed_line'
msgmerge.c:(.text+0x128): undefined reference to `poparser_finish'
collect2: error: ld returned 1 exit status
Makefile.in:120: recipe for target '../utils/msgmerge.host' failed
make[2]: *** [../utils/msgmerge.host] Error 1
make[2]: *** Waiting for unfinished jobs....
/tmp/user/20321/ccF8V8jF.o: In function `process':
msgfmt.c:(.text+0xbf3): undefined reference to `poparser_init'
msgfmt.c:(.text+0xc1f): undefined reference to `poparser_feed_line'
msgfmt.c:(.text+0xc2b): undefined reference to `poparser_finish'
collect2: error: ld returned 1 exit status
Makefile.in:120: recipe for target '../utils/msgfmt.host' failed
make[2]: *** [../utils/msgfmt.host] Error 1
package/pkg-generic.mk:227: recipe for target '/data/git/linux-kernel-module-cheat/out/aarch64/buildroot/build/uclibc-custom/.stamp_built' failed
make[1]: *** [/data/git/linux-kernel-module-cheat/out/aarch64/buildroot/build/uclibc-custom/.stamp_built] Error 2
Makefile:79: recipe for target '_all' failed
make: *** [_all] Error 2

Buildroot master has already moved to uclibc 1.0.29 at f8546e836784c17aa26970f6345db9d515411700, but it is not yet in any tag…​ so I’m not tempted to update it yet just for this.

16.13.2. ftrace

Trace a single function:

cd /sys/kernel/debug/tracing/

# Stop tracing.
echo 0 > tracing_on

# Clear previous trace.
echo > trace

# List the available tracers, and pick one.
cat available_tracers
echo function > current_tracer

# List all functions that can be traced
# cat available_filter_functions
# Choose one.
echo __kmalloc > set_ftrace_filter
# Confirm that only __kmalloc is enabled.
cat enabled_functions

echo 1 > tracing_on

# Latest events.
head trace

# Observe trace continuously, and drain seen events out.
cat trace_pipe &

Sample output:

# tracer: function
#
# entries-in-buffer/entries-written: 97/97   #P:1
#
#                              _-----=> irqs-off
#                             / _----=> need-resched
#                            | / _---=> hardirq/softirq
#                            || / _--=> preempt-depth
#                            ||| /     delay
#           TASK-PID   CPU#  ||||    TIMESTAMP  FUNCTION
#              | |       |   ||||       |         |
            head-228   [000] ....   825.534637: __kmalloc <-load_elf_phdrs
            head-228   [000] ....   825.534692: __kmalloc <-load_elf_binary
            head-228   [000] ....   825.534815: __kmalloc <-load_elf_phdrs
            head-228   [000] ....   825.550917: __kmalloc <-__seq_open_private
            head-228   [000] ....   825.550953: __kmalloc <-tracing_open
            head-229   [000] ....   826.756585: __kmalloc <-load_elf_phdrs
            head-229   [000] ....   826.756627: __kmalloc <-load_elf_binary
            head-229   [000] ....   826.756719: __kmalloc <-load_elf_phdrs
            head-229   [000] ....   826.773796: __kmalloc <-__seq_open_private
            head-229   [000] ....   826.773835: __kmalloc <-tracing_open
            head-230   [000] ....   827.174988: __kmalloc <-load_elf_phdrs
            head-230   [000] ....   827.175046: __kmalloc <-load_elf_binary
            head-230   [000] ....   827.175171: __kmalloc <-load_elf_phdrs

Trace all possible functions, and draw a call graph:

echo 1 > max_graph_depth
echo 1 > events/enable
echo function_graph > current_tracer

Sample output:

# CPU  DURATION                  FUNCTION CALLS
# |     |   |                     |   |   |   |
 0)   2.173 us    |                  } /* ntp_tick_length */
 0)               |                  timekeeping_update() {
 0)   4.176 us    |                    ntp_get_next_leap();
 0)   5.016 us    |                    update_vsyscall();
 0)               |                    raw_notifier_call_chain() {
 0)   2.241 us    |                      notifier_call_chain();
 0) + 19.879 us   |                    }
 0)   3.144 us    |                    update_fast_timekeeper();
 0)   2.738 us    |                    update_fast_timekeeper();
 0) ! 117.147 us  |                  }
 0)               |                  _raw_spin_unlock_irqrestore() {
 0)   4.045 us    |                    _raw_write_unlock_irqrestore();
 0) + 22.066 us   |                  }
 0) ! 265.278 us  |                } /* update_wall_time */

TODO: what do + and ! mean?

Each enable under the events/ tree enables a certain set of functions, the higher the enable more functions are enabled.

TODO: can you get function arguments? https://stackoverflow.com/questions/27608752/does-ftrace-allow-capture-of-system-call-arguments-to-the-linux-kernel-or-only

16.13.2.1. ftrace system calls

https://stackoverflow.com/questions/29840213/how-do-i-trace-a-system-call-in-linux/51856306#51856306

16.13.2.2. trace-cmd

TODO example:

./build-buildroot --config 'BR2_PACKAGE_TRACE_CMD=y'

16.13.3. Kprobes

kprobes is an instrumentation mechanism that injects arbitrary code at a given address in a trap instruction, much like GDB. Oh, the good old kernel. :-)

./build-linux --config 'CONFIG_KPROBES=y'

Then on guest:

insmod kprobe_example.ko
sleep 4 & sleep 4 &'

Outcome: dmesg outputs on every fork:

<_do_fork> pre_handler: p->addr = 0x00000000e1360063, ip = ffffffff810531d1, flags = 0x246
<_do_fork> post_handler: p->addr = 0x00000000e1360063, flags = 0x246
<_do_fork> pre_handler: p->addr = 0x00000000e1360063, ip = ffffffff810531d1, flags = 0x246
<_do_fork> post_handler: p->addr = 0x00000000e1360063, flags = 0x246

Source: kernel_modules/kprobe_example.c

TODO: it does not work if I try to immediately launch sleep, why?

insmod kprobe_example.ko
sleep 4 & sleep 4 &

I don’t think your code can refer to the surrounding kernel code however: the only visible thing is the value of the registers.

You can then hack it up to read the stack and read argument values, but do you really want to?

There is also a kprobes + ftrace based mechanism with CONFIG_KPROBE_EVENTS=y which does read the memory for us based on format strings that indicate type…​ https://github.com/torvalds/linux/blob/v4.16/Documentation/trace/kprobetrace.txt Horrendous. Used by: https://github.com/brendangregg/perf-tools/blob/98d42a2a1493d2d1c651a5c396e015d4f082eb20/execsnoop

Bibliography:

• https://github.com/torvalds/linux/blob/v4.16/Documentation/kprobes.txt

• https://github.com/torvalds/linux/blob/v4.17/samples/kprobes/kprobe_example.c

16.13.4. Count boot instructions

TODO: didn’t port during refactor after 3b0a343647bed577586989fb702b760bd280844a. Reimplementing should not be hard.

• https://www.quora.com/How-many-instructions-does-a-typical-Linux-kernel-boot-take

• https://github.com/cirosantilli/chat/issues/31

• https://rwmj.wordpress.com/2016/03/17/tracing-qemu-guest-execution/

• qemu/docs/tracing.txt and qemu/docs/replay.txt

• https://stackoverflow.com/questions/39149446/how-to-use-qemus-simple-trace-backend/46497873#46497873

Results (boot not excluded) are shown at: Table 1, “Boot instruction counts for various setups”

Table 1. Boot instruction counts for various setups

Commit	Arch	Simulator	Instruction count
7228f75ac74c896417fb8c5ba3d375a14ed4d36b	arm	QEMU	680k
7228f75ac74c896417fb8c5ba3d375a14ed4d36b	arm	gem5 AtomicSimpleCPU	160M
7228f75ac74c896417fb8c5ba3d375a14ed4d36b	arm	gem5 HPI	155M
7228f75ac74c896417fb8c5ba3d375a14ed4d36b	x86_64	QEMU	3M
7228f75ac74c896417fb8c5ba3d375a14ed4d36b	x86_64	gem5 AtomicSimpleCPU	528M

QEMU:

./trace-boot --arch x86_64

sample output:

instructions 1833863
entry_address 0x1000000
instructions_firmware 20708

gem5:

./run --arch aarch64 --emulator gem5 --eval 'm5 exit'
# Or:
# ./run --arch aarch64 --emulator gem5 --eval 'm5 exit' -- --cpu-type=HPI --caches
./gem5-stat --arch aarch64 sim_insts

Notes:

• 0x1000000 is the address where QEMU puts the Linux kernel at with -kernel in x86.

  It can be found from:

  ./run-toolchain readelf -- -e "$(./getvar vmlinux)" | grep Entry

  TODO confirm further. If I try to break there with:

  ./run-gdb *0x1000000

  but I have no corresponding source line. Also note that this line is not actually the first line, since the kernel messages such as early console in extract_kernel have already shown on screen at that point. This does not break at all:

  ./run-gdb extract_kernel

  It only appears once on every log I’ve seen so far, checked with grep 0x1000000 trace.txt

  Then when we count the instructions that run before the kernel entry point, there is only about 100k instructions, which is insignificant compared to the kernel boot itself.

  TODO --arch arm and --arch aarch64 does not count firmware instructions properly because the entry point address of the ELF file (ffffff8008080000 for aarch64) does not show up on the trace at all. Tested on f8c0502bb2680f2dbe7c1f3d7958f60265347005.

• We can also discount the instructions after init runs by using readelf to get the initial address of init. One easy way to do that now is to just run:

  ./run-gdb --userland "$(./getvar userland_build_dir)/linux/poweroff.out" main

  And get that from the traces, e.g. if the address is 4003a0, then we search:

  grep -n 4003a0 trace.txt

  I have observed a single match for that instruction, so it must be the init, and there were only 20k instructions after it, so the impact is negligible.

• to disable networking. Is replacing init enough?

  • https://superuser.com/questions/181254/how-do-you-boot-linux-with-networking-disabled

  • https://superuser.com/questions/684005/how-does-one-permanently-disable-gnu-linux-networking/1255015#1255015

  CONFIG_NET=n did not significantly reduce instruction counts, so maybe replacing init is enough.

• gem5 simulates memory latencies. So I think that the CPU loops idle while waiting for memory, and counts will be higher.

16.14. Linux kernel hardening

Make it harder to get hacked and easier to notice that you were, at the cost of some (small?) runtime overhead.

16.14.1. CONFIG_FORTIFY_SOURCE

Detects buffer overflows for us:

./build-linux --config 'CONFIG_FORTIFY_SOURCE=y' --linux-build-id fortify
./build-modules --clean
./build-modules
./build-buildroot
./run --eval-after 'insmod strlen_overflow.ko' --linux-build-id fortify

Possible dmesg output:

strlen_overflow: loading out-of-tree module taints kernel.
detected buffer overflow in strlen
------------[ cut here ]------------

followed by a trace.

You may not get this error because this depends on strlen overflowing at least until the next page: if a random \0 appears soon enough, it won’t blow up as desired.

TODO not always reproducible. Find a more reproducible failure. I could not observe it on:

insmod memcpy_overflow.ko

Source: kernel_modules/strlen_overflow.c

Bibliography: https://www.reddit.com/r/hacking/comments/8h4qxk/what_a_buffer_overflow_in_the_linux_kernel_looks/

16.14.2. Linux security modules

https://en.wikipedia.org/wiki/Linux_Security_Modules

16.14.2.1. SELinux

TODO get a hello world permission control working:

./build-linux \
  --config-fragment linux_config/selinux \
  --linux-build-id selinux \
;
./build-buildroot --config 'BR2_PACKAGE_REFPOLICY=y'
./run --enable-kvm --linux-build-id selinux

Source: linux_config/selinux

This builds:

• BR2_PACKAGE_REFPOLICY, which includes a reference /etc/selinux/config policy: https://github.com/SELinuxProject/refpolicy

  refpolicy in turn depends on:

• BR2_PACKAGE_SETOOLS, which contains tools such as getenforced: https://github.com/SELinuxProject/setools

  setools depends on:

• BR2_PACKAGE_LIBSELINUX, which is the backing userland library

After boot finishes, we see:

Starting auditd: mkdir: invalid option -- 'Z'

which comes from /etc/init.d/S01auditd, because BusyBox' mkdir does not have the crazy -Z option like Ubuntu. That’s amazing!

The kernel logs contain:

SELinux:  Initializing.

Inside the guest we now have:

getenforce

which initially says:

Disabled

TODO: if we try to enforce:

setenforce 1

it does not work and outputs:

setenforce: SELinux is disabled

SELinux requires glibc as mentioned at: [libc-choice].

16.15. User mode Linux

I once got UML running on a minimal Buildroot setup at: https://unix.stackexchange.com/questions/73203/how-to-create-rootfs-for-user-mode-linux-on-fedora-18/372207#372207

But in part because it is dying, I didn’t spend much effort to integrate it into this repo, although it would be a good fit in principle, since it is essentially a virtualization method.

Maybe some brave soul will send a pull request one day.

16.16. UIO

UIO is a kernel subsystem that allows to do certain types of driver operations from userland.

This would be awesome to improve debuggability and safety of kernel modules.

VFIO looks like a newer and better UIO replacement, but there do not exist any examples of how to use it: https://stackoverflow.com/questions/49309162/interfacing-with-qemu-edu-device-via-userspace-i-o-uio-linux-driver

TODO get something interesting working. I currently don’t understand the behaviour very well.

TODO how to ACK interrupts? How to ensure that every interrupt gets handled separately?

TODO how to write to registers. Currently using /dev/mem and lspci.

This example should handle interrupts from userland and print a message to stdout:

./uio_read.sh

TODO: what is the expected behaviour? I should have documented this when I wrote this stuff, and I’m that lazy right now that I’m in the middle of a refactor :-)

UIO interface in a nutshell:

• blocking read / poll: waits until interrupts

• write: call irqcontrol callback. Default: 0 or 1 to enable / disable interrupts.

• mmap: access device memory

Sources:

• userland/kernel_modules/uio_read.c

• rootfs_overlay/lkmc/uio_read.sh

Bibliography:

• https://stackoverflow.com/questions/15286772/userspace-vs-kernel-space-driver

• https://01.org/linuxgraphics/gfx-docs/drm/driver-api/uio-howto.html

• https://stackoverflow.com/questions/7986260/linux-interrupt-handling-in-user-space

• https://yurovsky.github.io/2014/10/10/linux-uio-gpio-interrupt/

• https://github.com/bmartini/zynq-axis/blob/65a3a448fda1f0ea4977adfba899eb487201853d/dev/axis.c

• https://yurovsky.github.io/2014/10/10/linux-uio-gpio-interrupt/

• http://nairobi-embedded.org/uio_example.html that website has QEMU examples for everything as usual. The example has a kernel-side which creates the memory mappings and is used by the user.

• https://stackoverflow.com/questions/49309162/interfacing-with-qemu-edu-device-via-userspace-i-o-uio-linux-driver

• userland driver stability questions:

  • https://stackoverflow.com/questions/8030758/getting-kernel-version-from-linux-kernel-module-at-runtime/45430233#45430233

  • https://stackoverflow.com/questions/37098482/how-to-build-a-linux-kernel-module-so-that-it-is-compatible-with-all-kernel-rele/45429681#45429681

  • https://liquidat.wordpress.com/2007/07/21/linux-kernel-2623-to-have-stable-userspace-driver-api/

16.17. Linux kernel interactive stuff

16.17.1. Linux kernel console fun

Requires Graphics.

You can also try those on the Ctrl-Alt-F3 of your Ubuntu host, but it is much more fun inside a VM!

Stop the cursor from blinking:

echo 0 > /sys/class/graphics/fbcon/cursor_blink

Rotate the console 90 degrees! https://askubuntu.com/questions/237963/how-do-i-rotate-my-display-when-not-using-an-x-server

echo 1 > /sys/class/graphics/fbcon/rotate

Relies on: CONFIG_FRAMEBUFFER_CONSOLE_ROTATION=y.

Documented under: Documentation/fb/.

TODO: font and keymap. Mentioned at: https://cmcenroe.me/2017/05/05/linux-console.html and I think can be done with BusyBox loadkmap and loadfont, we just have to understand their formats, related:

• https://unix.stackexchange.com/questions/177024/remap-keyboard-on-the-linux-console

• https://superuser.com/questions/194202/remapping-keys-system-wide-in-linux-not-just-in-x

16.17.2. Linux kernel magic keys

Requires Graphics.

Let’s have some fun.

I think most are implemented under:

drivers/tty

TODO find all.

Scroll up / down the terminal:

Shift-PgDown
Shift-PgUp

Or inside ./qemu-monitor:

sendkey shift-pgup
sendkey shift-pgdown

16.17.2.1. Ctrl Alt Del

If you run in QEMU graphic mode:

./run --graphic

and then from the graphic window you enter the keys:

Ctrl-Alt-Del

then this runs the following command on the guest:

/sbin/reboot

This is enabled from our rootfs_overlay/etc/inittab:

::ctrlaltdel:/sbin/reboot

This leads Linux to try to reboot, and QEMU shutdowns due to the -no-reboot option which we set by default for, see: Section 16.6.1.3, “Exit emulator on panic”.

Here is a minimal example of Ctrl Alt Del:

./run --kernel-cli 'init=/lkmc/linux/ctrl_alt_del.out' --graphic

Source: userland/linux/ctrl_alt_del.c

When you hit Ctrl-Alt-Del in the guest, our tiny init handles a SIGINT sent by the kernel and outputs to stdout:

cad

To map between man 2 reboot and the uClibc RB_* magic constants see:

less "$(./getvar buildroot_build_build_dir)"/uclibc-*/include/sys/reboot.h"

The procfs mechanism is documented at:

less linux/Documentation/sysctl/kernel.txt

which says:

When the value in this file is 0, ctrl-alt-del is trapped and
sent to the init(1) program to handle a graceful restart.
When, however, the value is > 0, Linux's reaction to a Vulcan
Nerve Pinch (tm) will be an immediate reboot, without even
syncing its dirty buffers.

Note: when a program (like dosemu) has the keyboard in 'raw'
mode, the ctrl-alt-del is intercepted by the program before it
ever reaches the kernel tty layer, and it's up to the program
to decide what to do with it.

Under the hood, behaviour is controlled by the reboot syscall:

man 2 reboot

reboot system calls can set either of the these behaviours for Ctrl-Alt-Del:

• do a hard shutdown syscall. Set in uClibc C code with:

  reboot(RB_ENABLE_CAD)

  or from procfs with:

  echo 1 > /proc/sys/kernel/ctrl-alt-del

  Done by BusyBox' reboot -f.

• send a SIGINT to the init process. This is what BusyBox' init does, and it then execs the string set in inittab.

  Set in uclibc C code with:

  reboot(RB_DISABLE_CAD)

  or from procfs with:

  echo 0 > /proc/sys/kernel/ctrl-alt-del

  Done by BusyBox' reboot.

When a BusyBox init is with the signal, it prints the following lines:

The system is going down NOW!
Sent SIGTERM to all processes
Sent SIGKILL to all processes
Requesting system reboot

On busybox-1.29.2’s init at init/init.c we see how the kill signals are sent:

static void run_shutdown_and_kill_processes(void)
{
	/* Run everything to be run at "shutdown".  This is done _prior_
	 * to killing everything, in case people wish to use scripts to
	 * shut things down gracefully... */
	run_actions(SHUTDOWN);

	message(L_CONSOLE | L_LOG, "The system is going down NOW!");

	/* Send signals to every process _except_ pid 1 */
	kill(-1, SIGTERM);
	message(L_CONSOLE, "Sent SIG%s to all processes", "TERM");
	sync();
	sleep(1);

	kill(-1, SIGKILL);
	message(L_CONSOLE, "Sent SIG%s to all processes", "KILL");
	sync();
	/*sleep(1); - callers take care about making a pause */
}

and run_shutdown_and_kill_processes is called from:

/* The SIGPWR/SIGUSR[12]/SIGTERM handler */
static void halt_reboot_pwoff(int sig) NORETURN;
static void halt_reboot_pwoff(int sig)

which also prints the final line:

	message(L_CONSOLE, "Requesting system %s", m);

which is set as the signal handler via TODO.

Bibliography:

• https://superuser.com/questions/193652/does-linux-have-a-ctrlaltdel-equivalent/1324415#1324415

• https://unix.stackexchange.com/questions/42573/meaning-and-commands-for-ctrlaltdel/444969#444969

16.17.2.2. SysRq

We cannot test these actual shortcuts on QEMU since the host captures them at a lower level, but from:

./qemu-monitor

we can for example crash the system with:

sendkey alt-sysrq-c

Same but boring because no magic key:

echo c > /proc/sysrq-trigger

Implemented in:

drivers/tty/sysrq.c

On your host, on modern systems that don’t have the SysRq key you can do:

Alt-PrtSc-space

which prints a message to dmesg of type:

sysrq: SysRq : HELP : loglevel(0-9) reboot(b) crash(c) terminate-all-tasks(e) memory-full-oom-kill(f) kill-all-tasks(i) thaw-filesystems(j) sak(k) show-backtrace-all-active-cpus(l) show-memory-usage(m) nice-all-RT-tasks(n) poweroff(o) show-registers(p) show-all-timers(q) unraw(r) sync(s) show-task-states(t) unmount(u) show-blocked-tasks(w) dump-ftrace-buffer(z)

Individual SysRq can be enabled or disabled with the bitmask:

/proc/sys/kernel/sysrq

The bitmask is documented at:

less linux/Documentation/admin-guide/sysrq.rst

Bibliography: https://en.wikipedia.org/wiki/Magic_SysRq_key

16.17.3. TTY

In order to play with TTYs, do this:

printf '
tty2::respawn:/sbin/getty -n -L -l /lkmc/loginroot.sh tty2 0 vt100
tty3::respawn:-/bin/sh
tty4::respawn:/sbin/getty 0 tty4
tty63::respawn:-/bin/sh
::respawn:/sbin/getty -L ttyS0 0 vt100
::respawn:/sbin/getty -L ttyS1 0 vt100
::respawn:/sbin/getty -L ttyS2 0 vt100
# Leave one serial empty.
#::respawn:/sbin/getty -L ttyS3 0 vt100
' >> rootfs_overlay/etc/inittab
./build-buildroot
./run --graphic -- \
  -serial telnet::1235,server,nowait \
  -serial vc:800x600 \
  -serial telnet::1236,server,nowait \
;

and on a second shell:

telnet localhost 1235

We don’t add more TTYs by default because it would spawn more processes, even if we use askfirst instead of respawn.

On the GUI, switch TTYs with:

• Alt-Left or Alt-Right: go to previous / next populated /dev/ttyN TTY. Skips over empty TTYs.

• Alt-Fn: go to the nth TTY. If it is not populated, don’t go there.

• chvt <n>: go to the n-th virtual TTY, even if it is empty: https://superuser.com/questions/33065/console-commands-to-change-virtual-ttys-in-linux-and-openbsd

You can also test this on most hosts such as Ubuntu 18.04, except that when in the GUI, you must use Ctrl-Alt-Fx to switch to another terminal.

Next, we also have the following shells running on the serial ports, hit enter to activate them:

• /dev/ttyS0: first shell that was used to run QEMU, corresponds to QEMU’s -serial mon:stdio.

  It would also work if we used -serial stdio, but:

  • Ctrl-C would kill QEMU instead of going to the guest

  • Ctrl-A C wouldn’t open the QEMU console there

  see also: https://stackoverflow.com/questions/49716931/how-to-run-qemu-with-nographic-and-monitor-but-still-be-able-to-send-ctrlc-to

• /dev/ttyS1: second shell running telnet

• /dev/ttyS2: go on the GUI and enter Ctrl-Alt-2, corresponds to QEMU’s -serial vc. Go back to the main console with Ctrl-Alt-1.

although we cannot change between terminals from there.

Each populated TTY contains a "shell":

• -/bin/sh: goes directly into an sh without a login prompt.

  The trailing dash - can be used on any command. It makes the command that follows take over the TTY, which is what we typically want for interactive shells: https://askubuntu.com/questions/902998/how-to-check-which-tty-am-i-using

  The getty executable however also does this operation and therefore dispenses the -.

• /sbin/getty asks for password, and then gives you an sh

  We can overcome the password prompt with the -l /lkmc/loginroot.sh technique explained at: https://askubuntu.com/questions/902998/how-to-check-which-tty-am-i-using but I don’t see any advantage over -/bin/sh currently.

Identify the current TTY with the command:

tty

Bibliography:

• https://unix.stackexchange.com/questions/270272/how-to-get-the-tty-in-which-bash-is-running/270372

• https://unix.stackexchange.com/questions/187319/how-to-get-the-real-name-of-the-controlling-terminal

• https://unix.stackexchange.com/questions/77796/how-to-get-the-current-terminal-name

• https://askubuntu.com/questions/902998/how-to-check-which-tty-am-i-using

This outputs:

• /dev/console for the initial GUI terminal. But I think it is the same as /dev/tty1, because if I try to do

  tty1::respawn:-/bin/sh

  it makes the terminal go crazy, as if multiple processes are randomly eating up the characters.

• /dev/ttyN for the other graphic TTYs. Note that there are only 63 available ones, from /dev/tty1 to /dev/tty63 (/dev/tty0 is the current one): https://superuser.com/questions/449781/why-is-there-so-many-linux-dev-tty. I think this is determined by:

  #define MAX_NR_CONSOLES 63

  in linux/include/uapi/linux/vt.h.

• /dev/ttySN for the text shells.

  These are Serial ports, see this to understand what those represent physically: https://unix.stackexchange.com/questions/307390/what-is-the-difference-between-ttys0-ttyusb0-and-ttyama0-in-linux/367882#367882

  There are only 4 serial ports, I think this is determined by QEMU. TODO check.

  See also: https://stackoverflow.com/questions/16706423/two-instances-of-busybox-on-separate-serial-lines-ttysn

Get the TTY in bulk for all processes:

./psa.sh

Source: rootfs_overlay/lkmc/psa.sh.

The TTY appears under the TT section, which is enabled by -o tty. This shows the TTY device number, e.g.:

4,1

and we can then confirm it with:

ls -l /dev/tty1

Next try:

insmod kthread.ko

and switch between virtual terminals, to understand that the dmesg goes to whatever current virtual terminal you are on, but not the others, and not to the serial terminals.

Bibliography:

• https://serverfault.com/questions/119736/how-to-enable-multiple-virtual-consoles-on-linux

• https://github.com/mirror/busybox/blob/1_28_3/examples/inittab#L60

• http://web.archive.org/web/20180117124612/http://nairobi-embedded.org/qemu_serial_port_system_console.html

16.17.3.1. Start a getty from outside of init

TODO: https://unix.stackexchange.com/questions/196704/getty-start-from-command-line

TODO: how to place an sh directly on a TTY as well without getty?

If I try the exact same command that the inittab is doing from a regular shell after boot:

/sbin/getty 0 tty1

it fails with:

getty: setsid: Operation not permi