Skip to content

Using Rock 4C plus boards with LTSP

Jump to bottom Edit New page
Jorge Tornero edited this page Jun 7, 2023 · 1 revision

Introduction

Having a LTSP setup with Raspberry Pi as clients is very cool and very easy of setup as well. However, Raspberry Pi market went crazy and we chose Rock 4C+ board to replace them. However, this was not as easy as RPi to setup mainly for the difficulties of getting help from the Rockpi community (nice but not as developed as RPi's)

The main source of problems was the ability of a particular Rock 4C+ OS image able to modify uboot parameters easily and with a partition table compatible with LTSP way of doing the things.

Caution: first of all, the whole process has been done taking from there and there until it has worked!!! So maybe there is a simpler, cleaner way of doing this. I apologize and, in case you devise a better way to get the task done, please share it!!!

Server installation

I started with a fresh Ubuntu 20.04 LTS server install. This choice was made because the server is an old HP xw8400 workstation with a RAID and after many trials I was unable to run the 22.04 LTS installer. However, at least for the purposes of this guide it doesn't matter.

For LTSP installation I followed the steps in LTSP site, but without all the epoptes stuff.

apt install --install-recommends ltsp ltsp-binaries dnsmasq nfs-kernel-server openssh-server squashfs-tools ethtool net-tools

Because we have 2 NIC's in our server, I had to run

ltsp-dnsmasq --proxy-dhcp=0

and also, after identifying each of the NIC's, set up the network properly.

Also I had to edit /etc/dnsmasq.d/ltsp-dnsmasq.conf so I ensure the DHCP server only listens to the LTSP network interface, the range of the IP for dhcp is properly set for our needs and the DNS server is set properly:

interface=ens6
dhcp-range=192.168.2.150,192.168.2.250,12h

It may be necessary to set another options depending on your network setup.

Then we do

ltsp ipxe
ltsp nfs
ltsp initrd

This will create, among other stuff, the folders /srv/ltsp y /srv/tftp in our server.

Now with /srv/ltsp available we can setup our operating system image.

Creation of operating system image in the server (Preparing the chroot)

First of all I have to mention that there are several OS available for Rock 4C+. However, after many trial-and-error tests, AFAIK that only the official Rock 4C+ Debian image makes it possible. The reasons are:

  • The image is divided into 2 partitions, one for first setup scripts of boards (/config) and the other for all the / filesystem. Other images are divided in 5 partitions and it was impossible (for me) to make it work.

  • The main reason, however, is that most of the tested images are not able to do a PXE boot unless you enter uboot menu on every boot and issue the appropiate uboot commands. Moreover, the uboot implementation in these images makes not possible to save the environment in a SD card. Fortunately the official Rock 4C+ Debian image provides a reasonably simple way to get this done. We will get back on this later when preparing the SD card for booting.

  • Now we are going to follow most of the steps given in ltsp.org for Raspios but adapted to our needs:

  1. Download the official Rock 4C+ Debian image, decompress the downloaded file and if you want, rename the obtained img file to a simpler name to ease the things later, let's say rock4cplus.img.

  2. Then we setup a loopback device with the downloaded image:

losetup -f # to get a free device, we will assume /dev/loop8
losetup -rP /dev/loop8 rock4cplus.img

If we make ls /dev we notice that two new partitions will appear, in my case they were /dev/sdd1 and /dev/sdd2. You must figure out which are the names of yours, anyway you must mount the root filesystem partition in the next step. In my case it was /dev/sdd2.

  1. We mount it and copy the files in the root filesystem to our /srv/ltsp folder: cp -a /mnt/. /srv/ltsp/rock4cplus

  2. After that, we unmount the filesystem and detachthe loopback device.

umount /mnt
losetup -d /dev/loop8

Now we have our chroot under /srv/ltsp/rock4cplus, but we still have things to do to be able to use it properly.

  1. Because the different architectures involved, you'll need to install qemu-user-static to be able to chroot: apt-get-install qemu-user-static

  2. We have to be able to resolve names, etc in our chroot, so we need to do this (To be better documented, help here would be very appreciated)

sudo mount -o bind /run /mnt/run
echo 'nameserver 8.8.4.4' | sudo tee -a /etc/resolv.conf

Now we are able to chroot /srv/ltsp/rock4cplus

  1. We must install LTSP packages into the client's chroot:
apt-get update
apt install --install-recommends ltsp

  1. VERY IMPORTANT the Debian image is not provided with the package nfs-common by default. We need to install it to be able to mount all the NFS partitions in the clients. apt install nfs-common

  2. After this we exit from the chroot (just send exit in the chroot prompt) and send some LTSP commands to make it work:

ltsp kernel
ltsp ipxe
ltsp nfs
ltsp initrd
  1. Now it's just matter of creating the compressed rock4cplus image:

ltsp image rock4cplus --mksquashfs-params='-comp lzo'

Modifying the OS image to boot from network

This has been made mostly by trial-and-error so better protocols for doing it could exists for sure. As I said before, only the Debian image is capable of modify kernel/uboot parameters to make possible PXE boot. To shorten we need to follow these steps:

Write the Debian Image in a SD card: You can do this with many softwares out there, however I got better results with Startup Disk Creator

Then comes the tricky part: to update the kernel arguments so uboot can do PXE boot. Because we are able to mantain the server filesystem with chroot, we can set it up for work just in LTSP mode. This means that the booted cards couldn't modify the operating system and also that the user folder will be those of the server.

I reccomend you do all these steps through the serial console. You can checkout how to do it here Because serial console speed is the not-very-standard 1500000, you must have a USB to RS232 adapter capable of managing such speeds. After some trials I found this inexpensive device based in the chip CH340G that works like a charm.

  1. Once the image is written, boot one of your boards with it.
  2. Log in and gain sudo/root access (User is radxa, password radxa)
cd /boot
  1. Inside the boot folder there is a text file, uEnv.txt with the following content:
# uEnv.txt has been retired.
# Please use:
#   'rsetup' command to confiure device overlays;
#   'sudo nano /etc/kernel/cmdline && sudo u-boot-update' to update kernel arguments.

We will modify cmdline adding mostly the same parameters as stated in Raspbios installation.

So just do sudo nano /etc/kernel/cmdline && sudo u-boot-update and modify cmdlin file that should be left as follows (notice there is a single line:

loglevel=4 ip=dhcp root=/dev/nfs nfsroot=192.168.2.1:/srv/ltsp/rock4cplus,vers=3,tcp,nolock init=/usr/share/ltsp/client/init/init ltsp.image=images/rock4cplus.img console=ttyS2,1500000n8 console=ttyFIQ0,1500000n8 coherent_pool=2M irqchip.gicv3_pseudo_nmi=0 quiet splash

IMPORTANT: Remember to match nfsroot server IP and path for those of your particular setup (by default, LTSP uses 192.168.67.1) and also the name of the image.

Modifyng /etc/kernel/cmdlineand doingu-boot-updatewill update kernel arguments and also modifiy another of our targets, the fileextlinux.confin/boot/extlinux` . This file is where the uboot menu lives and with this steps done it is enough to boot the boards with the OS inside the SD card. However, it is desirable to take some further steps, because in this state the board will need one particular file, the Device Tree descriptor file, to be able to boot properly. With this modification we will be able to reduce a lot the size of the SD card necesary to boot the boards and also simplify it. The idea is to reduce the SD card filesystem as much as possible, but the board need three files from the SD card to make the first booting steps:

  • initrd.img
  • vmlinuz
  • and the device tree file, in our case rk3399-rock-4c-plus.dtb

The first two files are located at SD card /boot but the third is located outside /boot, at /usr/lib/linux-image-5.10.110-1-rockchip/ So if we want to reduce the filesystem the best thing to do is to remove all the unnecesary files from the SD card. So what we do is copy the dtb file in /boot. This can be done also mounting the SD card in an appropiate mount point of the server, in that case you'll have to provide the appropiate paths.

mkdir /boot/fdt
cp  /usr/lib/linux-image-5.10.110-1-rockchip/rockchip/rk3399-rock-4c-plus.dtb /boot/fdt

But still we need to tell uboot where to find the dtb file, because every time we mess editing cmdline and then u-boot-update the extlinux.conf file is overwritten and a default path for kernel, initrd, etc is set. So we will just manually edit /boot/extlinux/extlinux.conf providing the proper path for the dtb file. This can be done modifyng the line fdtdir in the appropiate section of the file (in our case, the commands under l0, the default label)

Instead of creating an specific folder for the dtb file you can just just in /boot, anyway, just make sure that you provide the correct path for fdtdir in the previous step. Also you should remove the root={A NICE UUID} command after append.

Your extlinux.conf file should look more or less like this:

default l0
menu title U-Boot menu
prompt 0
timeout 10

label l0
        menu label LTSP boot
        linux /boot/vmlinuz-5.10.110-1-rockchip
        initrd /boot/initrd.img-5.10.110-1-rockchip
        fdtdir /boot/fdt/

        append ip=dhcp root=/dev/nfs nfsroot=192.168.2.1:/srv/ltsp/rock4cplus,vers=3,tcp,nolock init=/usr/share/ltsp/client/init/init ltsp.image=images/rock4cplus.img loglevel=4 console=ttyS2,1500000n8 console=ttyFIQ0,1500000n8 coherent_pool=2M irqchip.gicv3_pseudo_nmi=0 quiet splash

label l0r
        menu label Debian GNU/Linux 11 (bullseye) 5.10.110-1-rockchip (rescue target)
        linux /boot/vmlinuz-5.10.110-1-rockchip
        initrd /boot/initrd.img-5.10.110-1-rockchip
        fdtdir /usr/lib/linux-image-5.10.110-1-rockchip/

        append root=UUID=8087ba46-93d3-4f58-a640-231fd0aa806c ip=dhcp root=/dev/nfs nfsroot=192.168.2.1:/srv/ltsp/rock4cplus,vers=3,tcp,nolock init=/usr/share/ltsp/client/init/init ltsp.image=images/rock4cplus.img loglevel=4 console=ttyS2,1500000n8 console=ttyFIQ0,1500000n8 coherent_pool=2M irqchip.gicv3_pseudo_nmi=0 splash single

Then you can take your SD card and enjoy booting your boards into LTSP mode.

Shrinking clients SD image file

The SD card we have just prepared can be cloned for booting more clients. However, despite having the minimum possible set of files for boot it is still big: more than 5 GB, because we have deleted files but the filesystem has not been reduced.

Looking around I found this where the process is well explained.

As a summary, you'll have to follow these steps:

  1. With dd, make a dump of the SD card to a img file
sudo dd if=/dev/sdd of=SD.img bs=512
  1. Find a free loopback device and create a device from the image file
sudo losetup -f
losetup /dev/loop7 SD.img
  1. Because we need to access the partitions we must tell the kernell to do so
sudo partprobe /dev/loop7

  1. Now with gparted we resize our partition to the minimum possible space. To do so select the device in gparted, then select the partition and resize it (Sorry, no pictures of it)

  2. Detach the device with sudo losetup -d /dev/loop7 and then you'll have the SD.img file with the partition reduced.

Now comes the tricky part. We have reduced the partitions but the image file is not reduced yet (Remember that you left a lot of unallocated space when used gparted??) so it's time to shave the img file. This is tricky because of the type of partition table, fortunately there is a comment to the stackoverflow answer which solves the problem.

  1. Use fdisk -l SD.img to list the partition table of our image. It will render this (sorry my system is not in english):
Disco SD.img: 210,02 MiB, 220217856 bytes, 430113 sectores
Unidades: sectores de 1 * 512 = 512 bytes 
Tamaño de sector (lógico/físico): 512 bytes / 512 bytes
Tamaño de E/S (mínimo/óptimo): 512 bytes / 512 bytes
Tipo de etiqueta de disco: gpt
Identificador del disco: E9E74BE8-F67F-4473-962F-035298072EEC

Dispositivo Comienzo  Final Sectores Tamaño Tipo
SD.img1        32768  65535    32768    16M Datos básicos de Microsoft
SD.img2        65536 430079   364544   178M Sistema EFI
  1. We need to keep in mind two data from the previous output: The sector size (512 bytes) and the end of the partition (in our case the partition 2 ends at sector 430079.
  2. With those data in mind we will shave the image file with truncate. But because we have a GPT partition table, we must take that into account and add a certain amount of free space at the end (33 sectors)
truncate --size=$[(430079+1+33)*512] SD.img <-- Notice the 33 extra sectors for GPT table backup

  1. The problem now is the partition table is corrupted, because it looks like with GPT there is a copy of the partition table at the very end of the space and we just shaved it off, so we have to fix it. You can find a detailed description of the process here, but to summarize you'll have to use gdisk to fix it. To do so, you just:

  2. Verify the image file with v

  3. Enter into expert mode with x

  4. Relocate backup data structures to the end of the disk with e

  5. Write changes to the image file with w

I've left the dump of the process in this gist.

¡Done! Now you have your tiny boot image (in my case just about 210 Mb) that is easy and fast to be written to your Rock 4C+ Sd cards.

Add a custom footer
Add a custom sidebar
Clone this wiki locally