-
Notifications
You must be signed in to change notification settings - Fork 59
Using Rock 4C plus boards with LTSP
Having a LTSP setup with Raspberry Pi as clients is very cool and very easy of setup as well. However, Raspberry Pi market went crazy and we chose Rock 4C+ board to replace them. However, this was not as easy as RPi to setup mainly for the difficulties of getting help from the Rockpi community (nice but not as developed as RPi's)
The main source of problems was the ability of a particular Rock 4C+ OS image able to modify uboot parameters easily and with a partition table compatible with LTSP way of doing the things.
Caution: first of all, the whole process has been done taking from there and there until it has worked!!! So maybe there is a simpler, cleaner way of doing this. I apologize and, in case you devise a better way to get the task done, please share it!!!
I started with a fresh Ubuntu 20.04 LTS server install. This choice was made because the server is an old HP xw8400 workstation with a RAID and after many trials I was unable to run the 22.04 LTS installer. However, at least for the purposes of this guide it doesn't matter.
For LTSP installation I followed the steps in LTSP site, but without all the epoptes stuff.
apt install --install-recommends ltsp ltsp-binaries dnsmasq nfs-kernel-server openssh-server squashfs-tools ethtool net-tools
Because we have 2 NIC's in our server, I had to run
ltsp-dnsmasq --proxy-dhcp=0
and also, after identifying each of the NIC's, set up the network properly.
Also I had to edit /etc/dnsmasq.d/ltsp-dnsmasq.conf
so I ensure the DHCP server only listens to the LTSP network interface, the range of the IP for dhcp is properly set for our needs and the DNS server is set properly:
interface=ens6
dhcp-range=192.168.2.150,192.168.2.250,12h
It may be necessary to set another options depending on your network setup.
Then we do
ltsp ipxe
ltsp nfs
ltsp initrd
This will create, among other stuff, the folders /srv/ltsp
y /srv/tftp
in our server.
Now with /srv/ltsp
available we can setup our operating system image.
First of all I have to mention that there are several OS available for Rock 4C+. However, after many trial-and-error tests, AFAIK that only the official Rock 4C+ Debian image makes it possible. The reasons are:
-
The image is divided into 2 partitions, one for first setup scripts of boards (/config) and the other for all the / filesystem. Other images are divided in 5 partitions and it was impossible (for me) to make it work.
-
The main reason, however, is that most of the tested images are not able to do a PXE boot unless you enter uboot menu on every boot and issue the appropiate uboot commands. Moreover, the uboot implementation in these images makes not possible to save the environment in a SD card. Fortunately the official Rock 4C+ Debian image provides a reasonably simple way to get this done. We will get back on this later when preparing the SD card for booting.
-
Now we are going to follow most of the steps given in ltsp.org for Raspios but adapted to our needs:
-
Download the official Rock 4C+ Debian image, decompress the downloaded file and if you want, rename the obtained
img
file to a simpler name to ease the things later, let's sayrock4cplus.img
. -
Then we setup a loopback device with the downloaded image:
losetup -f # to get a free device, we will assume /dev/loop8
losetup -rP /dev/loop8 rock4cplus.img
If we make ls /dev we notice that two new partitions will appear, in my case they were /dev/sdd1
and /dev/sdd2.
You must figure out which are the names of yours, anyway you must mount the root filesystem partition in the next step. In my case it was /dev/sdd2.
-
We mount it and copy the files in the root filesystem to our /srv/ltsp folder:
cp -a /mnt/. /srv/ltsp/rock4cplus
-
After that, we unmount the filesystem and detachthe loopback device.
umount /mnt
losetup -d /dev/loop8
Now we have our chroot under /srv/ltsp/rock4cplus, but we still have things to do to be able to use it properly.
-
Because the different architectures involved, you'll need to install qemu-user-static to be able to chroot:
apt-get-install qemu-user-static
-
We have to be able to resolve names, etc in our chroot, so we need to do this (To be better documented, help here would be very appreciated)
sudo mount -o bind /run /mnt/run
echo 'nameserver 8.8.4.4' | sudo tee -a /etc/resolv.conf
Now we are able to chroot /srv/ltsp/rock4cplus
- We must install LTSP packages into the client's chroot:
apt-get update
apt install --install-recommends ltsp
-
VERY IMPORTANT the Debian image is not provided with the package nfs-common by default. We need to install it to be able to mount all the NFS partitions in the clients.
apt install nfs-common
-
After this we exit from the chroot (just send
exit
in the chroot prompt) and send some LTSP commands to make it work:
ltsp kernel
ltsp ipxe
ltsp nfs
ltsp initrd
- Now it's just matter of creating the compressed rock4cplus image:
ltsp image rock4cplus --mksquashfs-params='-comp lzo'
This has been made mostly by trial-and-error so better protocols for doing it could exists for sure. As I said before, only the Debian image is capable of modify kernel/uboot parameters to make possible PXE boot. To shorten we need to follow these steps:
Write the Debian Image in a SD card: You can do this with many softwares out there, however I got better results with Startup Disk Creator
Then comes the tricky part: to update the kernel arguments so uboot can do PXE boot. Because we are able to mantain the server filesystem with chroot, we can set it up for work just in LTSP mode. This means that the booted cards couldn't modify the operating system and also that the user folder will be those of the server.
I reccomend you do all these steps through the serial console. You can checkout how to do it here Because serial console speed is the not-very-standard 1500000, you must have a USB to RS232 adapter capable of managing such speeds. After some trials I found this inexpensive device based in the chip CH340G that works like a charm.
- Once the image is written, boot one of your boards with it.
- Log in and gain sudo/root access (User is
radxa
, passwordradxa
)
cd /boot
- Inside the boot folder there is a text file, uEnv.txt with the following content:
# uEnv.txt has been retired.
# Please use:
# 'rsetup' command to confiure device overlays;
# 'sudo nano /etc/kernel/cmdline && sudo u-boot-update' to update kernel arguments.
We will modify cmdline adding mostly the same parameters as stated in Raspbios installation.
So just do sudo nano /etc/kernel/cmdline && sudo u-boot-update
and modify cmdlin file that should be left as follows (notice there is a single line:
loglevel=4 ip=dhcp root=/dev/nfs nfsroot=192.168.2.1:/srv/ltsp/rock4cplus,vers=3,tcp,nolock init=/usr/share/ltsp/client/init/init ltsp.image=images/rock4cplus.img console=ttyS2,1500000n8 console=ttyFIQ0,1500000n8 coherent_pool=2M irqchip.gicv3_pseudo_nmi=0 quiet splash
IMPORTANT: Remember to match nfsroot server IP and path for those of your particular setup (by default, LTSP uses 192.168.67.1) and also the name of the image.
Modifyng
/etc/kernel/cmdlineand doing
u-boot-updatewill update kernel arguments and also modifiy another of our targets, the file
extlinux.confin
/boot/extlinux` . This file is where the uboot menu lives and with this steps done it is enough to boot the boards with the OS inside the SD card. However, it is desirable to take some further steps, because in this state the board will need one particular file, the Device Tree descriptor file, to be able to boot properly. With this modification we will be able to reduce a lot the size of the SD card necesary to boot the boards and also simplify it. The idea is to reduce the SD card filesystem as much as possible, but the board need three files from the SD card to make the first booting steps:
- initrd.img
- vmlinuz
- and the device tree file, in our case
rk3399-rock-4c-plus.dtb
The first two files are located at SD card /boot but the third is located outside /boot, at /usr/lib/linux-image-5.10.110-1-rockchip/ So if we want to reduce the filesystem the best thing to do is to remove all the unnecesary files from the SD card. So what we do is copy the dtb file in /boot. This can be done also mounting the SD card in an appropiate mount point of the server, in that case you'll have to provide the appropiate paths.
mkdir /boot/fdt
cp /usr/lib/linux-image-5.10.110-1-rockchip/rockchip/rk3399-rock-4c-plus.dtb /boot/fdt
But still we need to tell uboot where to find the dtb file, because every time we mess editing cmdline
and then u-boot-update
the extlinux.conf
file is overwritten and a default path for kernel, initrd, etc is set. So we will just manually edit /boot/extlinux/extlinux.conf
providing the proper path for the dtb file. This can be done modifyng the line fdtdir
in the appropiate section of the file (in our case, the commands under l0, the default label)
Instead of creating an specific folder for the dtb file you can just just in /boot, anyway, just make sure that you provide the correct path for fdtdir
in the previous step. Also you should remove the root={A NICE UUID}
command after append
.
Your extlinux.conf file should look more or less like this:
default l0
menu title U-Boot menu
prompt 0
timeout 10
label l0
menu label LTSP boot
linux /boot/vmlinuz-5.10.110-1-rockchip
initrd /boot/initrd.img-5.10.110-1-rockchip
fdtdir /boot/fdt/
append ip=dhcp root=/dev/nfs nfsroot=192.168.2.1:/srv/ltsp/rock4cplus,vers=3,tcp,nolock init=/usr/share/ltsp/client/init/init ltsp.image=images/rock4cplus.img loglevel=4 console=ttyS2,1500000n8 console=ttyFIQ0,1500000n8 coherent_pool=2M irqchip.gicv3_pseudo_nmi=0 quiet splash
label l0r
menu label Debian GNU/Linux 11 (bullseye) 5.10.110-1-rockchip (rescue target)
linux /boot/vmlinuz-5.10.110-1-rockchip
initrd /boot/initrd.img-5.10.110-1-rockchip
fdtdir /usr/lib/linux-image-5.10.110-1-rockchip/
append root=UUID=8087ba46-93d3-4f58-a640-231fd0aa806c ip=dhcp root=/dev/nfs nfsroot=192.168.2.1:/srv/ltsp/rock4cplus,vers=3,tcp,nolock init=/usr/share/ltsp/client/init/init ltsp.image=images/rock4cplus.img loglevel=4 console=ttyS2,1500000n8 console=ttyFIQ0,1500000n8 coherent_pool=2M irqchip.gicv3_pseudo_nmi=0 splash single
Then you can take your SD card and enjoy booting your boards into LTSP mode.
The SD card we have just prepared can be cloned for booting more clients. However, despite having the minimum possible set of files for boot it is still big: more than 5 GB, because we have deleted files but the filesystem has not been reduced.
Looking around I found this where the process is well explained.
As a summary, you'll have to follow these steps:
- With dd, make a dump of the SD card to a img file
sudo dd if=/dev/sdd of=SD.img bs=512
- Find a free loopback device and create a device from the image file
sudo losetup -f
losetup /dev/loop7 SD.img
- Because we need to access the partitions we must tell the kernell to do so
sudo partprobe /dev/loop7
-
Now with
gparted
we resize our partition to the minimum possible space. To do so select the device in gparted, then select the partition and resize it (Sorry, no pictures of it) -
Detach the device with
sudo losetup -d /dev/loop7
and then you'll have the SD.img file with the partition reduced.
Now comes the tricky part. We have reduced the partitions but the image file is not reduced yet (Remember that you left a lot of unallocated space when used gparted??) so it's time to shave the img file. This is tricky because of the type of partition table, fortunately there is a comment to the stackoverflow answer which solves the problem.
- Use
fdisk -l SD.img
to list the partition table of our image. It will render this (sorry my system is not in english):
Disco SD.img: 210,02 MiB, 220217856 bytes, 430113 sectores
Unidades: sectores de 1 * 512 = 512 bytes
Tamaño de sector (lógico/físico): 512 bytes / 512 bytes
Tamaño de E/S (mínimo/óptimo): 512 bytes / 512 bytes
Tipo de etiqueta de disco: gpt
Identificador del disco: E9E74BE8-F67F-4473-962F-035298072EEC
Dispositivo Comienzo Final Sectores Tamaño Tipo
SD.img1 32768 65535 32768 16M Datos básicos de Microsoft
SD.img2 65536 430079 364544 178M Sistema EFI
- We need to keep in mind two data from the previous output: The sector size (512 bytes) and the end of the partition (in our case the partition 2 ends at sector 430079.
- With those data in mind we will shave the image file with
truncate
. But because we have a GPT partition table, we must take that into account and add a certain amount of free space at the end (33 sectors)
truncate --size=$[(430079+1+33)*512] SD.img <-- Notice the 33 extra sectors for GPT table backup
-
The problem now is the partition table is corrupted, because it looks like with GPT there is a copy of the partition table at the very end of the space and we just shaved it off, so we have to fix it. You can find a detailed description of the process here, but to summarize you'll have to use gdisk to fix it. To do so, you just:
-
Verify the image file with
v
-
Enter into expert mode with
x
-
Relocate backup data structures to the end of the disk with
e
-
Write changes to the image file with
w
I've left the dump of the process in this gist.
¡Done! Now you have your tiny boot image (in my case just about 210 Mb) that is easy and fast to be written to your Rock 4C+ Sd cards.