Skip to content

Commit

Permalink
feat: add support for edk2-rk3588
Browse files Browse the repository at this point in the history 
feat: use emulated system for everything else except the kernel
docs: update README for both U-boot & UEFI
  • Loading branch information
@ryan4yin
ryan4yin committed Mar 6, 2024
1 parent 0eea4e4 commit c785186
Show file tree
Hide file tree
Showing 17 changed files with 588 additions and 296 deletions.
142 changes: 36 additions & 106 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,16 +2,25 @@

> :warning: Work in progress, use at your own risk...
A minimal flake to run NixOS on RK3588/RK3588s based SBCs.
A minimal flake to run NixOS on RK3588/RK3588s based SBCs, support both UEFI & U-Boot.

![](_img/nixos-on-orangepi5.webp)
![](_img/nixos-orangepi5plus.webp)
![](_img/nixos-rock5a.webp)

Default user: `rk`, default password: `rk3588`

## Boards

UEFI support:

| Singal Board Computer | Boot from SD card | Boot from SSD |
| --------------------- | ------------------ | ------------------ |
| Orange Pi 5 | :heavy_check_mark: | :heavy_check_mark: |
| Orange Pi 5 Plus | :heavy_check_mark: | :heavy_check_mark: |
| Rock 5A | :heavy_check_mark: | :heavy_check_mark: |

U-Boot support:

| Singal Board Computer | Boot from SD card | Boot from SSD |
| --------------------- | ------------------ | ------------------ |
| Orange Pi 5 | :heavy_check_mark: | :heavy_check_mark: |
Expand All @@ -20,110 +29,25 @@ Default user: `rk`, default password: `rk3588`

## TODO

- [ ] support [edk2-rk3588](https://github.com/edk2-porting/edk2-rk3588), which is an UEFI firmware for Rockchip RK3588 platforms
- [ ] build u-boot with nix
- [x] support boot from emmc/ssd
- [ ] verify all the hardware features available by RK3588/RK3588s
- [x] ethernet (rj45)
- wifi/bluetooth
- audio
- [ ] wifi/bluetooth
- [ ] audio
- [x] gpio
- [x] uart/ttl
- gpu(mali-g610-firmware + [panfork/mesa](https://gitlab.com/panfork/mesa))
- npu
- [ ] gpu(mali-g610-firmware + [panfork/mesa](https://gitlab.com/panfork/mesa))
- [ ] npu
- ...

## Flash into SD card

### 1. Flash U-Boot to SPI flash

You should get the uboot from the vendor and flash it to the SPI flash before doing anything NixOS

1. [Armbian on Orange Pi 5 / Orange Pi 5 Plus](https://www.armbian.com/orange-pi-5/) as an example:
1. download the image and flash it to a sd card first
2. boot the board with the sd card, and then run `sudo armbian-install` to flash the uboot to the SPI flash(maybe named as `MTD devices`)

For Rock 5A, we've bundled the uboot into the sdImage, so you don't need to flash it into the SPI flash manually.

### 2. Flash NixOS to SD card

Build an sdImage by `nix build`, and then flash it to a sd card using `dd`(please replace `/dev/sdX` with the correct device name of your sd card)):

> Instead of build from source, you can also download the prebuilt image from [Releases](https://github.com/ryan4yin/nixos-rk3588/releases).
```shell
# for orange pi 5 plus
nix build .#sdImage-opi5plus
zstdcat result/sd-image/orangepi5plus-sd-image-*.img.zst | sudo dd status=progress bs=4M of=/dev/sdX

# for orange pi 5
nix build .#sdImage-opi5
zstdcat result/sd-image/orangepi5-sd-image-*.img.zst | sudo dd status=progress bs=4M of=/dev/sdX
```


For Rock 5A, it requires a little more work to flash the image to the sd card:

```shell
nix build .#sdImage-rock5a
zstd -d result/sd-image/rock5a-sd-image-*.img.zst -o rock5a.img

# increase img's file size
dd if=/dev/zero bs=1M count=16 >> rock5a.img
sudo losetup --find --partscan rock5a.img
## Flash & Boot NixOS

nix shell nixpkgs#parted
## rock 5a's u-boot require to use gpt partition table, and the root partition must be the first partition!
## so we need to remove all the partitions on the sd card first
## and then recreate the root partition with the same start sector as the original partition 2
START=$(sudo fdisk -l /dev/loop0 | grep /dev/loop0p2 | awk '{print $2}')
sudo parted /dev/loop0 rm 1
sudo parted /dev/loop0 rm 2
sudo parted /dev/loop0 mkpart primary ext4 ${START}s 100%
For UEFI, see [UEFI.md](./UEFI.md).

# check rootfs's status, it's broken.
sudo fsck /dev/loop0p1

# umount the image file
sudo losetup -d /dev/loop0


# flash the image to the sd card
sudo dd status=progress bs=4M if=rock5a.img of=/dev/sdX
```

1. insert the sd card to the board, and power on
2. resize the root partition to the full size of the sd card.
3. then having fun with NixOS

Once the system is booted, you can use `nixos-rebuild` to update the system.

## Flash into SSD/eMMC

To flash the image into the board's eMMC / SSD, you need to flash the image into the board and start into NixOS first.

Then, use the following command to flash the image into the board's SSD / eMMC:

```bash
# upload the sdImage to the NixOS system on the board
scp result/sd-image/orangepi5-sd-image-*.img.zst rk@<ip-of-your-board>:~/

# login to the board via ssh or serial port
ssh rk@<ip-of-your-board>

# check all the block devices
# you should see nvme0n1(SSD)
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
mtdblock0 31:0 0 16M 0 disk
zram0 254:0 0 0B 0 disk
nvme0n1 259:0 0 238.5G 0 disk
......

# flash the image into the board's SSD
zstdcat orangepi5-sd-image-*.img.zst | sudo dd bs=4M status=progress of=/dev/nvme0n1
```

After the flash is complete, remove the SD card and reboot, you should see NixOS booting from SSD / eMMC.
For U-Boot, see [U-Boot.md](./U-Boot.md).

## Debug via serial port(UART)

Expand All @@ -136,33 +60,39 @@ Here is an example configuration that you can use as a starting point: [Demo - D

## How this flake works

A complete Linux system typically consists of five components: U-Boot, the kernel, device trees, firmwares, and the root file system (rootfs).
A complete Linux system typically consists of five components:

Among these, U-Boot, the kernel, device trees, and firmwares are hardware-related and require customization for different SBCs.
1. Bootloader (typically U-Boot or EDKII)
1. Linux kernel
1. Device trees
1. Firmwares
1. Root file system (rootfs)

Among these, the bootloader, the kernel, device trees, and firmwares are hardware-related and require customization for different SBCs.
On the other hand, the majority of content in the rootfs is hardware-independent and can be shared across different SBCs.

Hence, the fundamental approach here is to use the hardware-specific components(U-Boot, kernel, and device trees, firmwares) provided by the vendor(orangepi/rockpi/...), and combine them with the NixOS rootfs to build a comprehensive system.
Hence, the fundamental approach here is to **use the hardware-specific components(bootloader, kernel, and device trees, firmwares) provided by the vendor(orangepi/rockpi/...), and combine them with the NixOS rootfs to build a comprehensive system**.

Regarding RK3588/RK3588s, a significant amount of work has been done by Armbian on their kernel, and device tree.
Therefore, by integrating these components from Armbian with the NixOS rootfs, we can create a complete NixOS system.

The primary steps involved are:

1. Build U-Boot using this Flake.
- Since no customization is required for U-Boot, it's also possible to directly use the precompiled U-Boot from Armbian or the hardware vendor.
2. Build the NixOS rootfs using this flake, leveraging the kernel and device tree provided by Armbian.
- To make all the hardware features available, we need to add its firmwares to the rootfs. Since there is no customization required for the firmwares too, we can directly use the precompiled firmwares from Armbian & Vendor.

Related Armbian projects:
1. Bootloader: Since no customization is required for U-Boot or [edk2-rk3588], it's also possible to directly use the precompiled image from [armbian], [edk2-rk3588], or the hardware vendor.
2. Build the NixOS rootfs using this flake, leveraging the kernel and device tree provided by [armbian].
- To make all the hardware features available, we need to add its firmwares to the rootfs. Since there is no customization required for the firmwares too, we can directly use the precompiled firmwares from Armbian & Vendor too.

- <https://github.com/armbian/build>
- <https://github.com/armbian/linux-rockchip>

## References

- [K900/nix](https://gitlab.com/K900/nix)
- [aciceri/rock5b-nixos](https://github.com/aciceri/rock5b-nixos)
- [nabam/nixos-rockchip](https://github.com/nabam/nixos-rockchip)
- [fb87/nixos-orangepi-5x](https://github.com/fb87/nixos-orangepi-5x)
- [edk2-rk3588]

And I also got a lot of help in the [NixOS on ARM Matrix group](https://matrix.to/#/#nixos-on-arm:nixos.org)!


[edk2-rk3588]: https://github.com/edk2-porting/edk2-rk3588
[armbian]: https://github.com/armbian/build
106 changes: 106 additions & 0 deletions U-Boot.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
# U-Boot

Here we describe how to use U-Boot to boot NixOS on RK3588/RK3588s based SBCs.

## 1. Flash U-Boot to SPI flash

You should get the uboot from the vendor and flash it to the SPI flash before doing anything NixOS

1. [Armbian on Orange Pi 5 / Orange Pi 5 Plus](https://www.armbian.com/orange-pi-5/) as an example:
1. download the image and flash it to a sd card first
2. boot the board with the sd card, and then run `sudo armbian-install` to flash the uboot to the SPI flash(maybe named as `MTD devices`)

For Rock 5A, we've bundled the uboot into the sdImage, so you don't need to flash it into the SPI flash manually.

## 2. Flash NixOS

There're two ways to flash NixOS to the board:

1. Flash NixOS to SD card
2. Flash NixOS into SSD/eMMC

## 2.1. Flash NixOS to SD card

This is the common way to flash NixOS to the board.

Build an sdImage by `nix build`, and then flash it to a SD card using `dd`(please replace `/dev/sdX` with the correct device name of your sd card):

> **Instead of build from source, you can also download the prebuilt image from [Releases](https://github.com/ryan4yin/nixos-rk3588/releases)**.
```shell
# for orange pi 5 plus
nix build .#sdImage-opi5plus
zstdcat result/sd-image/orangepi5plus-sd-image-*.img.zst | sudo dd status=progress bs=8M of=/dev/sdX

# for orange pi 5
nix build .#sdImage-opi5
zstdcat result/sd-image/orangepi5-sd-image-*.img.zst | sudo dd status=progress bs=8M of=/dev/sdX
```

For Rock 5A, it requires a little more work to flash the image to the sd card:

> The prebuilt image has been repaired before uploading, so you can use it directly.
```shell
nix build .#sdImage-rock5a
zstd -d result/sd-image/rock5a-sd-image-*.img.zst -o rock5a.img

# increase img's file size
dd if=/dev/zero bs=1M count=16 >> rock5a.img
sudo losetup --find --partscan rock5a.img

nix shell nixpkgs#parted
## rock 5a's u-boot require to use gpt partition table, and the root partition must be the first partition!
## so we need to remove all the partitions on the sd card first
## and then recreate the root partition with the same start sector as the original partition 2
START=$(sudo fdisk -l /dev/loop0 | grep /dev/loop0p2 | awk '{print $2}')
sudo parted /dev/loop0 rm 1
sudo parted /dev/loop0 rm 2
sudo parted /dev/loop0 mkpart primary ext4 ${START}s 100%

# check rootfs's status, it's broken.
sudo fsck /dev/loop0p1

# umount the image file
sudo losetup -d /dev/loop0


# flash the image to the sd card
cat rock5a.img | sudo dd status=progress bs=8M of=/dev/sdX
```

1. Insert the sd card to the board, and power on
2. Resize the root partition to the full size of the sd card.
3. Then having fun with NixOS <3.

Once the system is booted, you can use `nixos-rebuild` to update the system.

## Flash NixOS into SSD/eMMC

To flash the image into the board's eMMC / SSD, you need to flash the image into the SD card and start into NixOS first, as SSD / eMMC is not easy to remove and connect to your host.

Then, use the following command to flash the image into the board's SSD / eMMC:

```bash
# upload the sdImage to the NixOS system on the board
scp result/sd-image/orangepi5-sd-image-*.img.zst rk@<ip-of-your-board>:~/

# login to the board via ssh or serial port
ssh rk@<ip-of-your-board>

# check all the block devices
# you should see nvme0n1(SSD)
$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINTS
mtdblock0 31:0 0 16M 0 disk
zram0 254:0 0 0B 0 disk
nvme0n1 259:0 0 238.5G 0 disk
......

# flash the image into the board's SSD
zstdcat orangepi5-sd-image-*.img.zst | sudo dd bs=4M status=progress of=/dev/nvme0n1
```

After the flash is complete, remove the SD card and reboot, and NixOS should boot from the SSD / eMMC now.


69 changes: 69 additions & 0 deletions UEFI.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,69 @@
# UEFI

Here we describe how to use UEFI([edk2-rk3588]) to boot NixOS on RK3588/RK3588s based SBCs.

## 1. Flash [edk2-rk3588] to SPI flash

To use UEFI to boot NixOS, you need to flash the UEFI firmware to the SPI flash of the board.

The steps to do this:

1. Download the prebuilt UEFI firmware from [edk2-rk3588/releases](https://github.com/edk2-porting/edk2-rk3588/releases).
1. To flash the UEFI firmware to the SPI flash, you need to:
1. Boot the board with a Linux distro that supports your SBC, such as [armbian](https://www.armbian.com/download/) or your SBC's official image.
1. Then use `dd` to flash the UEFI firmware you downloaded to the SPI flash:
```bash
sudo dd if=./xxx-UEFI-xxx.img of=/dev/mtdblock0
```
1. Reboot the board, and you should see the UEFI boot menu.
1. In the UEFI boot menu
1. Enter [Device Manager] => [Rockchip Platform Configuration] => [ACPI / Device Tree]
1. Change [Config Table Mode] to `Both`.
1. Change [Device Tree Configuration] => [Support DTB override & overlays] to `Enabled`.


## 2. Flash NixOS to SD card

This is the common way to flash NixOS to the board.

First, build the raw efi image:

> It will takes a long time(about 12mins) to finish, as we have to use the emulated system on x64 host to build the raw efi image.
```bash
# for orange pi 5
nix build .#rawEfiImage-opi5 --show-trace -L --verbose

# for orange pi 5 plus
nix build .#rawEfiImage-opi5plus --show-trace -L --verbose

# for rock 5a
nix build .#rawEfiImage-rock5a --show-trace -L --verbose
```

If you encounter issues like [`cannot allocate memory` despite free reporting "available"](https://stackoverflow.com/questions/46464785/cannot-allocate-memory-despite-free-reporting-available) when building the raw efi image, check your `dmesg`, and try to fix it via:

```bash
echo 1 > /proc/sys/vm/compact_memory
```

Then, flash the raw efi image to the board's SSD / SD card:

> please replace `/dev/sdX` with the correct device name of your sd card
```bash
cat result | sudo dd status=progress bs=8M of=/dev/sdX
```

After the flash is complete, remove the SD card and reboot, you should see the UEFI boot menu.

## 3. Install NixOS into SSD / eMMC via `nixos-install`

> It's recommend to install your system into SSD / eMMC or a high-speed SD card, otherwise `nixos-install` may take a long time to finish.
It's also possible to install NixOS via `nixos-install`, this will give you a more flexible way to manage your system.
For example, you can use `nixos-install` to install NixOS with LUKS encryption, custom partition layout & filesystem, and with all your favorite packages pre-installed, etc.

To do this, you need to flash the image into the SD card and start into NixOS first, and then just follow the [official installation guide](https://nixos.org/manual/nixos/stable/#sec-installation-manual).

[edk2-rk3588]: https://github.com/edk2-porting/edk2-rk3588
Binary file removed _img/nixos-on-orangepi5.webp
View file
Binary file not shown.

0 comments on commit c785186

Please sign in to comment.