Setting up an SDM Build Server

[r-colvin]

Note: this is a fresh discussion that has it's root in this Raspberry Pi Forum post: https://forums.raspberrypi.com/viewtopic.php?p=2363582#p2363582 - as such more context might be found there, though I hope this is self contained.

Also I have done my best to read the documentation and as many issues and discussions here as possible - though it can be easy to miss information, if you suspect the answer has been documented elsewhere please point me in the right direction! Although well documented, the documentation (https://github.com/gitbls/sdm/blob/master/Docs/Index.md) suffers from being in alphabetical order - it may be preferential to order the documentation to start with concepts, hello world, then each of the relevant sections.

Having said that! I'll start with the problem I am trying to solve. The project I have is to manage a fleet of Raspberry Pi's - some are Pi 3B+'s, some are Pi 4's and some are Pi 5's. Some have NVME boot drives, others SD. Some have NVME for data storage but boot from SD's. The Pi's are used for various functions with different tools required - some require a desktop and VNC, some are headless! Different Pi's will have different users, users might need to be added to Pi's over time.

Until now, I have just imaged an SD using Raspberry Pi Imager either the Lite or Full image. Then followed a lazy-dog I wrote to add the packages and configuration required. I could make this into an Ansible playbook I guess... with Raspberry Pi Imager I of course set the WiFi, Hostname, add Users and SSH Keys - with cloud_init i can keep the user-data, network-config files and just copy+paste them (I think... haven't tried) as required. Thus I have a spare SD card for each Pi that can be swapped in if I am not available to fix it. This ensures a least a basic functionality.

When a Pi breaks, as happens, the desire is to just replace the SD and continue; if replacing the SD doesn't fix the issue, then it's possible to grab a spare Pi from the shelf, flash an SD and get back to work - the broken Pi/SD added to a box for troubleshooting and fixing. In just the last week I have had SD cards fail (expected!) as well as a Pi4 SOC fail - so this is not without precedent! Lost hours cost much more than replacing an SD or Pi.

As the needs have grown, this has become painful and there is enough commonality to warrant a core base-image with specific customizations. The idea is to have the core image specified, then have project/Pi specific customization scripts. I spent some time learning rpi-image-gen and built the core image. With a bash script to build the image I could make images for specific targets and a wrapper to rpi-imager --cli would allow me to add specific user-data and network-config customizations when flashing to the actual SD card. rpi-image-gen is great as I can run it in a Docker container or Lightsail build server (even hooked to GitHub actions) which produces an .img; the .img is easily customizable using Raspberry Pi Imager. This is somewhat easily automated but has the benefit that Raspberry Pi Imager has been whitelisted to write to USB devices on my Mac (which no other application is allowed via policy)

Unfortunately, although I can create the .img in rpi-image-gen, I am having issues with Raspberry Pi Imager performing the customizations requested when flashing the SD card. These are documented in this post: https://forums.raspberrypi.com/viewtopic.php?p=2363582#p2363582 and also this issue: raspberrypi/rpi-image-gen#182. In the Raspberry Pi Forum post gitbls suggested I look into SDM and that we continue the conversation here.

Having looked into SDM, it appears to do almost everything I want to do - I have raised two improvement tickets to allow installing debs from a local file (#389) and package version pinning (#390) - those are not show stoppers though.

My big question is given the above how should I configure a build server?

• I prefer to keep build systems isolated from my main work system as it prevents conflicts and issues
  • As far as I can see, any Debian host should be fine - but is it better to run on an ARM64 host or does it work just fine on an AMD64? I have a Lenovo x64 mini-pc running Ubuntu 22.04 LTS that I could use for SDM, it also doesn't have the restriction on writing to SD cards/USB devices, however it's a remote system that is painful to access
  • would installing SDM on such a PC have an impact on any of the other services that PC has to host?

I see a similar discussion here: #338 but sorry, I didn't quite understand the answer given - would it be possible to have SDM generate a .img that can be customized with Raspberry Pi Imager using cloud_init (or firstrun.sh) files? It seems that if you don't use SDM to flash the SD, you lose some of the customization power - is that true? what would be lost if you used Raspberry Pi Imager to flash the SD instead of SDM? (remember I only have one Ubuntu PC that I can foreseeably run SDM on natively (as intended) and I'm the only one with access - whereas others have access to Raspberry Pi Imager)

As mentioned, I have been using Docker to run rpi-image-gen which has worked well. I see the following issues, discussions and projects that have used Docker and it seems possible - however I know that gitbls is also waiting for a stable/reliable way of using Docker; So I am wondering if these are usable or any heads-up before I waste time and effort re-inventing the wheel!

• Docker steps? #262
• Using of sdm without of system-wide installation #216
• Sdm-based project #299 -> Sdm-based project #299

Lastly, I do have a Pi 3B+ or Pi 400 that I could use as a "build server" - though given I have been able to destroy a Pi4 whilst using it as a build system for rpi-image-gen (what prompted me to move to Docker) I am once-bitten-twice-shy on doing this...

Ok! that's a long post - hopefully everything is captured :) to summarise

• given my needs and restrictions
  1. is SDM the right solution for me, or am I better off waiting to see if Raspberry Pi fix the customization issue I am having?
  2. if SDM is the right solution - what is the best way to setup SDM? Taking into account the restrictions on flashing SD cards, and needing others to be able to flash SD cards
     • Docker on my M3 Mac?
     • Lightsail Server?
     • Ubuntu 22.04 on an x64 mini PC
     • Pi 3B+ / Pi 400

Thanks in advance! looking forward to hearing from those with more experience and knowledge than I

[gitbls]

Note: this is a fresh discussion that has it's root in this Raspberry Pi Forum post: https://forums.raspberrypi.com/viewtopic.php?p=2363582#p2363582 - as such more context might be found there, though I hope this is self contained.

Also I have done my best to read the documentation and as many issues and discussions here as possible - though it can be easy to miss information, if you suspect the answer has been documented elsewhere please point me in the right direction! Although well documented, the documentation (https://github.com/gitbls/sdm/blob/master/Docs/Index.md) suffers from being in alphabetical order - it may be preferential to order the documentation to start with concepts, hello world, then each of the relevant sections.

The documentation is what it is. Reference style is the most cost-effective to document from my perspective. Also, I'm working on a Getting Started with sdm video. Coming soon!

Having said that! I'll start with the problem I am trying to solve. The project I have is to manage a fleet of Raspberry Pi's - some are Pi 3B+'s, some are Pi 4's and some are Pi 5's. Some have NVME boot drives, others SD. Some have NVME for data storage but boot from SD's. The Pi's are used for various functions with different tools required - some require a desktop and VNC, some are headless! Different Pi's will have different users, users might need to be added to Pi's over time.

Until now, I have just imaged an SD using Raspberry Pi Imager either the Lite or Full image. Then followed a lazy-dog I wrote to add the packages and configuration required. I could make this into an Ansible playbook I guess... with Raspberry Pi Imager I of course set the WiFi, Hostname, add Users and SSH Keys - with cloud_init i can keep the user-data, network-config files and just copy+paste them (I think... haven't tried) as required. Thus I have a spare SD card for each Pi that can be swapped in if I am not available to fix it. This ensures a least a basic functionality.

When a Pi breaks, as happens, the desire is to just replace the SD and continue; if replacing the SD doesn't fix the issue, then it's possible to grab a spare Pi from the shelf, flash an SD and get back to work - the broken Pi/SD added to a box for troubleshooting and fixing. In just the last week I have had SD cards fail (expected!) as well as a Pi4 SOC fail - so this is not without precedent! Lost hours cost much more than replacing an SD or Pi.

Diligent, structured, scripted use of sdm can make this very easy for you.

As the needs have grown, this has become painful and there is enough commonality to warrant a core base-image with specific customizations. The idea is to have the core image specified, then have project/Pi specific customization scripts. I spent some time learning rpi-image-gen and built the core image. With a bash script to build the image I could make images for specific targets and a wrapper to rpi-imager --cli would allow me to add specific user-data and network-config customizations when flashing to the actual SD card. rpi-image-gen is great as I can run it in a Docker container or Lightsail build server (even hooked to GitHub actions) which produces an .img; the .img is easily customizable using Raspberry Pi Imager. This is somewhat easily automated but has the benefit that Raspberry Pi Imager has been whitelisted to write to USB devices on my Mac (which no other application is allowed via policy)

Unfortunately, although I can create the .img in rpi-image-gen, I am having issues with Raspberry Pi Imager performing the customizations requested when flashing the SD card. These are documented in this post: https://forums.raspberrypi.com/viewtopic.php?p=2363582#p2363582 and also this issue: raspberrypi/rpi-image-gen#182. In the Raspberry Pi Forum post gitbls suggested I look into SDM and that we continue the conversation here.

Having looked into SDM, it appears to do almost everything I want to do - I have raised two improvement tickets to allow installing debs from a local file (#389) and package version pinning (#390) - those are not show stoppers though.

Nothing you've described above is difficult with sdm, except for Docker, but I comment on that below.

My big question is given the above how should I configure a build server?

• I prefer to keep build systems isolated from my main work system as it prevents conflicts and issues

  • As far as I can see, any Debian host should be fine - but is it better to run on an ARM64 host or does it work just fine on an AMD64? I have a Lenovo x64 mini-pc running Ubuntu 22.04 LTS that I could use for SDM, it also doesn't have the restriction on writing to SD cards/USB devices, however it's a remote system that is painful to access
  • would installing SDM on such a PC have an impact on any of the other services that PC has to host?

sdm will run on an X64 Debian system BUT be aware that when sdm enters the IMG/disk container during Phase 1 and Phase 2 Raspberry Pi programs in the container will be run via QEMU, so it will run more slowly than native. I'm sure there are estimates/guesstimates on the internet as to the performance cost of using QEMU. But, running with QEMU is better than "Can't run at all", so there you go.

I see a similar discussion here: #338 but sorry, I didn't quite understand the answer given - would it be possible to have SDM generate a .img that can be customized with Raspberry Pi Imager using cloud_init (or firstrun.sh) files? It seems that if you don't use SDM to flash the SD, you lose some of the customization power - is that true? what would be lost if you used Raspberry Pi Imager to flash the SD instead of SDM? (remember I only have one Ubuntu PC that I can foreseeably run SDM on natively (as intended) and I'm the only one with access - whereas others have access to Raspberry Pi Imager)

I'm going to ignore cloudinit in my response. When burning an IMG, you can either burn it to a disk (more or less the normal case) or to another IMG. The latter is useful if a) you want to distribute customized IMGs (over the internet, for example), or b) if your sdm host is a Mac or Windows system where, for instance WSL (Windows) cannot write directly to a USB disk (similar situation exists on MacOS).

sdm IMGs (either customized or burned with burnfile) can be burned to a disk with Imager if properly configured. "properly" depends on what you want to do, so you'll have to experiment and/or ask specific questions.

Here are the things that sdm can do/does during burning to a disk that would not be available if you use something other than sdm to burn to a disk:

• Burning to a GPT partition table format
• Burning to a BTRFS rootfs

There may be others, but pretty much everything else should work if the appropriate switches are used and you use sdm burnfile output as the input to Imager.

As mentioned, I have been using Docker to run rpi-image-gen which has worked well. I see the following issues, discussions and projects that have used Docker and it seems possible - however I know that gitbls is also waiting for a stable/reliable way of using Docker; So I am wondering if these are usable or any heads-up before I waste time and effort re-inventing the wheel!

• Docker steps? #262
• Using of sdm without of system-wide installation #216
• Sdm-based project #299 -> Sdm-based project #299

I don't use Docker. Perhaps someone who does can comment on these. Or you can try them yourself.

Lastly, I do have a Pi 3B+ or Pi 400 that I could use as a "build server" - though given I have been able to destroy a Pi4 whilst using it as a build system for rpi-image-gen (what prompted me to move to Docker) I am once-bitten-twice-shy on doing this...

My suggestion is that you try using sdm on your X64 system and see how much you dislike the emulated performance. I'm a bad one to comment on this bc I demand high performance and I run sdm far too many times each day. If you're unhappy with the performance you'll probably want to use a Pi4-class or Pi5-class system for running sdm. Unless you're very patient.

Ok! that's a long post - hopefully everything is captured :) to summarise

• given my needs and restrictions

  1. is SDM the right solution for me, or am I better off waiting to see if Raspberry Pi fix the customization issue I am having?
  2. if SDM is the right solution - what is the best way to setup SDM? Taking into account the restrictions on flashing SD cards, and needing others to be able to flash SD cards

Only you can determine whether sdm is a good solution for you.

I expect that RPL will improve the boot-time customization that can be done using cloud-init by making it easier to use in Imager. If you're happy with sorting out yaml and having long boot times while extensive cloud-init customization is done on the first boot, then go that way.

On the other hand, if you want to produce tiered systems (common base config with per-system customizations done at burn time) so that the number of base config IMGs is minimized and the first boot is fast, then use sdm.

 * Docker on my M3 Mac?
 * Lightsail Server?
 * Ubuntu 22.04 on an x64 mini PC
 * Pi 3B+ / Pi 400

Nice technology list.

Thanks in advance! looking forward to hearing from those with more experience and knowledge than I

In general, sdm can do everything you've written about above. But it's very generalized and you've asked many broad questions. It will be much easier to respond with specifics if you ask specific questions, hopefully after using sdm to become familiar with it.