Provisioning device-specific/deployment-specific settings before boot

[beniroquai]

Feature Request: Customizable Parameters for Forklift-Built Images

As previously discussed, it would be highly beneficial to add functionality allowing us to customize images built with Forklift. The primary use case involves passing specific parameters, such as the IP address of robots (e.g., robot arms, pipetting robots, or microscopes) integrated into different architecture environments via Forklift, directly to the corresponding control software.

Requested Features:

• Ability to define and pass parameters (e.g., IP addresses) dynamically when building or deploying images.
• A mechanism (switch) to select the target control software based on the robot type (robot arm, pipetting robot, microscope, etc.).
• Configuration options to set default WLAN network settings.
• (Optional / To be clarified) Integration capability or settings for Tailscale networks for secure and simplified remote access.

@ethanjli Could you please evaluate how these features could be implemented and what additional information or clarification would be required to proceed?

[ethanjli]

I think the first iteration of our solution to this issue can look something like the following:

• After flashing the SD card, you can copy some files into /boot/init-root, e.g. /boot/init-root/etc/NetworkManager/system-connections/my-favorite-wifi-network.nmconnection. You can also use this mechanism to copy some sort of hardware configuration file for your control software, e.g. by dropping the config file into somewhere like /boot/init-root/var/lib/my-lab-automation-program/config.yml.
• A systemd service (named something like boot-init-root.service) deployed with Forklift will run on every boot boot in order to move anything in /boot/init-root into /.

Regarding other requested functionalities:

Ability to define and pass parameters (e.g., IP addresses)

Can you be more specific? i.e. what kinds of IP addresses do you want to customize, and what other kinds of settings do you want to change? How much of that needs to exist outside of a config file for your control software?

Integration capability or settings for Tailscale networks for secure and simplified remote access.

The above-described mechanism can already support generating new ZeroTier identities for joining ZeroTier networks and giving those identities to new devices by copying files to the appropriate locations (e.g. as described in https://github.com/zerotier/ZeroTierOne/blob/dev/doc/zerotier-idtool.1.md for generating machine identity files, and as described in https://github.com/zerotier/ZeroTierOne/blob/dev/doc/zerotier-one.8.md for where pre-generated machine identity files can be copied to). In order to achieve a similar level of integration/semi-automation for Tailscale, you'll need to figure out Tailscale's equivalent functionality (if that exists, which is not guaranteed given Tailscale's very different architecture for user/device identity).

IP address of robots (e.g., robot arms, pipetting robots, or microscopes)

What is the network architecture for your devices? e.g. will each robot (and thus each robot's sidecar RPi computer) only need to be preprogrammed with the IP address of a central hub to connect to? or do you have something more complex (e.g. every robot needs to know the IP address of every other robot, in which case you're trying to build a mesh network with service discovery like what is implemented in https://github.com/hashicorp/serf)?

[beniroquai]

Thanks a lot @ethanjli for your kind help! :)

1 Wifi Config

Regarding the first-boot-and-copy procedure: This sounds good! Just to undestand this right again, the /boot/init-root/ would be on the SD card and readable when plugin that into a normal computer (e.g. when I wanted to enable SSH I needed to create a file with name SSH in /boot, same goes for the wifi credentials).
Having a service that automatically does this on first boot sounds great!
I assume that this could be replaced with another SSID/PW combination e.g. via the cockpit's terminal for those who need to do it again/later?

2 Define other parameters

Can you be more specific? i.e. what kinds of IP addresses do you want to customize, and what other kinds of settings do you want to change? How much of that needs to exist outside of a config file for your control software?

I think this would be a feature that would be useful for multiple applications (e.g. place specific config files that customize the OS installation after downloading it from Github's build process). To be more specific:

• In case of e.g. ImSwitch this could be the Config file that specifies the currently active imswitch setup definition (defined e.g. here and copied by forklift currently: https://github.com/openUC2/ImSwitchConfig/tree/master/config)
• In case of e.g. Arkitekt this could be a token and host/port config to find the server
• In case of a robot (which would be a service running through Arkitekt), this could be its hostname, too

3 Autologin in Tailscale

A quick research showed me this:

So it should be possible to install tailscale and launch it with:

curl -fsSL https://tailscale.com/install.sh | sh
sudo tailscale up --authkey XXXXX

I'm not sure if this is persistent though (e.g. survives a reboot..right now it does).

4 Client-Server Situation

The situation was/is/will be (?) as follows:

So for those devices that cannot be wrapped into a dedicated forklifted image / OS already, we need the sidecar. This bridges an external hardware interface with the arkitekt world. In that case we thought about having a base OS image defined by forklift that has the key ingredients available for the interfacing to arkitekt, while the hardware interface (mostly a single python script) customizes the hardware adapter. This would need to be provided by the user after flashing the script through the /boot # 1.

CC: @jhnnsrs

A first arkitekt service/app would be this https://github.com/arkitektio-apps/dornado

Does this make sense? ;-)

[beniroquai]

During the hackathon (and now this issue becomes large..), we also come up with the idea to have a base image for the central arkitekt server in general. This would potentially not run on a raspberry pi - but rather something more "sophisticated". Are there ways to reuse the forklift mechanisms for this? Essentially it's "just" a series of docker containers plus network configurations, correct @jhnnsrs

[jhnnsrs]

Hi hi, chiming in!

Generally i believe that would work great.

Just as a thought once the wifi is setup and an internet connection is established some level of configuration could also happen through arkitekt. In this scenario the app would first contact the arkitekt server to authenticate itself and then could retrieve a headscale login key to subsequently join the users headscalenet (if the user has one).

This would only mean there is a need to hardcore the coordination server url like 'go.arkitekt.live' and then depending on the capabilities of the raspberry pi either a preauthkey for arkitekt (redeem token) or not even that if it's able to display some characters (a la device code oauth2). Or maybe analogous and kind of fun we could program the same logic by letting the raspberry pi led flash and use this as morsecode (instead of a device code) :D

Like this only wifi would be a needed configuration :D

Hope that makes sense?

[jhnnsrs]

And yes it's just docker containers ;)

[ethanjli]

Based on this discussion, I think I have enough information to proceed with prototyping the new functionality which I had proposed above - I'll create a pull request in https://github.com/openUC2/imswitch-os for you to test.

1 Wifi Config

the /boot/init-root/ would be on the SD card and readable when plugin that into a normal computer

Yes, that is also my understanding. I believe the /boot partition is FAT-formatted, so it should be readable/writable on any computer.

when I wanted to enable SSH I needed to create a file with name SSH in /boot, same goes for the wifi credentials)

To be precise, Raspberry Pi OS does special handling for /boot/ssh and /boot/ssh.txt (as described here). For Wi-Fi credentials, you can't just make a /boot/wifi file because nothing has been created for special handling of such a file. Instead, you would create a file inside /boot/init-root/etc/NetworkManager/system-connections.d/ so that it would be copied by our boot-init-root.service into /etc/NetworkManager/system-connections.d, and then the NetworkManager service in Raspberry Pi OS would handle that file like any other file during bootup.

I assume that this could be replaced with another SSID/PW combination e.g. via the cockpit's terminal for those who need to do it again/later?

Once you have access to the Cockpit terminal to edit files in e.g. /etc/NetworkManager/system-connections.d, then you can just edit those files directly in /etc/NetworkManager/system-connections.d. In PlanktoScope OS, we have a "System File Manager" which is just a deployment of https://github.com/filebrowser/filebrowser as a Docker container for browsing and editing any file in / from your web browser - let me know if you want me to add that to https://github.com/openUC2/pallet.

Probably the way I will implement boot-init-root.service is that it will run on every boot, not just the first boot - so if you add new files to /boot/init-root after a subsequent shutdown, then on the next boot those files will be moved by boot-init-root.service into their appropriate destinations. This way, after the first boot you could overwrite an existing file (e.g. /etc/NetworkManager/system-connections.d/my-lan.nmconnection) simply by copying an updated version of that file into /boot/init-root (e.g. /boot/init-root/etc/NetworkManager/system-connections.d/my-lan.nmconnection) and then booting up again.

2 Define other parameters

e.g. place specific config files

Ok, it sounds like your use-cases can all capture parameters in files; then I think boot-init-root.service will be sufficient, at least as the initial prototype.

3 Autologin in Tailscale

sudo tailscale up --authkey XXXXX

Interesting, okay. I think I might be able to automate this by creating a systemd service (e.g. tailscale-add-auth-key.service) which only runs if a file named /run/tailscale-auth-key exists, and which executes tailscale up --authkey {contents of /run/tailscale-auth-key}. /run's contents are only stored in RAM and aren't actually saved to the SD card - so if we place a file in /boot/init-root/run/tailscale-auth-key before boot which contains the auth key generated from the Tailscale control panel, then boot-init-root.service will move it to /run/tailscale-auth-key as a temporary file, and then tailscale-add-auth-key.service will use that temporary file to run the tailscale up --authkey XXXXX command.

I will include a prototype of this functionality for you to test, as part of the PR I'll make for imswitch-os.

4 Client-Server Situation

A first arkitekt service/app would be this https://github.com/arkitektio-apps/dornado

Ok - from inspecting that repo's app.py, it looks like your sidecar will be running a Python script which needs a way for the user/operator to specify the IP address of the device connected to the sidecar. It should be feasible for you to set such settings from a config file for the Python script, in which case we can see how it would feel to have you specify such settings via a file preloaded into /boot/init-root for boot-init-root.service to handle.

If I understand correctly, @jhnnsrs is proposing that it would be sufficient to preload (e.g. via boot-init-root.service) or preconfigure (e.g. via a constant which is baked into the imswitch-os image for all RPi's running that image) only the following kinds of settings before boot:

1. Settings needed to connect to the internet (e.g. a Wi-Fi network connection profile)
2. Settings needed to connect to the Arkitekt server (e.g. the server's URL)
3. A device identifier for authentication to the Arkitekt server (though if the sidecar can generate and display the identifier for the user/operator to register that identifier with the Arkitekt server, then this doesn't even need to be preloaded)

This information would be enough to bootstrap an authenticated connection from the sidecar to the Arkitekt server so that the sidecar can download any necessary machine-specific settings which the user/operator has set on the Arkitekt server (e.g. a Tailscale auth key, and a setting for the IP address of a robot connected to the sidecar). This approach is similar to the function of secret stores like Hashicorp Vault, which are like databases that can give device-specific passwords/API keys/secrets to authenticated devices according to device identity.

I think it could make sense to aim for this approach as proposed by @jhnnsrs, because it could simplify the user/operator's process of configuring and setting up new sidecar computers. I'm not sure how much of the necessary functionality is already implemented in Arkitekt, but we can incrementally/gradually move towards that from an initial prototype which completely relies on boot-init-root.service.

[ethanjli]

we also come up with the idea to have a base image for the central arkitekt server in general. This would potentially not run on a raspberry pi - but rather something more "sophisticated". Are there ways to reuse the forklift mechanisms for this?

From what I understand, the Arkitekt server is currently deployed as a bunch of Docker Compose apps onto a single computer/VM. Forklift is (and will continue to be) optimized for managing a collection of Docker Compose apps on a single computer/VM. If that matches how you want to deploy the Arkitekt server for lab automation (i.e. not deploying it as a cluster of Arkitekt servers via Kubernetes or Docker Swarm or Hashicorp Nomad), then it should be pretty straightforward to use Forklift for managing the Docker containers. Depending on what kind of compute resource (on-premises bare-metal computer vs. cloud VM) you want to deploy to, it might or might not make sense to use Forklift for managing network configurations.

I'd be happy to list some follow-up questions and/or to share any relevant information on choosing/managing the underlying OS for the Arkitekt server deployment, if you start a separate Github Issues/GitHub Discussions item for that topic. If it's not clear where that issue/discussion post belongs, you can just create the new issue in this repo and later transfer it elsewhere.

For completeness, I should also note that if you can deploy the Arkitekt server on something like Fedora CoreOS (instead of Raspberry Pi OS) and you can run your Docker containers using Podman (instead of Docker), then it may make more sense to just build a custom OS image as a bootable container image (in GitHub Actions, with a Dockerfile/Containerfile like in this example for making a custom image with Tailscale) based on Fedora CoreOS; then your bootable container image can include good defaults for all networking configs and it can manage your Docker container deployments (via Podman Quadlets). Upgrading/downgrading your containers would just be a matter of upgrading/downgrading the bootable container image and rebooting. This approach would probably make Forklift redundant for such a deployment scenario - and in fact Forklift was mostly created as a very rough/partial approximation of such an approach, as a workaround for Raspberry Pi OS's lack of support for being deployed as a bootable container image. On the other hand, if the Docker Compose apps for Arkitekt are likely to change in the future, it might be better to use Fedora CoreOS as the base OS, and Forklift for deploying the Arkitect Docker Compose apps.

[ethanjli]

If it's not clear where that issue/discussion post belongs, you can just create the new issue in this repo and later transfer it elsewhere.

@beniroquai following up on your invitation for me to join https://github.com/stainSTORM, it looks like maybe https://github.com/stainSTORM/smart_arkitekt would be the correct place to start a follow-up issue/discussion about choosing an appropriate way to build/deploy/administer the OS for the Arkitekt server? Or would it make more sense for this issue to be created in one of the 40 non-archived repositories (!! that's an overwhelming number of repositories) in https://github.com/orgs/arkitektio/repositories?q=visibility%3Apublic+archived%3Afalse ? https://arkitekt.live/docs/introduction/installation#docker-compose 's link to the GitHub repo for a Docker Compose app is broken, so I'm kind of lost in looking for the Docker Compose app(s).