Example architecture

[thebsdbox]

My initial idea is to have two main containers in our LinuxKit image:

Docker with a bind to /var/run:

  - name: docker-osie
    image: docker:19.03.8-dind
    capabilities:
     - all
    net: host
    mounts:
     - type: cgroup
       options: ["rw","nosuid","noexec","nodev","relatime"]
    binds:
     - /etc/resolv.conf:/etc/resolv.conf
     - /var/lib/docker:/var/lib/docker
     - /lib/modules:/lib/modules
     - /var/run:/var/run
     - /etc/docker/daemon.json:/etc/docker/daemon.json
    command: ["/usr/local/bin/docker-init", "/usr/local/bin/dockerd"]
    runtime:
      mkdir: ["/var/lib/docker"]

We can interact with it:

ctr -n services.linuxkit t exec --exec-id test docker-osie docker pull nginx

Custom Go binary:

  - name: workflow
    image: workflow:beta
    binds:
     - /var/run:/var/run
     - /proc/cmdline:/proc/cmdline

The custom go binary will use the docker SDK to speak to the docker.sock in /var/run...

The final piece of the puzzle is getting the registry certificate into this, there are two options I can see:

• Users build their own custom kernel/initramfs with that cert
• We have an onboot container that gets certificates and puts them somewhere on the filesystem that we can pass into the docker container.

[thebsdbox]

The alternative would be to just use the containerd Sdk, however I think the tinkerbell-worker requires the Docker socket.. but we could re-write that to use containerd too. We’d need to parse the workflow and pull the images before running them but 🤷🏼‍♂️

[gianarb]

I think the value we can get from this initial work is better sized and easier to customize OS. That's why I think we should go with the most transparent possible solution, the one you described is perfect because ideally, it should not require any change.

1. Download the new image
2. Place it in the right place inside sandbox/deploy/state
3. Business as usual

That's I think what we should get to.

I personally like the idea to have a well-defined interface that other workers can implement to run actions in their own way, almost like a kubelet has, that's how I think we should solve the hard dependency with Docker (yes I know Kubernetes has kubelet interface and CRI interface but I think we should have only one for simplicity, but we have to see how big that interface will be in order to have the right answer).

I am not sure I can clearly translate what you wrote with what I know. We need a container that runs onboot as you wrote that will download certificates and do all the other configurations. We need a container that starts tink-worker and it has to communicate with the docker socket. This image should only work in workflow mode, so all the other code that osie has should not be ported over I think.

[thebsdbox]

I think we've achieved this!

🦖