miner update HIP

[fvasquez]

Rendered view: https://github.com/helium/HIP/blob/e4a3b70c8f6234c49b9264156ba0ea250d5ac705/0000-miner-update.md

[Vagabond]

@mrallen1 and I have been playing around with Open Container Initiative (OCI) images. I think we can mount and boot them on the hotspot (using archivemount and chroot) without installing any container runtimes. This would, however, allow other people to run the container in docker/oci/etc on non hotspots as well.

I also think we could boot a OCI image using https://github.com/containers/bubblewrap

(base) ^andrew@system76:: ls -lah `which bwrap`
-rwxr-xr-x 1 root root 55K Nov 30 11:01 /bin/bwrap
(base) ^andrew@system76:: ldd `which bwrap`
        linux-vdso.so.1 (0x00007ffc36bf9000)
        libcap.so.2 => /usr/lib/libcap.so.2 (0x00007f833823d000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007f833807a000)
        /lib/ld-linux-x86-64.so.2 (0x00007f8338279000)

(base) ^andrew@system76:: ls -lah `which docker`
-rwxr-xr-x 1 root root 81M Feb 16 21:10 /bin/docker
(base) ^andrew@system76:: ldd `which docker`
        linux-vdso.so.1 (0x00007ffd58be2000)
        libpthread.so.0 => /usr/lib/libpthread.so.0 (0x00007f5ad85dd000)
        libdl.so.2 => /usr/lib/libdl.so.2 (0x00007f5ad85d8000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007f5ad8415000)
        /lib/ld-linux-x86-64.so.2 (0x00007f5adcbf1000)

[Vagabond]

More concretely I propose we'd make the following miner containers:

• amd64
• arm64 full
• arm64 slim (does not include Erlang's runtime)
• arm32 full

We'd run the arm64 slim container on the hotspot, it would essentially just contain /opt/miner and we'd mount through the needed system paths into the bubblewrap environment. The other containers would contain a full system image and an Erlang distribution (these could be done via composing layers). They would be served through a 'container registry' such that 'docker pull' or suchlike would be compatible. Most cloud platforms support a compatible cloud registry API now (eg Amazon ECR) and open source implementations exist. These registries can already contain OCI images.

https://docs.docker.com/registry/spec/api/
https://docs.microsoft.com/en-us/azure/container-registry/container-registry-oci-artifacts
https://github.com/opencontainers/artifacts

[Vagabond]

The v2 registry can be queried with bash/jq via something like https://github.com/gforghetti/docker_registry_api_bash_functions

[jadeallenx]

I made a story in Clubhouse for this work.

[fvasquez]

I made a story in Clubhouse for this work.

@Vagabond @mrallen1 Your proposed OCI and bubblewrap solution is interesting. I do have my doubts as to how mature the bubblewrap tooling is compared to Docker's. Namely does bubblewrap really have something akin to docker pull and can it authenticate image registries and verify signed images. Regardless of these concerns I find a chroot-based solution like this one compelling enough to warrant further investigation. I will incorporate an OCI/bubblewrap option into this HIP.

[Vagabond]

I made a story in Clubhouse for this work.

@Vagabond @mrallen1 Your proposed OCI and bubblewrap solution is interesting. I do have my doubts as to how mature the bubblewrap tooling is compared to Docker's. Namely does bubblewrap really have something akin to docker pull and can it authenticate image registries and verify signed images. Regardless of these concerns I find a chroot-based solution like this one compelling enough to warrant further investigation. I will incorporate an OCI/bubblewrap option into this HIP.

Bubblewrap relies on cgroups, same as Docker does (it's just a much thinner shim over the top). Bubblewrap is used in a ton of places, including flatpak applications and I found out yesterday it's how void linux sandboxes their package building.

https://wiki.archlinux.org/index.php/Bubblewrap
https://github.com/void-linux/void-packages#requirements

It definitely does not have a docker pull implementation, but the docker protocol is HTTP based, and as I linked above, shell libraries exist to query it (so we'd do docker pull via bash/curl).

I've been thinking about the OCI "layers" we might want to use:

• Base image (eg alpine-linux with Erlang)
• Portable BEAM files and release scripts/configs
• NIFs and other native code

This would allow us to reuse 'layer 2' for all of our docker images and just crosscompile layer3 for arm/arm64 (and swap out the base image for an architecture appropriate image). As a bonus, since our NIFs don't change a lot, we could simply update the middle layer the majority of the time.

[Vagabond]

Additionally we can do stuff like @macpie did in the router docker stuff, where we use the environment to configure a sys.config.src template at runtime (for things like paths, key mode, etc)