Add support for Sysbox on AL2 and AL2023

[ctalledo]

Sysbox should work on AL2 distro. This will make it easier to use Sysbox on AWS services that use AL2.

Currently this is gated by the lack of support for ID-mapped mounts in AL2, which uses kernel 5.10. We expect that once AL2 moves to kernel >= 5.12, Sysbox should work on it without much problem.

This Epic tracks this effort.

[ctalledo]

Update: per this AWS page, AL2 now supports kernels up to 5.15, which should include ID-mapped-mounts, therefore allowing Sysbox to work on it (without shiftfs).

[abhishady]

Hi @ctalledo

I tried to configures sysbox using daemonset manifest file in EKS cluster which is using EKS optimized AL2 AMI but it still throwing compatibility issue. I even updated kernel version to 5.15.
Any idea?

[ctalledo]

Hi @abhishady,

Thanks for giving this a try (we've not had a chance to do it yet).

I tried to configures sysbox using daemonset manifest file in EKS cluster which is using EKS optimized AL2 AMI but it still throwing compatibility issue.

What error are you seeing?

[twoodhouse]

I'm actually seeing the same thing.
ERROR: Sysbox is not supported on this host's distro ("amzn"-2).

I'm on Linux 5.4.209-116.363.amzn2.x86_64

Possibly I just need to move to >= 5.12

[abhishady]

Hi @ctalledo

Please see the screenshot I've added.

As you can see in the image I'm using 5.15 kernel version.

[ctalledo]

Hi @abhishady,

Apologies for the late reply.

I checked the code, and the error you are getting when installing Sysbox on your K8s cluster via the sysbox-deploy-k8s daemonset is due to a hardcoded check in the daemonset that allows it to only work on "officially supported" distros (Ubuntu mostly).

sysbox-deploy-k8s is a bit tricky since it has to install both CRI-O and Sysbox on the running K8s node, and then swap the kubelet's config from containerd -> CRI-O. Therefore adjusting sysbox-deploy-k8s to work on EKS with AL2-based nodes will require some trial and error on our side.

And before trying that, we would start by installing Sysbox on an AL2 host (non-EKS) to verify that it works; I am confident it will. Unfortunately AL2 uses RPM (not APT) for package management, and we don't yet have an RPM package for Sysbox,
so we would need to clone the sysbox repo, build the binaries, and run them to verify all is good.

Realistically we won't get to this until October, due to some other stuff we are working on; however, enabling Linux on AL2 is an important task for us. Thanks for your patience.

[ctalledo]

By the way, folks do run Sysbox on EKS, but they configure the EKS nodes to use Ubuntu (instead of AL2). Just thought I would mention it to see if it's a work-around while we enable Sysbox on AL2.

[DekusDenial]

Hi @ctalledo, I am wondering if you can provide any update on this?
Also, does building sysbox from source work in AL2 in theory since it's basically based off centos?
Thanks!

[DekusDenial]

@rodnymolina wondering if you can chime in and provide related update on this? thanks

[MichaelGerhart]

My org is also interested in sysbox, but we have a requirement to use AL2.

hoping there is an updated ETA coming soon?

[ctalledo]

Hi @DekusDenial, @MichaelGerhart, apologies for the belated reply.

I think we are close to formally supporting Sysbox on AL2, given that AL2 can easily be upgraded from kernel 5.10 (default) to kernel 5.15 with the amazon linux extras package. This in turn means Sysbox can now use the kernel's new ID-mapped mounts mechanism (available since Linux kernel 5.12+) so that host files can show up with proper ownership inside the unprivileged Sysbox container. Previously this required the kernel's shiftfs module which was only supported on Debian/Ubuntu, but not on AL2.

In fact, if you install the latest Sysbox binaries (v0.5.2) on AL2 with kernel 5.15+, it will likely work (though I've not personally tested yet).

However in order for us to provide formal support for AL2, we would need to ensure the full Sysbox test suite runs on AL2, which is a bit more work (but likely not a lot, so I would expect we will have this within ~2 months or so).

[DekusDenial]

In fact, if you install the latest Sysbox binaries (v0.5.2) on AL2 with kernel 5.15+, it will likely work (though I've not personally tested yet).

sysbox and patched cri-o bundle on AL2 with 5.15 kernel should work as almost everything is statically linked. It will be really nice to see sysbox and its dependencies can be installed via yum-based package manager (mentioned by other issue IIRC) as many orgs have strict requirements on streamline deterministic AMI build process which may not like something like patching kubelet or installing another container runtime on the fly via a privileged daemonset. Also, my org has a business license with Docker and we will be very interested in a potential revamped sysbox EE offering if “native” AWS/GCP support is officially landed.

(but likely not a lot, so I would expect we will have this within ~2 months or so).

I am currently working on testing our use case with sysbox on AL2 on my end, and I can post any finding or update as well.

[ctalledo]

Thanks @DekusDenial for the response.

Also, my org has a business license with Docker and we will be very interested in a potential revamped sysbox EE offering if “native” AWS/GCP support is officially landed

Thanks, noted.

I am currently working on testing our use case with sysbox on AL2 on my end, and I can post any finding or update as well.

Very cool; I'll be spending cycles on this too and will post updates as I progress through it.

[MichaelGerhart]

Hi @DekusDenial, @MichaelGerhart, apologies for the belated reply.

I think we are close to formally supporting Sysbox on AL2, given that AL2 can easily be upgraded from kernel 5.10 (default) to kernel 5.15 with the amazon linux extras package. This in turn means Sysbox can now use the kernel's new ID-mapped mounts mechanism (available since Linux kernel 5.12+) so that host files can show up with proper ownership inside the unprivileged Sysbox container. Previously this required the kernel's shiftfs module which was only supported on Debian/Ubuntu, but not on AL2.

In fact, if you install the latest Sysbox binaries (v0.5.2) on AL2 with kernel 5.15+, it will likely work (though I've not personally tested yet).

However in order for us to provide formal support for AL2, we would need to ensure the full Sysbox test suite runs on AL2, which is a bit more work (but likely not a lot, so I would expect we will have this within ~2 months or so).

Hey @ctalledo sorry to be a pain, just wanted to see if there were any updates on this?

For context, we are both a docker business customer and a Coder Enterprise customer (I think you and Kyle have direct communication?).

Our org only approved AL2 for production use on AWS with EKS, thusly we are stuck waiting for this support. Our use case is specifically for providing docker in Coder workspaces.

I am trying to determine if this is going to be available soon, or if I need to go through our internal processes of vetting an additional OS (Ubuntu) as a workaround until AL2 is supported.

Thanks for any info you can provide!