Enable full offline source build

[SchoolGuy]

Proposed Changes

Currently it is not possible to build k3s from source due to the build scripts mixing the stages of download and build.

Types of Changes

I would consider this a bugfix as this was possible in the past.

Verification

The build in the OBS (which doesn't allow network access) succeeds: https://build.opensuse.org/package/show/home:SchoolGuy:branches:devel:kubic/k3s?expand=0

Testing

No we don't have testing but a CI/CD pipeline could be added via the OBS CI/CD Beta features.

Linked Issues

Fixes #5618

User-Facing Change

None

Further Comments

This PR is atm a WIP. I opened it to provide a place for discussion of my current implementation. I haven't yet figured out how I can slim down the source tarball. Because it is so huge I wasn't able to upload it to OBS in a single file. I bet that I could skip some parts of the git history to safe space.

Open points include but are not limited to:

• Split containerd, Traefik Helmchart and other components into subpackages
• Make it build in the OBS not only locally
• Update the specfile in a way that one could build it in a Dockerfile here in GitHub CI

[SchoolGuy]

Update: I updated the specfile in the OBS and I am now smoothing out a few architecture dependant naming shenanigans for the binary (the arch suffix is not desired in case of OBS builds). Other then that the source tarballs now can be uploaded since I limited the Git History (as described in BUILDING.md) and thus cut it down from ~600MB to 128MB (including vendored sources).

To make this build here with dapper I need to invest some more time and I need to actually commit the specfile back. I have used openSUSE specific systemd macros thus I need to add a few %if so Fedora can build this too. Debian package I will not take care of.

Splitting the packages requires more work on the build scripts and I will probably do this even though there is no benefit at the moment except having a clean separation between the components. For this I need to split the buildscripts more though and this requires also time.

I hope someone from the k3s team is able to have a look at this PR soon and give an opinion on my work so far.

[SchoolGuy]

Update: Packaging works better already after my changes in the OBS. I have however problems with the cni-plugins, iptables and flannel combination I am using for build after I start the server (essentially everything except the local network is blocked). Although I am not a fan of vendoring, I am using it as the scripts recommend. I will dig deeper and refactor more to make the structure of the folder better documented and visible.

[SchoolGuy]

Okay I saw that my specfile supplied containerd and runc from the OS. I have removed those and now containerd appears to be missing. It appears that I need to fix that first...

[SchoolGuy]

Okay fixed. Now k3s is really working with embedded containerd and runc! However networking problems are still persisting.

[SchoolGuy]

Okay the networking is related to the Wireguard VPN on the host. I have not found the reason why this is the case but I am one step further. Now Traefik is not starting (which is related to the changes in this PR most likely).

[SchoolGuy]

Okay it is not quite the PR. It is SELinux that is in the way... Let's see what awaits me after that issue.

[SchoolGuy]

Okay I can't figure out SELinux. However since I didn't adjust any paths for the installation target directories afaik, I will mark this PR as ready now since without enforcing SELinux it works like a charm.

[SchoolGuy]

@brandond I am aware that not all changes may be desired, however I believe they clean up the build scripts quite a bit and separate better between downloading, building and packaging. I am sure I bring this PR in a much better shape with the help of your review. The current state enables me to do everything I want due to the full separation of download, build and package. A neat (and desired) side-effect is also that neither k3s-buildroot nor Dapper are required anymore for building. Of course they can be used, however in environments like the OBS they are just extra weight that is not needed. Feel free to discuss with me synchronous in Slack if you desire more extensive explanations and reasons. If there is documentation missing I will happily provide it in your desired manner!

[brandond]

We don't really want to encourage Linux distros to build and package K3s in their own build systems. We support running K3s from the binaries that are released here on Github. If folks start building and offering their own versions of K3s, those are essentially other Kubernetes distros downstream from K3s.

We already had problems with one distro in particular stripping out the bundled containerd and userspace, resulting in a version of K3s that did not behave at all like our shipping releases. I know that's not what you're attempting to do here, but we want to take a firm line on our support boundary.

[SchoolGuy]

@brandond So does this mean you do want to merge this PR or not? I understand your point (although I don't agree). I am not trying to maintain the changes required for building k3s downstream to another place. However the changes in this PR are unrelated to RPM in generell. They just enable you to build from source and seperate the stages between git clone and the final binary in a matter of their different concerns.

[septatrix]

We don't really want to encourage Linux distros to build and package K3s in their own build systems. We support running K3s from the binaries that are released here on Github. If folks start building and offering their own versions of K3s, those are essentially other Kubernetes distros downstream from K3s.

I would actually presume that providing official repos for DEBs and RPMs will lower the incentive for distros to package K3S themselves. The reasons why people request packages for inclusion in the official repos is often that other installation means do not work offline, interfere with the package manager, offer no rollback, no signatures/checksums etc.

Providing first-party DEB/RPM packages already will fit into this a lot better and solve some of these problems. Later down the line you could additionally provide your own repositories such that updates etc could also be managed by a distros package manager.

[SchoolGuy]

@septatrix I second this. However with the slight adjustment that is is generally a good practice (imho), to have downstream packages. However for me official RPMs/DEBs would be a compromise I could live with.

[SchoolGuy]

CI is failing due to issue #4042. This is a known flaky test. PR should be in a mergable state again.

[SchoolGuy]

@cwayne18 why was this closed?

[c4t3l]

@cwayne18 Can you please explain why this was closed? There are several folks looking forward to having distro builds available.

[irreleph4nt]

Hi @SchoolGuy. Are you still maintaining the build scripts for RPMs of k3s? I was wondering whether any other routes have been explored to get RPMs and DEBs published, despite the opinionated stance of the k3s maintainers.

[SchoolGuy]

Hi @SchoolGuy. Are you still maintaining the build scripts for RPMs of k3s? I was wondering whether any other routes have been explored to get RPMs and DEBs published, despite the opinionated stance of the k3s maintainers.

It can be done through rebasing this PR. The effort is without cooperation of upstream not worth it.

Maybe it is worth it asking the maintainers of differnt projects (that take the same approach as k3s) if they are willing to cooperate in this matter. Essentially every Kubernetes distro at its core is a golang program that can be built in the openSUSE Build Service through the means of vendoring.