Does BurmillaOS need newer kernel version and how it can be provided?

[olljanat]

Rancher OS is currently using 4.14.x kernels which will get security patches until January 2024. That why there is no urgent need switch to newer releases.

However I would like to get comments about what are new features on newer kernels which would be useful for Rancher OS?

Kernel build process is described on https://rancher.com/docs/os/v1.x/en/installation/custom-builds/custom-kernels/ and I already got 5.4 build process working on tree https://github.com/rancher-os-community/os-kernel/tree/5107e0728769ea4afceff06cb9bf21b20c14d55d (automatic build using GitHub actions) and it pushed release to https://github.com/rancher-os-community/os-kernel/releases/tag/v5.4.61-rancheroscommunity but boot fails with that one so we need figure out which changes are needed to kernel configuration.

[jokay]

@olljanat thank you for all your effort on this 💪🏼

[PrplHaz4]

Since I think RancherOS makes a great homelab OS, almost all kernel updates improve gpu and network device support. Also looking through these makes me think some thought needs to be given to kernel vs kernel-extras...

niusmallnan
The kernel-base in v1.5.1 has that firmware, but not in v1.5.3. Because ROS defaults to streamlining the kernel and firmware, only the latest version of each firmware driver is reserved for ROS. When we built v1.5.3, the bnx2x driver was updated in the firmware upstream.

Combing through old issues:

• CPU Quota Bug Update linux kernel to fix cpu quota rancher/os#2972
• Wireguard Update linux kernel to fix cpu quota rancher/os#2972
• Docker in User Mode Ability to run Docker in User Mode Linux (diuid) on RancherOS rancher/os#2859
• Improved Intel QuickSync/GPU Support Intel QuickSync/GPU Support rancher/os#2849
• Ryzen CPU temp feature: Ask upgrade the kernel to 4.19 or provide a option to use it rancher/os#2728

[olljanat]

@PrplHaz4 makes sense.

Then I propose that we proceed on two tracks:

1. We keep updating 4.14.x kernels and keeps changes on that in minimum. I created 4.14.x branch for it and updated it to 4.14.202 version.
2. We make clear separation from Rancher OS by:

• Deciding new name for OS on We need new name (and logo) for community maintained Rancher OS #3
• Figure out on How/where you use Rancher OS? Please share your configuration? #6 that which kind of configurations people who have found here are using Rancher OS/plan to use with that renamed OS and drop services which are not used.
• Create first release on new name with next LTS kernel version (and probably Docker 20.10.x). Based on this it looks to be LTS kernel is released once of year but there is no LTS release from 2020 yet so I assume that 5.9.x kernel will be promoted to LTS later. I created v5.9.x branch now and updated it to 5.9.1 version.

Also looking through these makes me think some thought needs to be given to kernel vs kernel-extras...

Yep. With this kind of plan we most can include all modules which people are using to kernel (and also can try move some which are rarely used to kernel-extras). You can see from burmilla/os-kernel#3 what I needed to change to be able to pass 5.9.1 build. I also added support for menu based kernel configuration (instructions are on readme) which hopefully helps people to create pull requests on this area.

[ToeiRei]

A newer kernel adds fstrim support - that's why I had to update my debian kernels to 5.x

[olljanat]

FYI. First beta versions are now available for amd64 architecture (arm64/raspberry support is still WIP)

• With 4.14.202 kernel + Docker 19.03.13: https://github.com/rancher-os-community/os/releases/tag/v1.9.0-beta1
• With 5.9.1 kernel + Docker 20.10.0-beta1: https://github.com/rancher-os-community/os/releases/tag/v2.0.0-beta1
  and I quickly tested that I was able to boot VM from both of those ISO file.

If/when you find issues plz create own issues of those.

[olljanat]

Also looking through these makes me think some thought needs to be given to kernel vs kernel-extras...

@PrplHaz4 I see that you use "kernel-extras" package to get "nouveau" module. It looks to be on 3,5MB module so I can include it to kernel package but I wonder that why you also need kernel headers? Are building some custom modules?

[PrplHaz4]

Also looking through these makes me think some thought needs to be given to kernel vs kernel-extras...

@PrplHaz4 I see that you use "kernel-extras" package to get "nouveau" module. It looks to be on 3,5MB module so I can include it to kernel package but I wonder that why you also need kernel headers? Are building some custom modules?

Definitely not building any custom modules - it may be a remnant of troubleshooting the usb bus issues I was having on that system. I'd also guess nouveau is rare enough that it may warrant staying in extras - most people probably don't care about ancient nvidia gpu support :)

Sorry for the lack of good input - one of my favorite things about RancherOS is that it's been running without issues or tinkering for so long I don't quite remember all the answers...

[olljanat]

Looks 5.10.1 is marked as stable and it will be LTS version https://9to5linux.com/linux-kernel-5-10-lts-officially-released-this-is-whats-new Will update kernel repos to use that one.

[ToeiRei]

got a few small things regarding the kernel which I'd like to talk about

• Live patch support: it's currently enabled and I doubt that we'll use it
• Throwing out obsolete HW support for joysticks, bluetooth support,... (stuff that we do not need)
• Kernel 'flavors' if that's possible (like VM, real HW, ...)

[olljanat]

* Live patch support: it's currently enabled and I doubt that we'll use it

Good catch. Will disable it on next update.

* Throwing out obsolete HW support for joysticks, bluetooth support,...  (stuff that we do not need)

Yea well it a bit hard to be sure that we do no remove anything which someone might be using. That why I think that current way to split those to kernel-extras is nice. However target should be that no one is actually needed to use that package so we should include all needed modules from all known use cases to standard package.

Much harder part is detect those modules which are not used by anyone and which should be moved to kernel extras. That is area where subscriber would be able to help us #14

* Kernel 'flavors' if that's possible (like VM, real HW, ...)

We already have option to build VM images without firmwares. I took that from rancher#2810 and VMware + ProxmoxVE images are build with that option. Other proposals are welcome.

[ToeiRei]

I am currently building some debian kernels using recent sources (building kernels since around 1999) and I would suggest going at least 3 flavors here:

1. VM
   all vmware, KVM/qemu, xen, hyperV drivers

2. real HW
   as it is

3. special cases like Raspberry 'n friends

[olljanat]

Our build scripts on os-kernel already supports that. Then it just matter of someone to create VM configuration version. For Rapsberry we have anyway separate repo already os-rpi-kernel

@ToeiRei so you create PR to that repo which provides VM version of config with some extra info that what are actually differences, etc? And plz split it to two commits where first is just copy of current config to new name and second one contains actual changes so it is easier to review.

Btw. Current status of next beta version is that 5.10.1 works fine on amd64 and Raspberry Pi 3B+ but boot fails on Raspberry Pi 4B.

[ToeiRei]

I'll have to look into it as Virtualbox is my current problem on debian vm drivers only.

[olljanat]

Afaiu actually can generate kernel config automatically like this https://unix.stackexchange.com/a/507267 We just need to run it once on each hypervisor we want to support and combine results.

@ToeiRei can you try it with our latest 2.0.0 draft version which uses 5.10.1 kernel?

[ToeiRei]

I am aware of localmodconfig. It just does not solve my issue as it seems to be kms related.

[ToeiRei]

I did some benchmarks with custom kernels and I'm not really seeing any improvements of any kind for our docker workloads. Going the extra mile is just for space then. The only time I really see a big difference is running some gameservers with some really pervert configs to combat bad programming techniques of the game devs.

I'll be just cleaning out the configs and get the pull request ready.

[olljanat]

I ended up to to thinking that what if we start using k3OS kernel on BurmillaOS 2.x releases? It is safe to do because we have anyway locked v1.9.x versions to v4.14.x kernels.

It is worth to mention that currently k3OS kernels are far from optimal especially what comes to size but that we can improve by contributing to their kernel config development and we can publish separate virtual machines specific version of ISO like I proposed for them too rancher/k3os#650 Then that work would be much bigger impact and also users would be able to expect that all same devices are supported by both BurmillaOS and k3OS.

[olljanat]

FYI. Latest beta version uses now k3OS kernel https://github.com/burmilla/os/releases/tag/v2.0.0-beta3

Closing this one now.

[zimme]

I ended up to to thinking that what if we start using k3OS kernel on BurmillaOS 2.x releases? It is safe to do because we have anyway locked v1.9.x versions to v4.14.x kernels.

It is worth to mention that currently k3OS kernels are far from optimal especially what comes to size but that we can improve by contributing to their kernel config development and we can publish separate virtual machines specific version of ISO like I proposed for them too rancher/k3os#650 Then that work would be much bigger impact and also users would be able to expect that all same devices are supported by both BurmillaOS and k3OS.

It seems as though Suse (Rancher) specifically ibuildthecloud is working on a new k3os/rke2os that seens to be based on opensuse and seem to have a different build setup so using k30s' kernel might not be the best way forward as k3os' future might go in a bit of a different direction.

https://github.com/ibuildthecloud/k3os
As you can see this code base is different from the upstream k3os code base. Got this info from the rancher user slack k3os channel.

[zimme]

Scratch that... Just realized that that repo still uses the k3os kernel 🤦

Edit:

Actually it seems to be using the opensuse kernel... I need to look into this more.

[olljanat]

@zimme very interesting, thanks to pointing it out. So far stuff on k3os-kernel have been actually just set of scripts used to build kernel based on Ubuntu sources. I guess that those will works at least with 5.4 kernels even if k3OS migrates to opensuse kernel so then we have option either take that to use too or example start using direct pre-build kernel binaries from Ubuntu repos.