Backports to LTS kernel version & light patch for CFS

[xalt7x]

Hello @hamadmarri
First of all, thanks for your work!
I believe it's very important that Cachy is built on top of mainline scheduler. CFS shows comparable performance with MuQSS and BMQ, usable with RT application etc. The main problem is multitasking under heavy load. MuQSS really shined in this aspect but seems like it's dying. Also, just like BMQ it misses support for modern mainline features (schedutil/uclamp, nohz_full, cgroups). So if it's possible to achieve similar results with CFS, it looks like the best choice.
So I have a few questions:

1. Cachy looks like a lighter copy of CFS. Is it possible to have some lighter patch for CFS which adds HRRN policy? I'm personally fine with CFS "bloat". More important for me is if patch would survive until the kernel lifecycle ends.
2. I'm trying to stick with Ubuntu kernels and luckily this time for LTS we have good LTS kernel version 5.4. Next Ubuntu version will probably have kernel 5.8 but sadly it has major regression in Intel graphics performance. As Cachy now is more stable and feature-complete, do you plan to extend support to LTS kernel versions?

Thanks!

[hamadmarri]

Hello @hamadmarri
First of all, thanks for your work!
I believe it's very important that Cachy is built on top of mainline scheduler. CFS shows comparable performance with MuQSS and BMQ, usable with RT application etc. The main problem is multitasking under heavy load. MuQSS really shined in this aspect but seems like it's dying. Also, just like BMQ it misses support for modern mainline features (schedutil/uclamp, nohz_full, cgroups). So if it's possible to achieve similar results with CFS, it looks like the best choice.
So I have a few questions:

1. Cachy looks like a lighter copy of CFS. Is it possible to have some lighter patch for CFS which adds HRRN policy? I'm personally  fine with CFS "bloat". More important for me is if patch would survive until the kernel lifecycle ends.

2. I'm trying to stick with Ubuntu kernels and luckily this time for LTS we have good LTS kernel version 5.4. Next Ubuntu version will probably have kernel 5.8 but sadly it has major regression in Intel graphics performance. As Cachy now is more stable and feature-complete, do you plan to extend support to LTS kernel versions?

Thanks!

Hi @Alt37 ,

BMQ it misses support for modern mainline features (schedutil/uclamp, nohz_full, cgroups)

I tried BMQ before, it was superior especially in gaming. I am surprised that it doesn't support the above features, which might be the reason why it is lightweight and fast.

1. Cachy looks like a lighter copy of CFS. Is it possible to have some lighter patch for CFS which adds HRRN policy? I'm personally  fine with CFS "bloat". More important for me is if patch would survive until the kernel lifecycle ends.

The current Cachy version has just slight modifications to CFS (HRRN policy with doubly linked list, and Dynamic Priorities). It is possible to have a patch on top of fair.c but there are no guarantees that there will be no conflicts with future mainline changes. I could make a test patch for 5.8 that doesn't create a new cachy.c file and just do the modifications on fair.c with the ability to choose which scheduler before compiling.

2. I'm trying to stick with Ubuntu kernels and luckily this time for LTS we have good LTS kernel version 5.4. Next Ubuntu version will probably have kernel 5.8 but sadly it has major regression in Intel graphics performance. As Cachy now is more stable and feature-complete, do you plan to extend support to LTS kernel versions?

I don't know exactly how to extend support to LTS kernel versions. Could you please explain more about this point?
Do you mean Ubuntu's LTS kernel specifically?

Thank you

[xalt7x]

@hamadmarri

I tried BMQ before, it was superior especially in gaming. I am surprised that it doesn't support the above features, which might be the reason why it is lightweight and fast.

BMQ originates from BFS which doesn't fully support them as well. Authors of such different (from mainline) schedulers need to implement new features on their side.

It is possible to have a patch on top of fair.c but there are no guarantees that there will be no conflicts with future mainline changes.

I can't tell what is more future-proof and easier to maintain - patched CFS or Cachy. On the first sight patched CFS looks easier to me. If nothing major happen during version lifecycle you will most likely just offset code on fair.c. For Cachy you'll probably need to merge mainline changes (duplicate code from CFS commits).

1. I'm trying to stick with Ubuntu kernels and luckily this time for LTS we have good LTS kernel version 5.4. Next Ubuntu version will probably have kernel 5.8 but sadly it has major regression in Intel graphics performance. As Cachy now is more stable and feature-complete, do you plan to extend support to LTS kernel versions?

I don't know exactly how to extend support to LTS kernel versions. Could you please explain more about this point?
Do you mean Ubuntu's LTS kernel specifically?

I meant that you have Cachy for latest stable kernel version (5.8). LTS vesion (5.4) is different and I can't just apply latest (or in this case any) patch there. Some distributions (Arch, Manjaro, Raspberry Pi OS, Debian stable, sometimes Ubuntu) provide LTS kernels. Users may benefit from LTS version when something is wrong with "latest & greatest" kernel.

[hamadmarri]

@hamadmarri

I tried BMQ before, it was superior especially in gaming. I am surprised that it doesn't support the above features, which might be the reason why it is lightweight and fast.

BMQ originates from BFS which doesn't fully support them as well. Authors of such different (from mainline) schedulers need to implement new features on their side.

It is possible to have a patch on top of fair.c but there are no guarantees that there will be no conflicts with future mainline changes.

I can't tell what is more future-proof and easier to maintain - patched CFS or Cachy. On the first sight patched CFS looks easier to me. If nothing major happen during version lifecycle you will most likely just offset code on fair.c. For Cachy you'll probably need to merge mainline changes (duplicate code from CFS commits).

1. I'm trying to stick with Ubuntu kernels and luckily this time for LTS we have good LTS kernel version 5.4. Next Ubuntu version will probably have kernel 5.8 but sadly it has major regression in Intel graphics performance. As Cachy now is more stable and feature-complete, do you plan to extend support to LTS kernel versions?

I don't know exactly how to extend support to LTS kernel versions. Could you please explain more about this point?
Do you mean Ubuntu's LTS kernel specifically?

I meant that you have Cachy for latest stable kernel version (5.8). LTS vesion (5.4) is different and I can't just apply latest (or in this case any) patch there. Some distributions (Arch, Manjaro, Raspberry Pi OS, Debian stable, sometimes Ubuntu) provide LTS kernels. Users may benefit from LTS version when something is wrong with "latest & greatest" kernel.

Sure, making patches for LTS releases seems a good idea. I wounder which old version you think to stop at?
like v5.4? or go further down like 5.3 ...?

[xalt7x]

Sure, making patches for LTS releases seems a good idea. I wounder which old version you think to stop at?
like v5.4? or go further down like 5.3 ...?

In my opinion, it's 5.4, latest official LTS. Support for 5.3 is dropped by mainline. Among long-term distributions I guess only openSUSE maintains it (for Leap 15.2) but they have a lot of backports. So it there's a need to support 5.3 then only for openSUSE.
Another official LTS version is 4.19 (used in Debian 10). Honestly, I'm not sure if there will be enough interest among Debian Stable and openSUSE Leap users.

[sirlucjan]

Also, just like BMQ it misses support for modern mainline features (schedutil/uclamp, nohz_full, cgroups).

As far as I know, both BMQ and PDS support cgroups (at least minimally). This was introduced some time ago so that you could use the docker without any problems.

lucjan@archlinux ~ $ wc -l /sys/fs/cgroup/cpu/tasks
617 /sys/fs/cgroup/cpu/tasks
lucjan@archlinux ~ $ dmesg | grep pds
[    0.095186] sched/pds: PDS CPU Scheduler v5.8-r3 by Alfred Chen.

For the rest, full agreement. MUQSS doesn't support it either.

[hamadmarri]

https://github.com/hamadmarri/cachy-sched/tree/master/patches/cachy/v5.4

I am not able to test on my machine I got compression error. Please check if the patch works properly with v5.4.

Thank you

[xalt7x]

I tried to compile it. Unfortunately build fails

kernel/sched/cachy.c:9824:28: error: initialization of ‘void (*)(struct rq *, struct task_struct , bool)’ {aka ‘void ()(struct rq *, struct task_struct , _Bool)’} from incompatible pointer type ‘void ()(struct rq *, struct task_struct *)’ [-Werror=incompatible-pointer-types]
9824 | .set_next_task = set_next_task_fair,
| ^~~~~~~~~~~~~~~~~~
kernel/sched/cachy.c:9824:28: note: (near initialization for ‘fair_sched_class.set_next_task’)
CC arch/x86/kernel/traps.o
CC arch/x86/kernel/idt.o
cc1: some warnings being treated as errors

[hamadmarri]

-Werror

I think you need to disable -Werror from treating warnings as errors.
I doubled check the declaration of set_next_task in cachy.c, fair.c, and rt.c

they have similar definitions

cachy.c: static void set_next_task_fair(struct rq *rq, struct task_struct *p)
fair.c: static void set_next_task_fair(struct rq *rq, struct task_struct *p)
rt.c: static inline void set_next_task_rt(struct rq *rq, struct task_struct *p)

[xalt7x]

I think you need to disable -Werror from treating warnings as errors.

I commented appropriate line in Makefile and was able to compile

 # enforce correct pointer usage
-KBUILD_CFLAGS   += $(call cc-option,-Werror=incompatible-pointer-types)
+#KBUILD_CFLAGS   += $(call cc-option,-Werror=incompatible-pointer-types)

Unfortunately kernel hangs on boot without any warnings (at least I can't see them even without "quiet" option in GRUB). Base kernel version was Ubuntu's "linux-image-5.4.0-48-generic"
linux-image-5.8.0-20-generic + cachy-5.8-r5.patch compiles with default parameters and boots fine.

[hamadmarri]

I think you need to disable -Werror from treating warnings as errors.

I commented appropriate line in Makefile and was able to compile

 # enforce correct pointer usage
-KBUILD_CFLAGS   += $(call cc-option,-Werror=incompatible-pointer-types)
+#KBUILD_CFLAGS   += $(call cc-option,-Werror=incompatible-pointer-types)

Unfortunately kernel hangs on boot without any warnings (at least I can't see them even without "quiet" option in GRUB). Base kernel version was Ubuntu's "linux-image-5.4.0-48-generic"
linux-image-5.8.0-20-generic + cachy-5.8-r5.patch compiles with default parameters and boots fine.

Could you please provide a download link for "linux-image-5.4.0-48-generic" because I am not able to find 5.4.0-48 exactly in github linux repo. I checked out the tag v5.4 which seems the last one in 5.4.*

[xalt7x]

Could you please provide a download link for "linux-image-5.4.0-48-generic"

It's a distro kernel, you can download flles here
Place 3 files (linux_5.4.0-48.52.dsc , linux_5.4.0-48.52.diff.gz , linux_5.4.0.orig.tar.gz) in one folder and run
dpkg-source --no-check -x *.dsc
If you have trouble installing/booting Debian/Ubuntu, I can upload repacked archive with kernel somewhere.

[hamadmarri]

Could you please provide a download link for "linux-image-5.4.0-48-generic"

It's a distro kernel, you can download flles here
Place 3 files (linux_5.4.0-48.52.dsc , linux_5.4.0-48.52.diff.gz , linux_5.4.0.orig.tar.gz) in one folder and run
dpkg-source --no-check -x *.dsc
If you have trouble installing/booting Debian/Ubuntu, I can upload repacked archive with kernel somewhere.

Sure, I will try. If I made a cachy patch for longterm: 5.4.69 in https://www.kernel.org, would it work too on 5.4.0-48.52?

[xalt7x]

Sure, I will try. If I made a cachy patch for longterm: 5.4.69 in https://www.kernel.org, would it work too on 5.4.0-48.52?

Probably
5.4.0-48.52 (Ubuntu's ABI version) is built on top of 5.4.60 mainline version. I may try to rebuild "mainline" version (without major Ubuntu patches) - https://kernel.ubuntu.com/~kernel-ppa/mainline/v5.4.69/ . Last time I tried it wasn't that easy to get sources.

[hamadmarri]

Please see the attached patch. It is for v5.4.70, it may work with ubuntu version

and here the branch on linux tree
https://github.com/hamadmarri/linux/tree/cachy-5.4

and the tag
https://github.com/hamadmarri/linux/tree/cachy-5.4-r7

cachy-5.4-r7.zip

[hamadmarri]

Try this fixed patch for r7
https://github.com/hamadmarri/cachy-sched/blob/master/patches/v5.4/cachy-5.4-r7-fix1.patch

Tree
https://github.com/hamadmarri/linux/commits/cachy-5.4-r7-fix1

[xalt7x]

I can confirm that latest patch works with Ubuntu version.
Also I see that with version r7 you patch original files instead of code duplication in cachy.c. Now code is easier to read and could be easier to maintain. That's just awesome!
Now the only thing I could ask is to separate generic CFS tweaks into different patch
Like this part
-const_debug unsigned int sysctl_sched_nr_migrate = 32; +const_debug unsigned int sysctl_sched_nr_migrate = +IS_ENABLED(CONFIG_CACHY_SCHED) ? 256 : 32;
For a long tome Xanmod kernel maintainer changed this value "128", this tweak was also adopted by Zen kernel maintainer. Nowadays Xan sets it to "256", someone wants "64" etc. The point is that if this CFS tweak is not specific to Cachy maybe it shouldn't be included in the main patch. But that's just a small proposal, patch in it's current form looks good already .
Thanks again for all your work and effort!

[hamadmarri]

I can confirm that latest patch works with Ubuntu version.
Also I see that with version r7 you patch original files instead of code duplication in cachy.c. Now code is easier to read and could be easier to maintain. That's just awesome!
Now the only thing I could ask is to separate generic CFS tweaks into different patch
Like this part
-const_debug unsigned int sysctl_sched_nr_migrate = 32; +const_debug unsigned int sysctl_sched_nr_migrate = +IS_ENABLED(CONFIG_CACHY_SCHED) ? 256 : 32;
Initially Xan kernel maintainer tried value 128, it was also adopted by Zen kernel maintainer. Nowadays Xan sets this value to "256", someone wants "64" etc.The point is that if this CFS tweak is not specific to Cachy maybe it shouldn't be included in a main patch.
Thanks again for all your work and effort!

I can confirm that latest patch works with Ubuntu version.
Also I see that with version r7 you patch original files instead of code duplication in cachy.c. Now code is easier to read and could be easier to maintain. That's just awesome!
Now the only thing I could ask is to separate generic CFS tweaks into different patch
Like this part
-const_debug unsigned int sysctl_sched_nr_migrate = 32; +const_debug unsigned int sysctl_sched_nr_migrate = +IS_ENABLED(CONFIG_CACHY_SCHED) ? 256 : 32;
Initially Xan kernel maintainer tried value 128, it was also adopted by Zen kernel maintainer. Nowadays Xan sets this value to "256", someone wants "64" etc.The point is that if this CFS tweak is not specific to Cachy maybe it shouldn't be included in a main patch.
Thanks again for all your work and effort!

Hi @Alt37 ,

I agree that it is better to move out sysctl_sched_nr_migrate to separate batch. I will fix that on the next patch.
Please let me know about the interactivity/performance of r7 compared to CFS.

Thank you for your suggestions and support.