[RFE] AMD CPU scaling drivers

[spnngl]

Current situation

Currently AMD's scaling drivers are not compiler in flatcar's kernel (stable & beta channel)

 ~ # zgrep PSTATE /proc/config.gz
CONFIG_X86_INTEL_PSTATE=y
# CONFIG_X86_AMD_PSTATE is not set
# CONFIG_X86_AMD_PSTATE_UT is not set

Impact

AMD new EPYC + ryzen CPU use acpi-cpufreq instead of amd-pstate epp mode reducing performance.

Ideal future situation

Enabling CONFIG_X86_AMD_PSTATE + CONFIG_HZ_{100,250,300} in kernel config for future release.

Implementation options

Enabling CONFIG_X86_AMD_PSTATE + CONFIG_HZ_{100,250,300} in kernel config for future release.

[jepio]

amd-pstate will be added, thanks for catching this.

Any reason you think CONFIG_HZ=1000 is bad? We also have CONFIG_NO_HZ_IDLE.

[spnngl]

Great !

Could we also get X86_AMD_PSTATE_DEFAULT_MODE=3 ?

Any reason you think CONFIG_HZ=1000 is bad? We also have CONFIG_NO_HZ_IDLE.

We would like to avoid a lot of interruption on heavily loaded servers and have the ability to always get 100Hz and periodic_hz.
At least to benchmark it

[jepio]

@spnngl any feedback on the images?

[spnngl]

Not yet, kubecon took most of my last week.
I'll test it soon

[spnngl]

I tested it on one of our server with AMD EPYC 9654 96-Core Processor and 512GB RAM.

amd-pstate-epp working perfectly, better frequencies handling and works out of the box.

Following this flatcar upgrade and all associated changes we got a 2% perf improvements (sysbench results) and especially a drastic decrease in irq/context switches that causes us some issues on real workload.

CONFIG_HZ=100 is maybe a little too low as seen on stress-ng graphs, maybe 250 or 300 is enough + CONFIG_NO_HZ_IDLE to avoid some useless tick.

Flatcar stable 3815.2.1

~ # uname -a
Linux 6.1.81-flatcar #1 SMP PREEMPT_DYNAMIC Sun Mar 17 21:32:50 -00 2024 x86_64 AMD EPYC 9654 96-Core Processor AuthenticAMD GNU/Linux

~ # cat /etc/os-release
NAME="Flatcar Container Linux by Kinvolk"
ID=flatcar
ID_LIKE=coreos
VERSION=3815.2.1
VERSION_ID=3815.2.1
BUILD_ID=2024-03-17-2158
SYSEXT_LEVEL=1.0
PRETTY_NAME="Flatcar Container Linux by Kinvolk 3815.2.1 (Oklo)"
ANSI_COLOR="38;5;75"
HOME_URL="https://flatcar.org/"
BUG_REPORT_URL="https://issues.flatcar.org"
FLATCAR_BOARD="amd64-usr"
CPE_NAME="cpe:2.3:o:flatcar-linux:flatcar_linux:3815.2.1:*:*:*:*:*:*:*"

 ~ # zgrep _HZ /proc/config.gz
CONFIG_NO_HZ_COMMON=y
# CONFIG_HZ_PERIODIC is not set
CONFIG_NO_HZ_IDLE=y
# CONFIG_NO_HZ_FULL is not set
CONFIG_NO_HZ=y
# CONFIG_HZ_100 is not set
# CONFIG_HZ_250 is not set
# CONFIG_HZ_300 is not set
CONFIG_HZ_1000=y
CONFIG_HZ=1000

~ # zgrep PSTATE /proc/config.gz
CONFIG_X86_INTEL_PSTATE=y
# CONFIG_X86_AMD_PSTATE is not set
# CONFIG_X86_AMD_PSTATE_UT is not set

~ # cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_driver
acpi-cpufreq
~ # cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
schedutil
~ # cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_frequencies
2400000 1900000 1500000
~ # cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors
performance schedutil

sysbench

Used this sysbench cpu --histogram=on --threads=256 --time=600 run to generate workload easily.

stress-ng

Used this stress-ng -M --all 10 --class=cpu --cpu=256 --perf -t 600 to generate workload.

Flatcar 9999.9.9+amd-pstate-hz100

NB: to use amd-pstate instead of acpi-cpufreq+schedutil I had to modify /usr/share/oem/grub.cfg

~ # cat /usr/share/oem/grub.cfg
# Flatcar GRUB settings

set oem_id="openstack"
set linux_append="$linux_append amd_pstate=active amd_pstate.shared_mem=1"

~ # uname -a
Linux 6.6.21-flatcar #1 SMP PREEMPT_DYNAMIC Tue Mar 19 17:04:32 -00 2024 x86_64 AMD EPYC 9654 96-Core Processor AuthenticAMD GNU/Linux

~ # cat /etc/os-release
NAME="Flatcar Container Linux by Kinvolk"
ID=flatcar
ID_LIKE=coreos
VERSION=9999.9.9+amd-pstate-hz100
VERSION_ID=9999.9.9
BUILD_ID=amd-pstate-hz100
SYSEXT_LEVEL=1.0
PRETTY_NAME="Flatcar Container Linux by Kinvolk 9999.9.9+amd-pstate-hz100 (Oklo)"
ANSI_COLOR="38;5;75"
HOME_URL="https://flatcar.org/"
BUG_REPORT_URL="https://issues.flatcar.org"
FLATCAR_BOARD="amd64-usr"
CPE_NAME="cpe:2.3:o:flatcar-linux:flatcar_linux:9999.9.9+amd-pstate-hz100:*:*:*:*:*:*:*"

~ # zgrep _HZ /proc/config.gz
CONFIG_HZ_PERIODIC=y
# CONFIG_NO_HZ_IDLE is not set
# CONFIG_NO_HZ_FULL is not set
# CONFIG_NO_HZ is not set
CONFIG_HZ_100=y
# CONFIG_HZ_250 is not set
# CONFIG_HZ_300 is not set
# CONFIG_HZ_1000 is not set
CONFIG_HZ=100

~ # zgrep PSTATE /proc/config.gz
CONFIG_X86_INTEL_PSTATE=y
CONFIG_X86_AMD_PSTATE=y
CONFIG_X86_AMD_PSTATE_DEFAULT_MODE=3
# CONFIG_X86_AMD_PSTATE_UT is not set

~ # cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_driver
amd-pstate-epp
~ # cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
performance
~ # cat /sys/devices/system/cpu/cpu0/cpufreq/amd_pstate_max_freq
3709000
~ # cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_available_governors
performance powersave

sysbench

Used this sysbench cpu --histogram=on --threads=256 --time=600 run to generate workload easily.

stress-ng

Used this stress-ng -M --all 10 --class=cpu --cpu=256 --perf -t 600 to generate workload.