CPU governors/cpufreq missing in 20220120-1

[AdvancedFollower]

Describe the bug

After upgrading from 20211118 (5.10.63) to 20220120 (5.10.92), CPU governors are not available and the Pi is locked to 600 MHz.

Downgrading to 1.20211118 kernel/bootloader restores the functionality.

Steps to reproduce the behaviour

Apt upgrade to raspberrypi-kernel/stable 1:1.20220120-1 and raspberrypi-bootloader/stable 1:1.20220120-1 and reboot

Device (s)

Raspberry Pi 3 Mod. B+

System

Debian 11.2

Jan 20 2022 13:58:39
Copyright (c) 2012 Broadcom
version bd88f66f8952d34e4e0613a85c7a6d3da49e13e2 (clean) (release) (start_cd)

Linux DietPi 5.10.92-v8+ #1514 SMP PREEMPT Mon Jan 17 17:39:38 GMT 2022 aarch64 GNU/Linux

Logs

dmesg -l emerg,alert,crit,err produces no errors

Additional context

ls /sys/devices/system/cpu/cpu0:
cpu_capacity of_node power regs subsystem topology uevent
(no cpufreq)

G_HW_MODEL=3
G_HW_MODEL_NAME='RPi 3 Model B+ (aarch64)'
G_HW_ARCH=3
G_HW_ARCH_NAME='aarch64'
G_HW_CPUID=0
G_HW_CPU_CORES=4
G_DISTRO=6
G_DISTRO_NAME='bullseye'
G_ROOTFS_DEV='/dev/mmcblk0p2'
G_HW_UUID='f98c179a-a620-47de-a789-a46af87bf052'
G_RASPBIAN=0
G_HW_ONBOARD_WIFI=1
G_HW_REVISION='a020d3'
G_HW_PCB_REVISION=3
G_HW_MEMORY_SIZE=1024
G_HW_MANUFACTURER='Sony UK'

[pelwell]

It's working for me, running the current 64-bit Raspberry Pi OS release:

pi@raspberrypi:~ $ uname -a
Linux raspberrypi 5.10.92-v8+ #1514 SMP PREEMPT Mon Jan 17 17:39:38 GMT 2022 aarch64 GNU/Linux

pi@raspberrypi:~ $ vcgencmd version
Jan 20 2022 13:58:22 
Copyright (c) 2012 Broadcom
version bd88f66f8952d34e4e0613a85c7a6d3da49e13e2 (clean) (release) (start)

pi@raspberrypi:~ $ cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

pi@raspberrypi:~ $ ls /sys/devices/system/cpu/cpu0
cpu_capacity  cpufreq  of_node  power  regs  subsystem  topology  uevent

pi@raspberrypi:~ $ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 
ondemand

pi@raspberrypi:~ $ vcgencmd measure_clock arm
frequency(48)=600000000

pi@raspberrypi:~ $ echo performance | sudo tee /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor 
performance

pi@raspberrypi:~ $ vcgencmd measure_clock arm
frequency(48)=1400000000

[Joulinar]

@pelwell
Did you tried on RPi3B+? Because for me it looks like this.

root@DietPi3:~# uname -a
Linux DietPi3 5.10.92-v8+ #1514 SMP PREEMPT Mon Jan 17 17:39:38 GMT 2022 aarch64 GNU/Linux

root@DietPi3:~# vcgencmd version
Jan 20 2022 13:58:39
Copyright (c) 2012 Broadcom
version bd88f66f8952d34e4e0613a85c7a6d3da49e13e2 (clean) (release) (start_cd)
root@DietPi3:~#
root@DietPi3:~# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

root@DietPi3:~# ls /sys/devices/system/cpu/cpu0
cpu_capacity  of_node  power  regs  subsystem  topology  uevent

root@DietPi3:~# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
cat: /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor: No such file or directory

root@DietPi3:~# vcgencmd measure_clock arm
frequency(48)=600000000

root@DietPi3:~# echo performance | sudo tee /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
tee: /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor: No such file or directory
performance

root@DietPi3:~# vcgencmd measure_clock arm
frequency(48)=600000000

scaling_governor doesn't seems to exist.

[pelwell]

Yes - that output is from a 3B+.

Please try the official 64-bit image, not DietPi.

[Joulinar]

interesting point, running rpi-update on the very same system, seems to bring back scaling_governor

root@DietPi3:~# uname -a
Linux DietPi3 5.15.18-v8+ #1520 SMP PREEMPT Fri Feb 4 12:04:45 GMT 2022 aarch64 GNU/Linux

root@DietPi3:~# cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
ondemand

@MichaIng
maybe you like to have a look as well.

[MichaIng]

Also downgrading to the previous 1.20211118 kernel and bootloader solves it, so something is different with this particular version 1.20220120-1. Indeed a good idea to test with official RPi OS 64-bit. Kernel, bootloader and firmware are however the same, so unless it's something in config.txt or cmdline.txt I'm not sure what could affect something like CPUFreq kernel feature, especially something which affects the RPi 3B+ 🤔 exclusively.

EDIT: Commenting/removal of initial_turbo=20 could be tried (respectively adding it on RPi OS 64-bit). I remember that it broke CPU scheduling in the past with one of the first 5.4 kernel releases.

[AdvancedFollower]

Can confirm, commenting out initial_turbo=20 from config.txt solves it on my DietPi 8.0.2. I'm guessing initial_turbo isn't enabled by default on stock Debian, then?

[MichaIng]

Great find. Nope is it not enabled on stock RPi OS. However, it shouldn't break CPUFreq, and it's strange that it does only on RPi 3B+.

@pelwell
Any idea what may cause this what was introduced with last kernel/bootloader release and has been fixed (explicitly or implicitly by another change) in master/rpi-update?

I think we'll apply a live patch for RPi 3B+ to disable initial turbo and in case, when fixed until next DietPi release, offer to have it re-enabled during update. CPUFreq of course is much more important than the little boot time speedup with initial turbo.

[pelwell]

This is more @popcornmix's domain.

[popcornmix]

I've run what I believe @Joulinar ran on RpiOS bullseye with the same firmware and kernel and it appeared to behave as expected.

pi@b3plus:~ $ uname -a
Linux b3plus 5.10.92-v8+ #1514 SMP PREEMPT Mon Jan 17 17:39:38 GMT 2022 aarch64 GNU/Linux
pi@b3plus:~ $ vcgencmd version
Jan 20 2022 13:58:22 
Copyright (c) 2012 Broadcom
version bd88f66f8952d34e4e0613a85c7a6d3da49e13e2 (clean) (release) (start)
pi@b3plus:~ $ cat /etc/os-release
PRETTY_NAME="Raspbian GNU/Linux 11 (bullseye)"
NAME="Raspbian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=raspbian
ID_LIKE=debian
HOME_URL="http://www.raspbian.org/"
SUPPORT_URL="http://www.raspbian.org/RaspbianForums"
BUG_REPORT_URL="http://www.raspbian.org/RaspbianBugs"
pi@b3plus:~ $ ls /sys/devices/system/cpu/cpu0
cpu_capacity  cpufreq  of_node  power  regs  subsystem  topology  uevent
pi@b3plus:~ $ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
ondemand
pi@b3plus:~ $ vcgencmd measure_clock arm
frequency(48)=600000000
pi@b3plus:~ $ echo performance | sudo tee /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
performance
pi@b3plus:~ $ vcgencmd measure_clock arm
frequency(48)=1400000000
pi@b3plus:~ $ 

[Joulinar]

@popcornmix
It seems the difference is, if you set initial_turbo=20 #4875 (comment). As this is not a default value, it most probably working on plain RPi OS

[popcornmix]

Yes, I had added that.

pi@b3plus:~ $ vcgencmd get_config initial_turbo
initial_turbo=20

cpufreq governor is still working. Also tried with initial_turbo=60 and it works.

[MichaIng]

Hmm, so removing initial turbo from our config solves it but adding it to RPi OS does not cause it? An issue with a combination of initial turbo and something else? There is no overclocking set on RPi 3B+, so what may be closest related is temp_limit and probably gpu_mem_*. Otherwise testing our config.txt (+ arm_64bit=1 which is added later on 64-bit image generation) on RPi OS as a whole. For comparison, here the RPi OS 64-bit config: https://github.com/RPi-Distro/pi-gen/blob/arm64/stage1/00-boot-files/files/config.txt

Sadly I have no 3B+ here for testing. While the issue is solved with next LTS and we have a workaround until then, investigating the underlying reason may still be valuable, probably there is some specific timing involved which may reappear any time later when other things change.

[MichaIng]

Another question is whether only the 64-bit kernel is affected or whether the same issue appears when using the ARMv6 (Raspbian-based) or ARMv7 (Debian-based) image on RPi 3B+.

[AdvancedFollower]

@MichaIng
Yes, it works on DietPi ARMv7, just did a fresh install of https://dietpi.com/downloads/images/DietPi_RPi-ARMv7-Bullseye.7z on my 3B+

uname -a
Linux DietPi 5.10.92-v7+ #1514 SMP Mon Jan 17 17:36:39 GMT 2022 armv7l GNU/Linux
vcgencmd version
Jan 20 2022 13:58:39
Copyright (c) 2012 Broadcom
version bd88f66f8952d34e4e0613a85c7a6d3da49e13e2 (clean) (release) (start_cd)
cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
ls /sys/devices/system/cpu/cpu0
cpu_capacity cpufreq of_node power subsystem topology uevent
cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
schedutil
vcgencmd measure_clock arm
frequency(48)=900096000
vcgencmd get_config initial_turbo
initial_turbo=20
echo performance | sudo tee /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
performance
vcgencmd measure_clock arm
frequency(48)=1400000000

[pelwell]

cpufreq governor is still working. Also tried with initial_turbo=60 and it works.

Same here - cpufreq is working fine running the most recent 64-bit kernel (5.15.21-v8+) on a 3B+ with initial_turbo=60.

[AdvancedFollower]

So it seems to be only on DietPi, only with 5.10.92-v8+/ARM64, only on the Pi 3B+, and only when Initial Turbo is enabled. And it magically fixed itself with the 5.15.21-v8+ kernel.

As long as it doesn't magically break itself again with the next kernel after that I guess that's good enough...

[pelwell]

Rewinding to rpi-5.10.92-v8+ (sudo rpi-update b4145cfaa) on 3B+ with initial_turbo=60 and RPiOS still works.

[MichaIng]

Steps for further debugging could be: #4875 (comment)
Either commenting other config.txt settings on DietPi which are present only on DietPi by default, while leaving initial turbo enabled, or adding those settings on RPi OS, in addition to initial turbo.

[pelwell]

Booting a 3B+ from a RPiOS image with 5.10.92 kernal and the DietPi config.txt (+arm_64bit=1 because the 32-bit kernel is still present) gives a working cpufreq.

It's not the config.txt settings.

[MichaIng]

I cannot imagine at all that something from cmdline.txt may cause this:

root=PARTUUID=$(findmnt -Ufnro PARTUUID -M /) rootfstype=ext4 rootwait fsck.repair=yes net.ifnames=0 logo.nologo console=serial0,115200 console=tty1

Otherwise what shall affect sysfs/whether CPUFreq is loaded or not, given we use the official raspberrypi-kernel, raspberrypi-bootloader, libraspberrypi0, libraspberrypi-bin and raspberrypi-sys-mods from archive.raspberrypi.org without any other modification to the contained files or configs 🤔?

We use different (no) filesystem feature flags on image generation, while pi-gen seems to at least disable huge_file, not sure whether metadata_csum and 64bit are still disabled? Present on one script but not in the others, seems to be only set by pi-gen when USE_QCOW2 env var was set explicitly. However, I cannot imaging how this shall affect anything related. Partitioning and filesystems are otherwise is exactly the same, regarding type, offset and flags.

@AdvancedFollower
Could you paste another full dmesg output from DietPi, with initial turbo set? A serial console really would be helpful 🤔.

[tdvantine]

I am having this exact issue, no cpu governor. However, changing turbo to 60 didn't fix it, commenting it out of config.txt solved the issue for me and brought back the governors. Below you will find the same commands from the others above, but with a big difference of being on a Pi4B:

Hardware	: BCM2835
Revision	: c03111
Serial		: 100000007138d936
Model		: Raspberry Pi 4 Model B Rev 1.1

initial_turbo=20

dietpi@Taskmaster:~$ uname -a
Linux Taskmaster 5.10.92-v8+ #1514 SMP PREEMPT Mon Jan 17 17:39:38 GMT 2022 aarch64 GNU/Linux

dietpi@Taskmaster:~$ vcgencmd version
Jan 20 2022 13:56:48 
Copyright (c) 2012 Broadcom
version bd88f66f8952d34e4e0613a85c7a6d3da49e13e2 (clean) (release) (start)

dietpi@Taskmaster:~$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

dietpi@Taskmaster:~$ ls /sys/devices/system/cpu/cpu0
cpu_capacity  of_node  power  regs  subsystem  topology  uevent

dietpi@Taskmaster:~$ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
cat: /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor: No such file or directory

dietpi@Taskmaster:~$ vcgencmd measure_clock arm

dietpi@Taskmaster:~$ echo performance | sudo tee /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
tee: /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor: No such file or directory
performance

dietpi@Taskmaster:~$ vcgencmd measure_clock arm
frequency(48)=300111328

No Turbo (working)

dietpi@Taskmaster:~$ uname -a
Linux Taskmaster 5.10.92-v8+ #1514 SMP PREEMPT Mon Jan 17 17:39:38 GMT 2022 aarch64 GNU/Linux

dietpi@Taskmaster:~$ vcgencmd version
Jan 20 2022 13:56:48 
Copyright (c) 2012 Broadcom
version bd88f66f8952d34e4e0613a85c7a6d3da49e13e2 (clean) (release) (start)

dietpi@Taskmaster:~$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

dietpi@Taskmaster:~$ ls /sys/devices/system/cpu/cpu0
cpu_capacity  cpufreq  of_node	power  regs  subsystem	topology  uevent

dietpi@Taskmaster:~$ cat /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
conservative

dietpi@Taskmaster:~$ vcgencmd measure_clock arm
frequency(48)=300058592

dietpi@Taskmaster:~$ echo performance | sudo tee /sys/devices/system/cpu/cpu0/cpufreq/scaling_governor
performance

dietpi@Taskmaster:~$ vcgencmd measure_clock arm
frequency(48)=1500345728

[MichaIng]

Very confusing, we have a bunch of RPi 4 systems where this issue does clearly not appear, probably the specific PCB revision 1.1?

However, to me this points more into the direction that there is a general issue with how firmware and CPUFreq play together, triggered by specific timings (or so) about what is when (not) loaded during boot, which are not met in most cases.

Does apt install rpi-update && rpi-update solve it as well in your case?

[wulfy23]

https://forum.openwrt.org/t/rpi4-community-build/69998/1986?u=wulfy23
https://forum.openwrt.org/t/rpi4-community-build/69998/1991?u=wulfy23

165bd7bc5622ee1c721aa5da9af68935075abedd + kernel 5.10.92-ish

edit: hash above was wrong re-tested with 5.10.100

boot-DAT-ELF_20220115_175984a6dc4c321553e295cf9679a6020231caf4/ GOOD
boot-DAT-ELF_20220117_20c5829b0afe5b99d7e807ce90c382bd6997993f/ NOTGOOD<PROBLEM
boot-DAT-ELF_20220118_3f20b832b27cd730deb6419b570f31a98167eef6/ STILLNOTGOOD
boot-DAT-ELF_20220121_827fdd073638fa7b7292d1148fe0af7465111eae/ STILLNOTGOOD
boot-DAT-ELF_20220125_9c04ed2c1ad06a615d8e6479806ab252dbbeb95a/ GOOD>FIXED
boot-DAT-ELF_20220128_e5013c9d596e1a392e5193bc8cb822966b9c7d7c/ GOOD
boot-DAT-ELF_20220219_a6496ae5cd5e9b4cd5b38f7274fa94dce315fa7a/ GOODCURRENT
[    0.028033] raspberrypi-firmware soc:firmware: Attached to firmware from 2022-02-04 12:05, variant start
[    0.032038] raspberrypi-firmware soc:firmware: Firmware hash is a26faf97e3bf76bcc23949d7cdab2f96f399a0c3

reverting to some 6months older elf-dat resolved...

[MichaIng]

Thanks, it makes me feel better when it is not limited to DietPi 😅. Is initial_turbo set on OpenWRT by default, so that this is one combining element for triggering the issue?

[wulfy23]

nope... this is all 5.10.92 and the newer firmware from around that date...

(approx 100 users of identical code with all sorts of config.txt stuff and all had the issue)

[popcornmix]

@wulfy23 posting output of
vcgencmd version
uname -a
and contents of config.txt and cmdline.txt would be useful (when cpufreq governor is missing).

[wulfy23]

config.txt or cmdline.txt play no role as discussed...

dca632 /usbstick 53°# cat /boot/cmdline.txt 
console=ttyAMA1,115200 root=PARTUUID=02070430-02 rootfstype=squashfs,ext4 rootwait fsckparts workqueue.power_efficient=0 dwc_otg.lpm_enable=0 usbcore.autosuspend=-1 snd_bcm2835.enable_headphones=1 snd_bcm2835.enable_hdmi=1 snd_bcm2835.enable_compat_alsa=0
dca632 /usbstick 54°# grep '^[a-z]' /boot/config.txt /boot/distroconfig.txt 
/boot/config.txt:enable_uart=1
/boot/config.txt:uart_2ndstage=1
/boot/config.txt:include distroconfig.txt
/boot/config.txt:arm_64bit=1
/boot/config.txt:boot_delay=3
/boot/config.txt:dtoverlay=uart2
/boot/config.txt:dtoverlay=led70
/boot/config.txt:dtparam=i2c_arm=on
/boot/config.txt:dtparam=sd_poll_once
/boot/distroconfig.txt:dtoverlay=disable-bt

vcgencmd version I don't collect (but will do so next time such an event occurs if it helps)... I have posted the git commit hash above that you'd have to extrapolate from

or maybe I do?

[    0.000000] Linux version 5.10.92 (builder@buildhost) (aarch64-openwrt-linux-musl-gcc (OpenWrt GCC 11.2.0 r18609-266b5c83c3) 11.2.0, GNU ld (GNU Binutils) 2.37) #0 SMP Mon Jan 17 11:38:43 2022
[    0.000000] Machine model: Raspberry Pi 4 Model B Rev 1.1

dca632 /usbstick 55°# cat _zPREMOVED/pdmesg-202201181413-5.0.19-7.log | grep firmware
[    0.049993] raspberrypi-firmware soc:firmware: Attached to firmware from 2022-01-17T19:20:34, variant start
[    0.060003] raspberrypi-firmware soc:firmware: Firmware hash is bd34f55ef7b01b0a367f131060b561a2a58b80bb

[    0.368364] cpu cpu0: Cannot get clock for CPU0
[    0.368386] raspberrypi-cpufreq: probe of raspberrypi-cpufreq failed with error -2

and should you need the image in question;
http://rpi4.wulfy23.info/builds/devel/rpi-4_snapshot_5.0.19-7_r18609_extra_popcornmix/rpi4.64-snapshot-27266-5.0.19-7-r18609-ext4-fac.img.gz
(allow 3mins for the device to initialize on firstboot, web-ui/ssh is accessible at 192.168.1.1 from dhcp enabled client)

[root@dca632 / 39°]# vcgencmd bootloader_version
Apr 29 2021 17:11:25