Switching to ondemand cpufreq governor

[ThomasKaiser]

I would suggest either patching kernel config to use ondemand by default (powersave is the default and not the best choice), adding cpufrequtils package by default or adding this to /etc/rc.local:

echo ondemand >/sys/devices/system/cpu/cpu0/cpufreq/scaling_governor

With the settings you use now the CPU cores are limited to 600 MHz. While I use a custom Armbian build on RPi 3 now where this isn't a problem the users of your distro should be able to benefit from twice the performance too.

With the above changes the stupid sysbench pseudo benchmark finishes in 3 seconds compared to 48 seconds as usual. With powersave it's 6.2 seconds. Just try it out yourself:

sysbench --test=cpu --cpu-max-prime=10000 run --num-threads=$(grep -c '^processor' /proc/cpuinfo)

Maybe it's also worth executing dpkg --add-architecture armhf and adding the following to /etc/apt/sources.list.d/raspbian.list:

deb [arch=armhf] https://archive.raspbian.org/raspbian stretch main contrib non-free
deb-src [arch=armhf] https://archive.raspbian.org/raspbian stretch main contrib non-free

(requires adding apt-transport-https though). Me still searching for the right packages containing vcgencmd command to talk to the mythical firmware having control over the ARM cores.

[bamarni]

Thanks for the tip, seems like a sane default indeed. However while trying your benchmark with the ondemand or performance governor I couldn't see the difference :

root@raspberrypi:/home/pi# sysbench --test=cpu --cpu-max-prime=10000 --num-threads=4 run
sysbench 0.4.12:  multi-threaded system evaluation benchmark

Running the test with following options:
Number of threads: 4

Doing CPU performance benchmark

Threads started!
Done.

Maximum prime number checked in CPU test: 10000


Test execution summary:
    total time:                          6.1502s
    total number of events:              10000
    total time taken by event execution: 24.5836
    per-request statistics:
         min:                                  1.23ms
         avg:                                  2.46ms
         max:                                  6.49ms
         approx.  95 percentile:               2.55ms

Threads fairness:
    events (avg/stddev):           2500.0000/2.92
    execution time (avg/stddev):   6.1459/0.00

I've notice though a yellow lightning image on the top-right durig the sysbench when using a direct screen instead of ssh, my guess is that the kernel is getting bypassed by the firmware on this, maybe because of over-heating or insufficient power supply? Mine is 2A, I'd be curious to know yours?

As for the vcgencmd I don't think it's available as a package, but it should be buildable from source on arm64 : raspberrypi/userland#347

I gave it a try but it couldn't run because VCHIQ is not included in the arm64 kernel at this point (cf. https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=158127).

[ThomasKaiser]

my guess is that the kernel is getting bypassed by the firmware on this, maybe because of over-heating or insufficient power supply? Mine is 2A, I'd be curious to know yours?

Amperage ratings aren't worth a look here since the problem is undervoltage. And that's why we/you would need vcgencmd since we have no chance to get a clue what the primary OS running on the VideoCore is doing without that command: https://forum.armbian.com/index.php?/topic/1665-rfc-using-a20-board-with-armbian-as-powermeter/&do=findComment&comment=13765

Thanks for the other pointers, will look into soon but currently busy with more capable hardware. Just as a recommendation: without good Micro USB cables voltage drops too much and ARM cores are limited to 600 MHz anway. I would do a search on eg. Amazon for '20awg micro usb' (and choose the Tronsmart set, at least the 30cm cable is personally verified to transport close to 10W with just 0.4V drop)

[bamarni]

It was indeed a voltage drop problem, I've tried with a similar power supply which has a much shorter cable and the lightning didn't show up, benchmark took only 3 seconds as you mentioned!

I'll definitely configure this during kernel build for the next release, thanks again 👍

[ThomasKaiser]

BTW: I allowed myself to cite this 'Micro USB related performance problem' over there (since it's important that people start to understand the impact of this unfortunate connector leading to all sorts of problems you would not even think about in the first place).

I'll continue to use your distro the next weeks to do some compatibility testing to prepare stuff like this to be compatible with (arm64) Stretch and also to get some experiences before we allow Stretch in Armbian's build system. So while I'm surely not the average user I hope it's ok bringing stuff to your attention with more issues here that's IMO worth a look?

[bamarni]

Sure, I'd really appreciate to get more advice like this, I'm also making this project to learn more about linux.

[ThomasKaiser]

Sorry, I'm too lazy to open an own repo for this but I think this should be 'backed up to the internet' and you with a 64-bit Raspberry distro have exactly the target audience who needs to know this :)

Background info: http://tech.scargill.net/a-question-of-lifespan/ (yeah, the average RPi user not trusting in SD cards is also partially related to under-voltage)

Additional info: I asked 7 Raspberry 3 users I personally know to try out the script while running sysbench --test=cpu --cpu-max-prime=10000 --num-threads=4 run, 6 came back to me with results in the meantime and all of them suffer from under-voltage. That's just sad.

Still missing component: vcgencmd

#!/bin/bash
#
# Watch your Pi! RPi foundation knows that Micro USB for DC-IN
# is sh*t but also doesn't give a sh*t. Voltage drops caused by
# average USB cables cause all sorts of instabilities and also
# data corruption but instead of fixing the problem they masked
# it in their 'firmware'. The main CPU on the Raspberry monitors
# voltage drops and then acts on accordingly. If heavy voltage
# drops directly after startup are monitored the firmware lowers
# the core voltage available to CPU cores and also caps CPU
# clockspeed. If voltage drops aren't that severe a more flexible
# approach is used and at least you gain performance back after
# periods of heavy load.
#
# The monitoring script prints SoC temperature, sysfs clockspeed
# (and real clockspeed if differing -- this can happen!),
# 'vcgencmd get_throttled' bits and core voltage. To interpret
# the 'get_throttled' bits please refer to this (based on tests
# on RPi 3 descriptions for bits 1/2 and 17/18 should be exchanged!)
#
# 0: under-voltage
# 1: arm frequency capped
# 2: currently throttled 
# 
# 16: under-voltage has occurred
# 17: arm frequency capped has occurred
# 18: throttling has occurred
#
# Background info: http://preview.tinyurl.com/mmwjfwy and
# http://tech.scargill.net/a-question-of-lifespan/
#
# With a crappy PSU and/or Micro USB cable output looks like this
# on a RPi 3:
#
# 44.0'C  600 MHz 1010000000000000000 1.2V
# 44.5'C  600 MHz 1010000000000000000 1.2V
# 44.0'C  600 MHz 1010000000000000101 1.2V
# 44.0'C  600 MHz 1010000000000000101 1.2V
# 44.0'C  600 MHz 1010000000000000101 1.2V
# 44.5'C  600 MHz 1010000000000000000 1.2V
# 45.1'C  600 MHz 1010000000000000101 1.2V
# 
# With an ok-ish cable it looks like this (when running cpuburn-a53):
# 
# 48.3'C 1200 MHz 0000000000000000000 1.3312V
# 48.3'C 1200 MHz 0000000000000000000 1.3312V
# 48.3'C 1200 MHz 0000000000000000000 1.3312V
# 48.3'C 1200 MHz 0000000000000000000 1.3312V
# 50.5'C 1200 MHz 0000000000000000000 1.3312V
# 56.4'C  600 MHz 0000000000000000000 1.2V
# 54.8'C  600 MHz 1010000000000000101 1.2V
# 55.3'C  600 MHz 1010000000000000101 1.2V
# 55.8'C  600 MHz 1010000000000000101 1.3312V
# 53.7'C  600 MHz 1010000000000000101 1.2V
# 51.5'C  600 MHz 1010000000000000101 1.2V
# 51.0'C  600 MHz 1010000000000000101 1.2V
# 
# And only by bypassing the crappy connector you can enjoy RPi 3
# performing as it should (please note, there's a heatsink on my RPi
# -- without throttling would start and then reported clockspeed
# numbers start to get funny):
# 
# 75.2'C 1200 MHz 1010000000000000000 1.3250V
# 75.8'C 1200 MHz 1010000000000000000 1.3250V
# 75.8'C 1200 MHz 1010000000000000000 1.3250V
# 76.3'C 1200 MHz 1010000000000000000 1.3250V
# 76.3'C 1200 MHz 1010000000000000000 1.3250V
# 73.6'C 1200 MHz 1010000000000000000 1.3250V
# 72.0'C 1200 MHz 1010000000000000000 1.3250V
# 70.4'C 1200 MHz 1010000000000000000 1.3250V
#
# Now with a pillow on top for some throttling:
#
# 82.2'C 1200/ 947 MHz 1110000000000000010 1.3250V
# 82.7'C 1200/ 933 MHz 1110000000000000010 1.3250V
# 82.7'C 1200/ 931 MHz 1110000000000000010 1.3250V
# 82.7'C 1200/ 918 MHz 1110000000000000010 1.3250V
# 82.2'C 1200/ 935 MHz 1110000000000000010 1.3250V
# 79.9'C 1200/1163 MHz 1110000000000000000 1.3250V
# 75.8'C 1200 MHz 1110000000000000000 1.3250V
#
# And here on RPi 2 with crappy USB cable and some load
#
# 50.8'C  900 MHz 1010000000000000000 1.3125V
# 49.8'C  900 MHz 1010000000000000000 1.3125V
# 49.8'C  900/ 600 MHz 1010000000000000101 1.2V
# 49.8'C  900/ 600 MHz 1010000000000000101 1.2V
# 48.7'C  900/ 600 MHz 1010000000000000101 1.2V
# 49.2'C  900/ 600 MHz 1010000000000000101 1.2V
# 48.7'C  900 MHz 1010000000000000000 1.3125V
# 46.5'C  900 MHz 1010000000000000000 1.3125V
#
# The funny thing is that while the kernel thinks it's running
# with 900 MHz (performance governor) in reality the 'firmware'
# throttles down to 600 MHz but no one knows :)

echo -e "To stop simply press [ctrl]-[c]\n"
Maxfreq=$(( $(awk '{printf ("%0.0f",$1/1000); }'  </sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq) -15 ))
while true ; do
	Health=$(perl -e "printf \"%19b\n\", $(vcgencmd get_throttled | cut -f2 -d=)")
	Temp=$(vcgencmd measure_temp | cut -f2 -d=)
	RealClockspeed=$(vcgencmd measure_clock arm | awk -F"=" '{printf ("%0.0f",$2/1000000); }' )
	SysFSClockspeed=$(awk '{printf ("%0.0f",$1/1000); }' </sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq)
	CoreVoltage=$(vcgencmd measure_volts | cut -f2 -d= | sed 's/000//')
	if [ ${RealClockspeed} -ge ${Maxfreq} ]; then
		echo -e "${Temp}$(printf "%5s" ${SysFSClockspeed}) MHz $(printf "%019d" ${Health}) ${CoreVoltage}"
	else
		echo -e "${Temp}$(printf "%5s" ${SysFSClockspeed})/$(printf "%4s" ${RealClockspeed}) MHz $(printf "%019d" ${Health}) ${CoreVoltage}"
	fi
	sleep 5
done

[bamarni]

Nice one, so you're running this script in parallel from the sysbench right?

I'm working on a pi64-config CLI tool providing a few helpers (configure timezone / locales, wifi, etc.), I'll try to add a Check CPU frequency action for this.

[ThomasKaiser]

Nice one, so you're running this script in parallel from the sysbench right?

Yep, unfortunately the vcgencmd queries are the most important thing since wrt cpufreq the kernel is cheating on us (will report on a RPi 3 still running at 1200 MHz even if in reality already throttled down to 601 MHz, only at exactly 600 MHz both numbers match again).

Regarding pi64-config: Great idea! You might get some inspiration/code here https://github.com/igorpecovnik/Debian-micro-home-server

We include the tool in the meantime as armbian-config in our OS images and call for monitoring purposes then armbianmonitor -m which could be replaced with the above script on your distro or maybe combined with armbianmonitor's reporting.

[ThomasKaiser]

Just to let you know: I chose your pi64 image as basis for latest official Raspberry Pi OpenMediaVault image: https://forum.openmediavault.org/index.php/Thread/18991 (just to ensure that 0% Raspbian is involved and to ship with a 64-bit mainline kernel for adventourus users).

Also an improved version of the above 'health check' script is included as /usr/sbin/raspimon (still needing vcgencmd to talk to the 'firmware'):

 #!/bin/bash
 #
 # Rasperry Pi Health script. (c) 2017 by tkaiser (Thomas Kaiser)
 #
 # Initial version and some more information can be found here:
 # https://github.com/bamarni/pi64/issues/4#issuecomment-292707581
 
 echo -e "To stop simply press [ctrl]-[c]\n"
 Maxfreq=$(( $(awk '{printf ("%0.0f",$1/1000); }'  </sys/devices/system/cpu/cpu0/cpufreq/scaling_max_freq) -15 ))
 Counter=14
 DisplayHeader="Time       Temp  CPU fake/real     Health state    Vcore"
 while true ; do
 	let Counter++
 	if [ ${Counter} -eq 15 ]; then
 		echo -e "${DisplayHeader}"
 		Counter=0
 	fi
 	Health=$(perl -e "printf \"%19b\n\", $(vcgencmd get_throttled | cut -f2 -d=)")
 	Temp=$(vcgencmd measure_temp | cut -f2 -d=)
 	RealClockspeed=$(vcgencmd measure_clock arm | awk -F"=" '{printf ("%0.0f",$2/1000000); }' )
 	SysFSClockspeed=$(awk '{printf ("%0.0f",$1/1000); }' </sys/devices/system/cpu/cpu0/cpufreq/scaling_cur_freq)
 	CoreVoltage=$(vcgencmd measure_volts | cut -f2 -d= | sed 's/000//')
 	echo -e "$(date "+%H:%M:%S"): ${Temp}$(printf "%5s" ${SysFSClockspeed})/$(printf "%4s" ${RealClockspeed}) MHz $(printf "%019d" ${Health}) ${CoreVoltage}"
 	sleep 5 
 done

[ThomasKaiser]

Me still searching for the right packages containing vcgencmd command to talk to the mythical firmware having control over the ARM cores.

Seems, all that's needed is to use -DARM64=ON at build time: https://github.com/sakaki-/rpi3-overlay/blob/master/media-libs/raspberrypi-userland/raspberrypi-userland-9999.ebuild#L31-L33

[ThomasKaiser]

Now realized that you already use this. Sorry for the noise...

[openoms]

Hi @ThomasKaiser we came across your script while dealing with undervoltage issues on the RaspiBlitz: raspiblitz/raspiblitz#474

Is that OK if I use some of it as a part of our project?
https://github.com/openoms/raspiblitz/blob/powertest/home.admin/XXpowertest.sh
Needed to uncomment the Health related line:
Health=$(perl -e "printf \"%19b\n\", $(vcgencmd get_throttled | cut -f2 -d=)")
because of the error:
printf: warning: 10000000000000000000: Numerical result out of range
Probably caused by the different platform: Raspbian currently (planning to move over to DietPi).

Thank you for your work 🙏

[ThomasKaiser]

@openoms sure can you use it. ThreadX (the mythical 'firmware' running on every RPi) got an update at least with the RPi 3B+ and now the returned value contains of 21 bits and needs to be interpreted slightly differently: https://forum.openmediavault.org/index.php/Thread/18991-New-approach-for-Raspberry-Pi-OMV-images/?postID=190184#post190184 -- I don't know whether this only applies to RPi 3B+/A+ or all models but this should be easy to detect if you have more than one RPi (I have none in the meantime any more).

planning to move over to DietPi

So you neither care for security nor performance. Good luck with this ;)

[openoms]

planning to move over to DietPi
So you neither care for security nor performance. Good luck with this ;)

Exploring DietPi because of the multi-platform support and the automated setup capabilities. We are building our own services and hardening on top and surely disabling something like automatic login with root. It is quite barebone to start with.
Do you have any specific concerns with the way the DietPi software is built?