Python interpreter start-up spikes CPU load triggering preempt

[ali1234]

When automatic.py first loads, the python start-up causes a CPU spike large enough to trigger the fan preempt mode. This causes the fan to spin for a few ms at start up - and if there is no other software running it will most likely never trigger again due to the 65 degree default threshold. This can lead to users thinking there is a problem if the CPU is hot enough for the fan to spin once, but then it never spins again.

I recommend suppressing preempt for a few seconds when starting up the daemon.

[Gadgetoid]

Thought we'd clinched this one with slightly rewritten preempt handling, but yes suppressing it for a brief period would be a good idea.

[jankais3r]

On a similar note, when I run top on my raspberry 4, python3 process responsible for automatic.py is constantly the process taking most CPU resources - and I am running bunch of other stuff on the Pi like homebridge, pihole and dnscrypt. The process is constantly utilizing between 3 and 4%, which in all honesty is ridiculous for a script that is supposed to check the CPU temperature from time to time and spin a fan if it goes over certain threshold. I am pretty sure that it is also responsible for most of the heat that then has to be cooled off by the fan.

[happapa]

The process is constantly utilizing between 3 and 4%

I have a hunch that it might be because of the button polling. Try running the service with --nobutton.

[jankais3r]

pi@raspberrypi:/opt $ sudo systemctl status pimoroni-fanshim
● pimoroni-fanshim.service - Fan Shim Service
   Loaded: loaded (/etc/systemd/system/pimoroni-fanshim.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2019-07-26 15:56:27 CEST; 14s ago
 Main PID: 18562 (python3)
    Tasks: 2 (limit: 3873)
   Memory: 4.2M
   CGroup: /system.slice/pimoroni-fanshim.service
           └─18562 python3 /opt/automatic.py --on-threshold 65 --off-threshold 55 --delay 2 --nobutton

Jul 26 15:56:27 raspberrypi systemd[1]: Started Fan Shim Service.

Unfortunately that did not help:

[flobernd]

It's exactly the same for me. Not even the java process is any close.

The hardware might be good, but so far im disappointed by the software quality.

[jankais3r]

I wouldn't be so harsh - there is probably one line of code where the load implications were not taken into consideration and once we find it, it will be all good. Except the high CPU load, the package works great.

[flobernd]

Not meant to be harsh. The software control does not work for me at all, so maybe I'm a bit frustrated.

[ali1234]

The really proper way to do this is using the gpio-fan module, which is specifically designed to do the job and runs entirely in kernel. Somebody will have to go and figure out how though.

[ali1234]

https://github.com/torvalds/linux/blob/master/Documentation/devicetree/bindings/hwmon/gpio-fan.txt

[daviehh]

A quick/very experimental C++ code using the latest version of WiringPi, this works fine for me so far on raspberry pi 4 (can be improved: right now the on/off temp and temperature-checking frequency are hard-coded; no LED control; and currently it does not handle exiting behavior, etc):

#include <wiringPi.h>
#include <iostream>
#include <fstream>
#include <unistd.h>
#include <string>
using namespace std;


int main (void)
{
  wiringPiSetupGpio();
  pinMode(18, OUTPUT);
  int microseconds = 10*1000*1000;

  fstream tmp_file;
  float tmp;

  while(1){
    tmp_file.open("/sys/class/thermal/thermal_zone0/temp", ios_base::in);
    tmp_file >> tmp;
    tmp_file.close();
    tmp = tmp/1000;
    cout<<"Temp: "<<tmp<<endl;
    
    if(tmp > 55){
        digitalWrite(18, 1);
    }
    
    if(tmp < 45){
        digitalWrite(18, 0);
    }

    string fanstate = digitalRead(18) == 0 ? "off" : "on";
    cout<<"fan state now: "<< fanstate <<endl;

    usleep(microseconds);
    }
  return 0 ;

}

compile with, e.g. g++ cpu_ctrl.cpp -o cpu_ctrl -O3 -lwiringPi

Alternatively, one may get temperature from the command line output from vcgencmd measure_temp; still checking which leads to lower spikes... Maybe someone more familiar with C++ can check?

[jankais3r]

A quick/very experimental C++ code using the latest version of WiringPi

This looks awesome. Would you mind creating a new repo for this, so we can improve further and implement new features?

[daviehh]

@jankais3r I only have very basic knowledge of C++ and GPIO programming, and the code is written within an hour or so. It can potentially have some bugs and make the fan run all the time, or using excessive cpu/memory/disk, which may or may not cause hardware damage, so the code is only for testing/playing now, and I wouldn't run it unattended and I do not think it's ready for serious use or valuable enough to put in a repo ... anyway, it should be easy to add in customization using a json file with the json.hpp file using this library: nlohmann/json, something like:

#include <wiringPi.h>
#include <iostream>
#include <fstream>
#include <unistd.h>
#include <string>
#include "json.hpp"


using json = nlohmann::json;
using namespace std;

map<string, int>  get_fs_conf()
{
    map<string, int> fs_conf { 
        {"on-threshold", 60}, 
        {"off-threshold", 50},
        {"on-budget",3},
        {"delay", 10}
    };

    try
    {
        ifstream fs_cfg_file("fanshim.json");
        json fs_cfg_custom;
        fs_cfg_file >> fs_cfg_custom;

        for (auto& el : fs_cfg_custom.items()) {
            fs_conf[el.key()] = el.value();
        }
    }
    catch (...)
    {
        cout<<"error parsing config file"<<endl;
    }

    for (map<string,int>::iterator it=fs_conf.begin(); it!=fs_conf.end(); ++it)
      cout << it->first << " => " << it->second << endl;

  return fs_conf;
}


int main (void)
{

  const int fanshim_pin = 18;

  wiringPiSetupGpio();
  pinMode(fanshim_pin, OUTPUT);

  map<string, int> fs_conf = get_fs_conf();

  const int sleep_msec = fs_conf["delay"]*1000*1000;
  const int on_threshold = fs_conf["on-threshold"];
  const int off_threshold = fs_conf["off-threshold"];

  const int on_budget = fs_conf["on-budget"];

  int read_fs_pin = 0;
  int on_count = 0;

  const string node_hdr = "# HELP cpu_fanshim text file output: fan state.\n# TYPE cpu_fanshim gauge\ncpu_fanshim ";
  const string node_hdr_t = "# HELP cpu_temp_fanshim text file output: temp.\n# TYPE cpu_temp_fanshim gauge\ncpu_temp_fanshim ";
  string nodex_out = "";

  fstream tmp_file;
  float tmp = 0;
  string fanstate = "-";
  tmp_file.open("/sys/class/thermal/thermal_zone0/temp", ios_base::in);

  while(1){
    tmp_file >> tmp;
    tmp_file.seekg(0, tmp_file.beg);
    tmp = tmp/1000;
    cout<<"Temp: "<<tmp<<endl;

    read_fs_pin = digitalRead(fanshim_pin);
    
    if(tmp >= on_threshold){

        on_count += 1;
        
        if(read_fs_pin == 0 && on_count == on_budget){
            digitalWrite(fanshim_pin, 1);
            on_count = 0;
        }
    }
    else {
        on_count = 0;
        if(tmp < off_threshold && read_fs_pin == 1){
            digitalWrite(fanshim_pin, 0);
        }
    }


    read_fs_pin = digitalRead(fanshim_pin);
    fanstate = read_fs_pin == 0 ? "off" : "on";
    cout<<"fan state now: "<< fanstate <<endl;

    ofstream nodex_fs;
    nodex_fs.open("/usr/local/etc/node_exp_txt/cpu_fan.prom");
    nodex_out = node_hdr + to_string(read_fs_pin) + "\n";
    nodex_out += node_hdr_t + to_string(int(tmp)) + "\n";
    nodex_fs<<nodex_out;
    nodex_fs.close();

    usleep(sleep_msec);
}

return 0 ;

}

where the json file can be

{
    "on-threshold": 60,
    "off-threshold": 50,
    "delay": 10
}

Feel free to tinker with it :-)

Also @Gadgetoid , are you familiar with c or care to look at it? Thanks!

grafana + prometheus +node_exporter monitoring: blue=temperature; yellow = fan on/off:

[flobernd]

I did not look into the python sources yet. Do you think it's possible to control the LED as well? Or more specific: How are the color values transfered from GPIO to the fanshim? Do they use a specific protocol?

[daviehh]

@flobernd maybe... the protocol is in the relevant code here. I glanced through and looks like it's using two pins to control the LED: one pin (data) to write the value of RGB/brightness bit-by-bit, where each bit is signaled by a jump in the other pin (clock).

The product page of fanshim also lists the LED as APA 102, same as blinkt which has a c library here but the pins used/number of LEDs looks different and thus needs to be modified for use here. I haven't tested though...

[flobernd]

@daviehh That looks promising to me. I might try it out if I have some spare time in the next few days.

[flobernd]

I played a little and LED control seems to work fine. You have to implement some kind of software SPI:

const int PIN_LED_CLCK = 14;
const int PIN_LED_MOSI = 15;
const int CLCK_STRETCH =  5;

...

wiringPiSetupGpio()
pinMode(PIN_LED_CLCK, OUTPUT);
pinMode(PIN_LED_MOSI, OUTPUT);

inline static void write_byte(uint8_t byte)
{
    for (int n = 0; n < 8; n++)
    {
        digitalWrite(PIN_LED_MOSI, (byte & (1 << (7 - n))) > 0);
        digitalWrite(PIN_LED_CLCK, HIGH);
        usleep(CLCK_STRETCH);
        digitalWrite(PIN_LED_CLCK, LOW);
        usleep(CLCK_STRETCH);
    }
}

Setting the LED works this way:

// A start frame of 32 zero bits (<0x00> <0x00> <0x00> <0x00>)
digitalWrite(PIN_LED_MOSI, 0);
for (int i = 0; i < 32; ++i)
{
    digitalWrite(PIN_LED_CLCK, HIGH);
    usleep(CLCK_STRETCH);
    digitalWrite(PIN_LED_CLCK, LOW);
    usleep(CLCK_STRETCH);
}

// A 32 bit LED frame for each LED in the string (<0xE0+brightness> <blue> <green> <red>)
write_byte(0b11100000 | 31); // in range of 0..15 for the fanshim
write_byte(255); // b
write_byte(  0); // g
write_byte(255); // r

// An end frame consisting of at least (n/2) bits of 1, where n is the number of LEDs in the string
digitalWrite(PIN_LED_MOSI, 1);
for (int i = 0; i < 1; ++i)
{
    digitalWrite(PIN_LED_CLCK, HIGH);
    usleep(CLCK_STRETCH);
    digitalWrite(PIN_LED_CLCK, LOW);
    usleep(CLCK_STRETCH);
}

[daviehh]

@flobernd wow, that's quick :-) Thanks! I'll try playing with it when I have some free time.

[daviehh]

using @flobernd 's code, looks like one may be able to convert temperature to RGB led, like this (example: get temperature from keyboard input for testing):

#include <wiringPi.h>
#include <iostream>
#include <fstream>
#include <unistd.h>
#include <string>
#include <deque>


#include <algorithm>
#include <cmath>
#include <vector>


using namespace std;

const int PIN_LED_CLCK = 14;
const int PIN_LED_MOSI = 15;
const int CLCK_STRETCH =  5;


inline static void write_byte(uint8_t byte)
{
    for (int n = 0; n < 8; n++)
    {
        digitalWrite(PIN_LED_MOSI, (byte & (1 << (7 - n))) > 0);
        digitalWrite(PIN_LED_CLCK, HIGH);
        usleep(CLCK_STRETCH);
        digitalWrite(PIN_LED_CLCK, LOW);
        usleep(CLCK_STRETCH);
    }
}

double tmp2hue(double tmp, double hi, double lo)
{
    double hue = 0;
    if (tmp < lo)
        return 1.0/3.0;
    else if (tmp > hi)
        return 0.0;
    else
        return (hi-tmp)/(hi-lo)/3.0;
}

/////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
////https://en.wikipedia.org/wiki/HSL_and_HSV#HSV_to_RGB
/////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

double hsv_k(int n, double hue)
{
    return fmod(n + hue/60.0, 6);
}

double hsv_f(int n, double hue,double s, double v)
{
    double k = hsv_k(n,hue);
    return v - v * s * max( { min( {k, 4-k,1.0} ), 0.0 } );
}

vector<int> hsv2rgb(double h, double s, double v)
{
    double hue = h * 360;
    int r = int(hsv_f(5,hue, s, v)*255);
    int g = int(hsv_f(3,hue, s, v)*255);
    int b = int(hsv_f(1,hue, s, v)*255);
    vector<int> rgb;
    rgb.push_back(r);
    rgb.push_back(g);
    rgb.push_back(b);
    return rgb;
}




int main (void)
{
    
    
    wiringPiSetupGpio();
    pinMode(PIN_LED_CLCK, OUTPUT);
    pinMode(PIN_LED_MOSI, OUTPUT);
    
    int r=0,g=0,b=0,br=10;
    double s,v;
    s=1;
    v=br/31.0;
    //// hsv: hue from temperature; s set to 1, 
    //v set to brightness like the official code 
    //https://github.com/pimoroni/fanshim-python/blob/5841386d252a80eeac4155e596d75ef01f86b1cf/examples/automatic.py#L44

    
    while(1)
    {
    cout<<"tmp: ";
    int tmp;
    cin>>tmp;

    vector<int> rgb=hsv2rgb(tmp2hue(tmp,60,40),s,v);
    r=rgb.at(0);
    g=rgb.at(1);
    b=rgb.at(2);


    digitalWrite(PIN_LED_MOSI, 0);
    for (int i = 0; i < 32; ++i)
    {
        digitalWrite(PIN_LED_CLCK, HIGH);
        usleep(CLCK_STRETCH);
        digitalWrite(PIN_LED_CLCK, LOW);
        usleep(CLCK_STRETCH);
    }
    
    // A 32 bit LED frame for each LED in the string (<0xE0+brightness> <blue> <green> <red>)
    write_byte(0b11100000 | br); // in range of 0..15 for the fanshim
    write_byte(b); // b
    write_byte(g); // g
    write_byte(r); // r
    
    // An end frame consisting of at least (n/2) bits of 1, where n is the number of LEDs in the string
    digitalWrite(PIN_LED_MOSI, 1);
    for (int i = 0; i < 1; ++i)
    {
        digitalWrite(PIN_LED_CLCK, HIGH);
        usleep(CLCK_STRETCH);
        digitalWrite(PIN_LED_CLCK, LOW);
        usleep(CLCK_STRETCH);
    }
    
    }
    
    return 0 ;
    
}

[daviehh]

The C++ code has been updated to experimentally support LEDs.
https://github.com/daviehh/fanshim-cpp

[flobernd]

Haha you were faster than me, but I'll link my own repository anyways:
https://github.com/flobernd/raspi-fanshim

Splitted it in two libraries and added CMake support. It's still work in progress tho.

[Gadgetoid]

@daviehh @flobernd it's great to see folks take matters into their own hands and put together software that works for them. I've been trying to keep on top of Fan SHIM's automatic.py but I'm juggling many, many devices so it's not easy.

My input may not be particularly useful here since I'm no C/C++ guru, but I do know that WiringPi is now deprecated and that libgpiod (as used by @daviehh) is the way forward. libgpiod is "slow" compared to the memory-mapped approach that, for example, lbcm2835 still uses but it's plenty fast enough for Fan SHIM.