Sporadic excessive CPU usage on RPi4B

[graysky2]

I am experiencing times of rngd saturating all 4 cores of my RPi4B and am wondering what is the cause. I see nothing in the logs nor in the journal. It happened when the headless box was just sitting idle just now. Other times, it can go for days without issue.

Distro: Arch ARM aarch64
rng-tools: 6.13-1

Note that the failure around opensc-pkcs11.so is not to blame. If I install opensc the problem will still occur.

% systemctl status rngd
● rngd.service - Hardware RNG Entropy Gatherer Daemon
     Loaded: loaded (/usr/lib/systemd/system/rngd.service; enabled; vendor preset: disabled)
     Active: active (running) since Wed 2021-05-19 10:12:09 EDT; 1 months 10 days ago
   Main PID: 273 (rngd)
      Tasks: 5 (limit: 8757)
        CPU: 1min 4.169s
     CGroup: /system.slice/rngd.service
             └─273 /usr/bin/rngd -f

Jun 29 11:47:39 workbench rngd[273]: Initializing available sources
Jun 29 11:47:39 workbench rngd[273]: [hwrng ]: Initialized
Jun 29 11:47:39 workbench rngd[273]: [rndr  ]: No HW SUPPORT
Jun 29 11:47:39 workbench rngd[273]: [rndr  ]: Initialization Failed
Jun 29 11:47:39 workbench rngd[273]: [jitter]: Initializing AES buffer
Jun 29 11:47:45 workbench rngd[273]: [jitter]: Enabling JITTER rng support
Jun 29 11:47:45 workbench rngd[273]: [jitter]: Initialized
Jun 29 11:47:45 workbench rngd[273]: [pkcs11]: PKCS11 Engine /usr/lib64/opensc-pkcs11.so Error: No such file or directory
Jun 29 11:47:45 workbench rngd[273]: [pkcs11]: Initialization Failed
Jun 29 11:47:45 workbench rngd[273]: [rtlsdr]: Initialization Failed

[nhorman]

what version of the jitterentropy library was this rng-tools package built with? If the jitter library has external thread support but not the ability to register external handlers, this will be the result. Fix is to update to the head of the jitterentropy-library tree

[graysky2]

3.0.2 from https://github.com/smuellerDD/jitterentropy-library/tags

It was built like this:

LDFLAGS=-lpthread make jitterentropy

Not sure if that gets at the external thread support/no ability to register external handlers?

[nhorman]

grrr, something is going on here. That should be recent enough to handle all the thread creation/management work, but it appears that @smuellerDD may have forced pushed something to the master branch, as a symbol rng-tool was using is now missing

I was going to suggest that you build the latest rng-tools and latest jitterentropy together, but its not going to work, I need to co-ordinate with @smuellerDD

[nhorman]

oh wait, never mind, I see whats happened. The rng-code that makes use of the exported soft timer thread interface is dependent on code in the jitterentropy external_threading branch. @smuellerDD hasn't merged it yet, so the software timer on arm systems like yours still suffers from issue #117 . Until @smuellerDD merges that code and its backported into arch, you probably want to either (a) run rngd with the -x jitter option (which disables the jitter entropy source), and use some other source (rtlsdr or a hwrng perhaps), or reduce the jitterentropy library version on your system to version 2.2.0 or earlier (in which the software timer doesn't exist, though that will probably just cause jittereentropy to not initialize due to an overly coarse hardware timer)

[graysky2]

Thanks for digging into this. I added the following to /etc/conf.d/rngd and the daemon is running fine with it. That is passed along to rngd.service via ExecStart=/usr/bin/rngd -f $RNGD_OPTS

RNGD_OPTS="-x jitter -r /dev/hwrng"

I wanted to verify stability. Can you describe how I can intentionally trigger this bug?

[nhorman]

on the system you are running on, it should be sufficient to simply run rngd -n jitter (to ensure that jitter is enabled). As long as you are running with the versions of the jitter library you have now, you should hit the problem

[graysky2]

You are right... running rngd -n jitter does trigger it. Guess I will just run with the options you recommended above until the jitterentropy stuff is fixed.

The rng-code that makes use of the exported soft timer thread interface is dependent on code in the jitterentropy external_threading branch. @smuellerDD hasn't merged it yet

Can you point me to the PR you referenced for tracking purposes?

[nhorman]

smuellerDD/jitterentropy-library#37

[smuellerDD]

Am Dienstag, 29. Juni 2021, 20:52:56 CEST schrieb Neil Horman: Hi Neil,

oh wait, never mind, I see whats happened. The rng-code that makes use of the exported soft timer thread interface is dependent on code in the jitterentropy external_threading branch. @smuellerDD hasn't merged it yet,

The code has now been merged. Thanks for the patience as I now finished work on a different project.

so the software timer on arm systems like yours still suffers from issue #117 . Until @smuellerDD merges that code and its backported into arch, you probably want to either (a) run rngd with the -x jitter option (which disables the jitter entropy source), and use some other source (rtlsdr or a hwrng perhaps), or reduce the jitterentropy library version on your system to version 2.2.0 or earlier (in which the software timer doesn't exist, though that will probably just cause jittereentropy to not initialize due to an overly coarse hardware timer)

Ciao Stephan

[nhorman]

thank you @smuellerDD ! @graysky2 if you update to the latest rng-tools and jitterentropy library, things should be somewhat better for you (at least you wont have to explicitly disable jitterentropy)

[graysky2]

I built smuellerDD/jitterentropy-library@e20a40a and installed it. Then I rebuilt rng-tools v6.13 against it. Is it normal for rngd to max out all cores for 5-10 sec on startup?

systemctl status rngd
● rngd.service - Hardware RNG Entropy Gatherer Daemon
     Loaded: loaded (/usr/lib/systemd/system/rngd.service; enabled; vendor preset: disabled)
     Active: active (running) since Fri 2021-07-09 14:33:12 EDT; 53s ago
   Main PID: 5429 (rngd)
      Tasks: 5 (limit: 3936)
        CPU: 1min 22.780s
     CGroup: /system.slice/rngd.service
             └─5429 /usr/bin/rngd -f

Jul 09 14:33:12 submyth rngd[5429]: Initializing available sources
Jul 09 14:33:12 submyth rngd[5429]: [hwrng ]: Initialized
Jul 09 14:33:12 submyth rngd[5429]: [rndr  ]: No HW SUPPORT
Jul 09 14:33:12 submyth rngd[5429]: [rndr  ]: Initialization Failed
Jul 09 14:33:12 submyth rngd[5429]: [jitter]: Initializing AES buffer
Jul 09 14:33:19 submyth rngd[5429]: [jitter]: Enabling JITTER rng support
Jul 09 14:33:19 submyth rngd[5429]: [jitter]: Initialized
Jul 09 14:33:19 submyth rngd[5429]: [pkcs11]: PKCS11 Engine /usr/lib64/opensc-pkcs11.so Error: No such file or directory
Jul 09 14:33:19 submyth rngd[5429]: [pkcs11]: Initialization Failed
Jul 09 14:33:19 submyth rngd[5429]: [rtlsdr]: Initialization Failed

[nhorman]

Its not uncommon to see that happen. At startup the jitter threads are all spinning the cpus to fill up their entropy pools, and that can take a few seconds. Once that work is done through, especially if you're using AES conditioning, that should subside , and not occur again, unless you completely exhaust the entropy pool

[graysky2]

OK, then I believe this issue has been fixed by smuellerDD/jitterentropy-library@e20a40a