On a CM3+, the regenerate_ssh_host_keys script in rasapberrypi-sys-mods takes a very long time to run on kernel v6.1

[JinShil]

Describe the bug

On a CM3+, the regenerate_ssh_host_keys script in rasapberrypi-sys-mods is taking a very long time to run after installing 6.1 via apt using our pi-gen-based OS creation script. The offending line appears to be dd if=/dev/hwrng of=/dev/urandom count=1 bs=4096 status=none

$ time sudo dd if=/dev/hwrng of=/dev/urandom count=1 bs=4096 status=none
real    3m28.951s
user    0m0.024s
sys     0m0.041s

The same command runs in under a second on a CM4 or CM4S.

Steps to reproduce the behaviour

Using a CM3+, install a 64-bit Bullseye OS with kernel v6.1 and run the regenerate_ssh_host_keys script.

Device (s)

Raspberry Pi CM3+, Raspberry Pi CM3+ Lite

System

OS was generated using a script based on pi-gen.

Kernel version 6.1.19-v8+

Logs

No response

Additional context

No response

[pelwell]

I can confirm the regression - a 3B+ shows the same problem (I've not checked other models yet).

[popcornmix]

Pi 4 seemed quick for me.

[popcornmix]

On Pi4:

sudo cat /dev/hwrng
+K���(��u<��pP�3�<0�\R�⾬}���F����

(spurts lots of random stuff).

On Pi3+ nothing comes out.

[pelwell]

It's interesting that there should be a difference (which I'll investigate), but on the surface the problem appears to come from this upstream commit: 96cb9d0, the payload of which is:

diff --git a/drivers/char/hw_random/bcm2835-rng.c b/drivers/char/hw_random/bcm2835-rng.c
index e7dd457e9b22b..e98fcac578d66 100644
--- a/drivers/char/hw_random/bcm2835-rng.c
+++ b/drivers/char/hw_random/bcm2835-rng.c
@@ -71,7 +71,7 @@ static int bcm2835_rng_read(struct hwrng *rng, void *buf, size_t max,
        while ((rng_readl(priv, RNG_STATUS) >> 24) == 0) {
                if (!wait)
                        return 0;
-               cpu_relax();
+               hwrng_msleep(rng, 1000);
        }
 
        num_words = rng_readl(priv, RNG_STATUS) >> 24;

i.e. wait for a whole second between reads. Dropping this to 1ms gives a big improvement:

$ time sudo dd if=/dev/hwrng of=/dev/urandom count=1 bs=10000 status=none
real    0m15.025s
user    0m0.011s
sys     0m0.041s

But the reversion to cpu_relax() is equally dramatic:

$ time sudo dd if=/dev/hwrng of=/dev/urandom count=1 bs=100000 status=none
real    0m0.855s
user    0m0.000s
sys     0m0.851s

(note this is for 0.855s for 100,000 bytes as opposed to 15.025s for 10,000 bytes).

I'm going to revert the patch for now, but we should investigate whether this is even needed - the purpose of the kthread the commit refers to seems to be to shovel data into the random number generator.

[pelwell]

A quick look at the process list shows that we're also running rngd, part of rng-tools:

root        57  0.0  0.0      0     0 ?        S    15:48   0:00 [hwrng]
root       928  0.2  0.0  27652    80 ?        SLsl 16:13   0:00 /usr/sbin/rngd -r /dev/hwrng

whose documentation says:

       rngd - Check and feed random data from an entropy source (e.g. hardware
       RNG device) to an entropy sink (e.g. kernel entropy pool)

@XECDesign Do we still need the explicit random number seeding in regenerate_ssh_host_keys?

[popcornmix]

i.e. wait for a whole second between reads

Surely that is a bug? I assuming author was thinking 1ms rather than 1s.

Commit message says:

Rather than busy looping, yield back to the scheduler and sleep for a
bit in the event that there's no data. This should hopefully prevent the
stalls that Mark reported:

I feel "sleep for a bit" in a kernel driver context doesn't equate to a second.

[pelwell]

Florian can't have been awake that day.

[pelwell]

It's reverted for now.

[lategoodbye]

Hi,
reverting the patch seems a short term solution to me. I think we should keep the cpu_relax(), but introduce another exit criteria for this busy loop like max cycles or a timeout. But we need to live with some kind of performance loss.

@ffainelli What's your opinion?

[XECDesign]

@XECDesign Do we still need the explicit random number seeding in regenerate_ssh_host_keys?

Yeah, it runs on the very first boot, before systemd runs or has a chance to start rngd.

[popcornmix]

I believe the RNG block has an interrupt. Looks like it's INTERRUPT_HW_OFFSET + 61.
Wouldn't waiting for the interrupt here be best?
Looks like you can use RNG_FF_THRESHOLD to indicate how many words in did you'd like to be available when interrupt occurs.

[pelwell]

That would be a very invasive change, with patches to the driver and DTS files. The iproc_rng200 driver that's used on 2711 (hence the absence of a problem on Pi 4) waits for up to 500us, with the actual length governed by the amount of data that's required:

			/* Can wait, give others chance to run */
			usleep_range(min(num_remaining * 10, 500U), 500);

[popcornmix]

@JinShil the latest rpi-update kernel contains the revert so should avoid your issue.

[JinShil]

Thank you, I'll test tomorrow. rpi-update will provide us with a temporary workaround, but our OS creation scripts are based on pi-gen, so can you estimate when this fix might be available through apt?

[popcornmix]

We don't have a scheduled date for the next apt update.
We're keeping an eye on issues reported due to the recent kernel bump,
and results of that may trigger an apt update.

[pelwell]

See #5396 for a better fix.

[pelwell]

To install a trial build with the more intelligent sleeping, run sudo rpi-update pulls/5396.

[JinShil]

To install a trial build with the more intelligent sleeping, run sudo rpi-update pulls/5396.

I tested that today, and it seems to work fine.

[foosel]

Just to add another data point, also seeing this on an RPi3, booting the current RPi Lite image with the kernel and bootloader packages updated to

Get:1 http://archive.raspberrypi.org/debian bullseye/main armhf raspberrypi-kernel armhf 1:1.20230317-1 [102 MB]
Get:2 http://archive.raspberrypi.org/debian bullseye/main armhf raspberrypi-bootloader armhf 1:1.20230317-1 [4542 kB]

A timeline for the fixed version on apt here would indeed be nice, this is currently a blocker for providing up-to-date OctoPi images for OctoPrint.