Install rngd(1) on all Jenkins controllers

[basil]

Service(s)

cert.ci.jenkins.io, ci.jenkins.io, infra.ci.jenkins.io, release.ci.jenkins.io, trusted.ci.jenkins.io, weekly.ci.jenkins.io

Summary

From /opt/java/openjdk/conf/security/java.security in the Docker container on ci.jenkins.io:

securerandom.source=file:/dev/random

This indicates Java is using /dev/random for random number generation. From outside the container:

basil@ci:~$ time head -n 1 /dev/random >/dev/null

real    0m37.251s
user    0m0.004s
sys     0m0.000s
basil@ci:~$

It took 37 seconds to produce random numbers, which is a long time. This indicates a failure to feed random data from hardware devices to the kernel entropy pool. This has downstream consequences when Java code creates an instance of SecureRandom, which then attempts to read from /dev/random. These reads can be slow when the kernel entropy pool is not populated.

To resolve the issue, simply install the rng-tools package on all Jenkins controllers and ensure that the rng.service systemd unit is started. As described in rng-tools, rngd(1) feeds random data from hardware devices to the kernel entropy pool. After this, time head -n 1 /dev/random >/dev/null should be very fast (and the same for SecureRandom invocations from Java).

Reproduction steps

Run time head -n 1 /dev/random >/dev/null. If it takes longer than a few milliseconds, the system is miconfigured.

[dduportal]

Let's start by ensuring that this package is installed on all the VM we manage (jenkins-infra/jenkins-infra). @smerle33 is handling it this first step.

[dduportal]

Thanks to @smerle33 work in #2167 (+ a mandatory reboot of the VMs to ensure that the Docker Engine propagate the mount into containers), it seems that all the rngd issues are of for the ci.jenkins.io, cert.ci and trusted.ci controllers:

• ci.jenkins.io:

# date
Mon May 16 12:07:50 UTC 2022
# time head -n 1 /dev/random >/dev/null

real    0m0.003s
user    0m0.002s
sys     0m0.000s
# docker exec -ti jenkins bash
  $ time head -n 1 /dev/random >/dev/null

real    0m0.001s
user    0m0.001s
sys     0m0.000s

• trusted.ci.jenkins.io:

# date 
Mon May 16 12:08:46 UTC 2022
# time head -n 1 /dev/random >/dev/null

real    0m0.001s
user    0m0.000s
sys     0m0.001s
# docker exec -ti jenkins bash
  $ time head -n 1 /dev/random >/dev/null

real    0m0.002s
user    0m0.001s
sys     0m0.000s

• cert.ci.jenkins.io:

# date
Mon May 16 12:11:34 UTC 2022
# time head -n 1 /dev/random >/dev/null

real    0m0.002s
user    0m0.000s
sys     0m0.002s
# docker exec -ti jenkins bash
  $ time head -n 1 /dev/random >/dev/null

real    0m0.007s
user    0m0.000s
sys     0m0.005s

[dduportal]

Regarding the Kubernetes-hosted controllers (where we do not have access to the underlying machines):

• infra.ci.jenkins.io

kubectl -n jenkins-infra exec jenkins-infra-0 -c jenkins -ti -- bash -c "time head -n 1 /dev/random >/dev/null"

real    0m0.001s
user    0m0.001s
sys     0m0.000s

• release.ci.jenkins.io:

kubectl -n release exec default-release-jenkins-0 -c jenkins -ti -- bash -c "time head -n 1 /dev/random >/dev/null"

real    0m0.014s
user    0m0.000s
sys     0m0.003s

• weekly.ci.jenkins.io:

kubectl -n jenkins-weekly exec jenkins-weekly-0 -c jenkins -ti -- bash -c "time head -n 1 /dev/random >/dev/null"

real    0m0.001s
user    0m0.001s
sys     0m0.000s

[dduportal]

Clogin the issue as it seems to be resolved. If I made a mistake by closing it, feel free to reopen it!

Many thanks for reporting/