Add logging for NthNextHmy panic

[MaxMustermann2]

Helps confirm that #4340 is caused by time drift.

• The crash occurred due to an index out of bounds error on the list of leader public keys.
• The index in question was negative.
• The index represented the location of the next leader's index in the list.
• The index is calculated as (gap in view ids + current leader index) % number of internal keys. The gap acts as an offset parameter, and can be negative.
• However, if the gap becomes too negative (such that the numerator is less than 0), we will get a negative number. For Go, a % b is negative if a is negative..
• Most of the times, the gap is positive, except when the fallback mechanism for view id calculation is used.
• The fallback mechanism of the gap calculation is triggered when the timestamp of the current block is larger than that of the current validator. This means, that either the current node is behind in time, or the proposer of the current block is ahead. In our case it was the latter.

See private Discord thread here for logs.

Solution

On all nodes which could potentially have this issue, install and enable the systemd service to synchronize time with NTP servers.

apt install -y systemd-timesyncd
cat<<-EOF > /etc/systemd/timesyncd.conf.d/99-working.conf
[Time]
NTP=pool.ntp.org
EOF
systemctl unmask systemd-timesyncd.service
systemctl enable --now systemd-timesyncd.service
timedatectl timesync-status

[ONECasey]

Seems to be failing one of the CI tests.

[MaxMustermann2]

Seems to be failing one of the CI tests.

The test was logging too much. Logging was piped to /dev/null and the test passes.