Signer runtime is stuck for some SPO

[jpraynaud]

Why

Some SPO report that their signer node has its state machine runtime stuck for multiple days:

• https://discord.com/channels/826816523368005654/1014019542504185876/1163827469258850334
• https://discord.com/channels/826816523368005654/1014019542504185876/1161299662851031211
• https://discord.com/channels/826816523368005654/1014019542504185876/1164541903174389791
• https://discord.com/channels/826816523368005654/1014019542504185876/1178885769352458260

This could be due to KES keys rotation, but there is no obvious link 🤔. We need to investigate further the problem.
Some SPO state that this could happen after a restart of the Block Producer node.

The signer nodes seem to be healthy.

In general, a restart of the signer node fixes the problem.

What

Can we reproduce the problem with?

• KES keys rotation
• Block Producer restart

Can we find in the code if there is a problem with the tick of the runtime?

How

• Try to reproduce the problem locally
• Add a goodbye message when signer is terminated with no crash
• Analyze the code and find potential problems that could lead to this behavior
• Add timeout on HTTP client(s)

[dlachaume]

Below are some tests performed in attempt to reproduce the behavior described by the SPOs.

These tests were executed locally with 3 signer nodes on a devnet with mithril-end-to-end. Signers are registered and sign before executing each case.

Tests are checked if the signer is able to run and sign without issue.

Scenario	Test passed
Restart the signer node	✅
Restart the cardano node while the signer is still running	✅
Rotate the KES keys	✅

[jpraynaud]

We close issue until the problem occurs once again.

In the mean time, the PR #1362 added a goodbye message that will help use identify unexpected crashes if any.