Signer can't sign with Invalid signature for party error

[jpraynaud]

Issue

An SPO is receiving the following error message: Invalid signature for party:

{"msg":"An error occured, runtime state kept. message = 'Could not send single signature during 'registered → signed' phase (current beacon Beacon { network: \"mainnet\", epoch: Epoch(427), immutable_file_number: 4592 })'. nested error = remote server technical error: '{\"message\":\"core error: 'Invalid signature for party: 'pool12wpfng6cu7dz38yduaul3ngfm44xhv5xmech68m5fwe4wu77udd''\"}'","v":0,"name":"slog-rs","level":50,"time":"2023-07-31T16:15:43.681384017Z","hostname":"cardano1","pid":2965342}

Troubleshooting

• Checked if the signer has successfully signed at least one certificate in the same epoch (with this procedure)

Yes, on the Mithril stake distribution

• Asked for the log of the signer to the SPO and verified the value of the computed digest by their node vs computed by network and stored in the certificate (`protocol_message/message_parts/snapshot_digest)

Digest do not match, this means that the signer node does not compute the digest correctly

• Asked for the cache file for immutable digest computations to the SPO, and diffed it with the cache file from the aggregator with command diff immutables_digests_mainnet-aggregator.json immutables_digests_mainnet.json

A difference existed for one immutable file 02789.chunk:

8369c8369
<   "02789.chunk": "a0e4e1ba075e320e452d9e98ea3adff8307f21986b9410bf8f171ca8cafed32b",
---
>   "02789.chunk": "09f28c7ce235849abd44158636a206e3a41a284c384115c0627a06bf0bbedc4c",

• Asked to compute the hash of the apparently corrupted file with command sha256sum db/immutable/02789.chunk and verified that this value matches the value of the cache 09f28c7ce235849abd44158636a206e3a41a284c384115c0627a06bf0bbedc4c

This is the case, the immutable file is corrupted

• Asked him to retrieve same immutable file from a different Cardano node and make sure that the hash is a0e4e1ba075e320e452d9e98ea3adff8307f21986b9410bf8f171ca8cafed32b

The SPO had a valid immutable available on another node

• Asked the SPO to replace the corrupted file with the correct file and to remove cache entry with command mv immutables_digests_mainnet.json immutables_digests_mainnet.json.bak && cat immutables_digests_mainnet.json.bak | jq 'del(.["02789.chunk"])' > immutables_digests_mainnet.json

The SPO has completed the operation and needs to wait at least 2 snapshots

• Checked if the signer has successfully signed another certificate in the same epoch (with this procedure)

Yes the signer has signed this certificate after fixing the probelm: https://aggregator.release-mainnet.api.mithril.network/aggregator/certificate/3f36967f09b7ea42dc4fb71dfcb07f542423efed6a630049b0626825657c07c9