Crashing hydra-node

[ch1bo]

Context & versions

λ bin/hydra-node --version
0.14.0-559eb142ed29b4a8093f731fc8fedc5d91e94a94

Steps to reproduce

1. Did the tutorial
2. On the last step I was trying to { "tag": "Fanout" } from both nodes

Actual behavior

Expectedly both nodes would try to submit a fanoutTx and only one succeeds. However, bob's node (in my case) received a submission error and crashed with:

hydra-node: DecoderFailure (LocalTxSubmission (GenTx (HardForkBlock (': * ByronBlock (': * (ShelleyBlock (TPraos StandardCrypto) (ShelleyEra StandardCrypto)) (': * (ShelleyBlock (TPraos StandardCrypto) (AllegraEra StandardCrypto)) (': * (ShelleyBlock (TPraos StandardCrypto) (MaryEra StandardCrypto)) (': * (ShelleyBlock (TPraos StandardCrypto) (AlonzoEra StandardCrypto)) (': * (ShelleyBlock (Praos StandardCrypto) (BabbageEra StandardCrypto)) (': * (ShelleyBlock (Praos StandardCrypto) (ConwayEra StandardCrypto)) ('[] *)))))))))) (HardForkApplyTxErr (': * ByronBlock (': * (ShelleyBlock (TPraos StandardCrypto) (ShelleyEra StandardCrypto)) (': * (ShelleyBlock (TPraos StandardCrypto) (AllegraEra StandardCrypto)) (': * (ShelleyBlock (TPraos StandardCrypto) (MaryEra StandardCrypto)) (': * (ShelleyBlock (TPraos StandardCrypto) (AlonzoEra StandardCrypto)) (': * (ShelleyBlock (Praos StandardCrypto) (BabbageEra StandardCrypto)) (': * (ShelleyBlock (Praos StandardCrypto) (ConwayEra StandardCrypto)) ('[] *)))))))))) ServerAgency TokBusy) (DeserialiseFailure 124 "expected word")

It might have been this block: https://preprod.cexplorer.io/block/15835403dc12390589a8260c391fec4ebbe147f83c947bb30389ca5d6a7fca1f

Expected behavior

The node should not crash and even ignore the false-positive (as the fanout transition worked using a different nodes' transaction)

[ch1bo]

Could reproduce this by doing a fanout of multiple nodes in one of our e2e test

modified   hydra-cluster/test/Test/EndToEndSpec.hs
@@ -225,8 +225,11 @@ spec = around (showLogsOnFailure "EndToEndSpec") $
                 waitFor hydraTracer (remainingTime + 3) [n1] $
                   output "ReadyToFanout" ["headId" .= headId]
 
-                send n1 $ input "Fanout" []
-                waitFor hydraTracer 3 [n1] $
+                -- TODO: revert or explain
+                race_
+                  (send n1 $ input "Fanout" [])
+                  (send n2 $ input "Fanout" [])
+                waitFor hydraTracer 3 [n1, n2] $

However, the error seems to be coming from the decoder used by the ApplyTxErr type of the LocalTxSubmission protocol. Tracing in a patched cardano-api or ouroboros-consensus was not helpful. Also tracing the DecCBOR instances in cardano-ledger was not very conclusive (it also had the node not crash anymore?).

The cardano-node log was somewhat helpful as it pointed me to some orphan ToObject instances for the ledger errors likely to be submitted: BadInputsUTxO, ValueNotConservedUTxO, ExtraRedeemers

Those are quite expected as the same inputs would be used by alternatively constructed fanout transactions.

I think this just comes from the fact that the hydra-node uses cardano-ledger-alonzo-1.4.1.0 while the cardano-node is on cardano-ledger-alonzo-1.2.1.0 and something changed in the CBOR encoding of the involved PredicateFailures.

[abailly-iohk]

Could you reproduce using an older hydra-node version that's on the same cardano-ledger-alonzo version?

[ch1bo]

• Reproducible on branch with further updated ledger plutus: 1.9 -> 1.15.0.1, cardano-api: 8.20 -> 8.29.1 #1168.
• Reproducible on hydra 0.12.0 which uses cardano-ledger-alonzo-1.3.3.0 against cardano-node 8.1.2
• Older release of hydra 0.11.0 is not easily buildable due to upstream repo removing a commit we used there as source-repository-package.

Not sure where to go from here. I do think this has been around already for a while and we should focus on using the same ledger versions as the node when targeting cardano-node 8.6.0 in #1177 and see whether this is still a problem.

[abailly-iohk]

Not sure I was clear in my comment, I was wondering whether or not the issue crops up if one runs a hydra-node using same ledger as cardano-node, eg. if you've been able to confirm your hypothesis which totally makes sense and seems highly likely to be right :)

[abailly-iohk]

Looking at cardano-node@8.1.2 and cardano-api@8.2.0.0 constraints I don't see any upper bound on cardano-ledger-alonzo so I guess this would be unexpected?