Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

NewEpochFailure panic #404

Closed
mmahut opened this issue Nov 12, 2020 · 1 comment
Closed

NewEpochFailure panic #404

mmahut opened this issue Nov 12, 2020 · 1 comment

Comments

@mmahut
Copy link

mmahut commented Nov 12, 2020 •
edited
Loading

Yesterday, we have seen a panic on one of our nodes.

cardano-db-sync release/6.0.x 6187081a7ea66954c86094578bd37e01bca8aaec
cardano-node 1.21.1 a819311473563cb2ab3cd91543cf0f63facdf43e

-- Logs begin at Tue 2020-11-10 12:41:31 UTC, end at Thu 2020-11-12 08:18:25 UTC. --
Nov 11 21:40:39 backend6 3wx77yqp4mndkxs6v5nwfw2l3245l0af-cardano-db-sync[23598]: [db-sync-node:Info:35] [2020-11-11 21:40:39.80 UTC] insertShelleyBlock: epoch 228, slot 13564548, block 4938486, hash 10f5a3e4f23bd85eed962ab0c36fd32c5708e7273683478dd16e89e919dfeeda
Nov 11 21:40:41 backend6 3wx77yqp4mndkxs6v5nwfw2l3245l0af-cardano-db-sync[23598]: [db-sync-node:Info:35] [2020-11-11 21:40:41.66 UTC] insertShelleyBlock: epoch 228, slot 13564550, block 4938487, hash 9600e42abd16df07047b5f0fb5207ea2a0a4a17c1d6cade0e96c2659cfa05594
Nov 11 21:41:15 backend6 3wx77yqp4mndkxs6v5nwfw2l3245l0af-cardano-db-sync[23598]: [db-sync-node:Info:35] [2020-11-11 21:41:15.21 UTC] insertShelleyBlock: epoch 228, slot 13564584, block 4938488, hash 5821e3fa5a23ce96a93784b20eb9430f39dea1c4247c8d0f368b0a77c670a341
Nov 11 21:41:22 backend6 3wx77yqp4mndkxs6v5nwfw2l3245l0af-cardano-db-sync[23598]: [db-sync-node:Info:35] [2020-11-11 21:41:22.05 UTC] insertShelleyBlock: epoch 228, slot 13564591, block 4938489, hash cf8abac299c830958267bd3e752d091b8132a3d10b4ab894f4010533a0b3a18c
Nov 11 21:41:30 backend6 3wx77yqp4mndkxs6v5nwfw2l3245l0af-cardano-db-sync[23598]: [db-sync-node:Info:35] [2020-11-11 21:41:30.22 UTC] insertShelleyBlock: epoch 228, slot 13564599, block 4938490, hash d2819b8f6f56b38445a429665c773bb5e006488169106a1c54a352a1abeea58f
Nov 11 21:42:04 backend6 3wx77yqp4mndkxs6v5nwfw2l3245l0af-cardano-db-sync[23598]: [db-sync-node:Info:35] [2020-11-11 21:42:04.16 UTC] insertShelleyBlock: epoch 228, slot 13564633, block 4938491, hash cf2242c3b8b4b1dad1572720aca87388df3a97630a4e7ad7f042d1c5c3731479
Nov 11 21:42:42 backend6 3wx77yqp4mndkxs6v5nwfw2l3245l0af-cardano-db-sync[23598]: [db-sync-node:Info:35] [2020-11-11 21:42:42.23 UTC] insertShelleyBlock: epoch 228, slot 13564671, block 4938492, hash 57f29e6e92af476a9d11aaacd3e991d976c07ce2337b209bf676485b23d9d266
Nov 11 21:42:46 backend6 3wx77yqp4mndkxs6v5nwfw2l3245l0af-cardano-db-sync[23598]: [db-sync-node:Info:35] [2020-11-11 21:42:46.19 UTC] insertShelleyBlock: epoch 228, slot 13564675, block 4938493, hash c7f029438746834b4ce88a251edfea03844e5c086f605f8811a5b128395ca3cd
Nov 11 21:43:27 backend6 3wx77yqp4mndkxs6v5nwfw2l3245l0af-cardano-db-sync[23598]: [db-sync-node:Info:35] [2020-11-11 21:43:27.32 UTC] insertShelleyBlock: epoch 228, slot 13564716, block 4938494, hash 193fb31cd8f78d1495f10c1ab47628646e83d27e1ab2d8e00cd8cf3c12a437e7
Nov 11 21:43:43 backend6 3wx77yqp4mndkxs6v5nwfw2l3245l0af-cardano-db-sync[23598]: [db-sync-node:Info:35] [2020-11-11 21:43:43.14 UTC] insertShelleyBlock: epoch 228, slot 13564732, block 4938495, hash 958fadf743b43e57a4840bf0d80bf2fb925ca5299ce3d84c8634163b8964d796
Nov 11 21:43:59 backend6 3wx77yqp4mndkxs6v5nwfw2l3245l0af-cardano-db-sync[23598]: [db-sync-node:Info:35] [2020-11-11 21:43:59.39 UTC] insertShelleyBlock: epoch 228, slot 13564748, block 4938496, hash e44f437c6e6ced56d46dec7942e57227a7b68236a0a9555a81563ea23bfeca10
Nov 11 21:44:09 backend6 3wx77yqp4mndkxs6v5nwfw2l3245l0af-cardano-db-sync[23598]: [db-sync-node:Info:35] [2020-11-11 21:44:09.10 UTC] insertShelleyBlock: epoch 228, slot 13564758, block 4938497, hash 7f9cf13ed2624874ffd38b913a0722e779a233462d139cd1926c42278100c6bd
Nov 11 21:44:48 backend6 3wx77yqp4mndkxs6v5nwfw2l3245l0af-cardano-db-sync[23598]: [db-sync-node:Info:35] [2020-11-11 21:44:48.14 UTC] insertShelleyBlock: epoch 228, slot 13564796, block 4938498, hash 5d8ea0d4cf2d4f46cc91aa48e83c029691f836d7200e11e26402f9a2bcb25987
Nov 11 21:45:13 backend6 3wx77yqp4mndkxs6v5nwfw2l3245l0af-cardano-db-sync[23598]: [db-sync-node:Error:35] [2020-11-11 21:45:13.62 UTC] runDBThread: Panic! applyHeaderTransition failed: [[NewEpochFailure (EpochFailure (NewPpFailure (UnexpectedDepositPot (Coin 831364000000) (Coin 831366000000))))]]
Nov 11 21:45:13 backend6 3wx77yqp4mndkxs6v5nwfw2l3245l0af-cardano-db-sync[23598]: CallStack (from HasCallStack):
Nov 11 21:45:13 backend6 3wx77yqp4mndkxs6v5nwfw2l3245l0af-cardano-db-sync[23598]:   error, called at src/Shelley/Spec/Ledger/API/Validation.hs:92:15 in shelley-spec-ledger-0.1.0.0-3QeazRqhkmeDSfJ73hDh1U:Shelley.Spec.Ledger.API.Validation
Nov 11 21:45:13 backend6 3wx77yqp4mndkxs6v5nwfw2l3245l0af-cardano-db-sync[23598]: [db-sync-node:Error:31] [2020-11-11 21:45:13.62 UTC] ChainSyncWithBlocksPtcl: Panic! applyHeaderTransition failed: [[NewEpochFailure (EpochFailure (NewPpFailure (UnexpectedDepositPot (Coin 831364000000) (Coin 831366000000))))]]
Nov 11 21:45:13 backend6 3wx77yqp4mndkxs6v5nwfw2l3245l0af-cardano-db-sync[23598]: CallStack (from HasCallStack):
Nov 11 21:45:13 backend6 3wx77yqp4mndkxs6v5nwfw2l3245l0af-cardano-db-sync[23598]:   error, called at src/Shelley/Spec/Ledger/API/Validation.hs:92:15 in shelley-spec-ledger-0.1.0.0-3QeazRqhkmeDSfJ73hDh1U:Shelley.Spec.Ledger.API.Validation
Nov 11 21:45:13 backend6 3wx77yqp4mndkxs6v5nwfw2l3245l0af-cardano-db-sync[23598]: [db-sync-node.Subscription:Error:26] [2020-11-11 21:45:13.62 UTC] [String "Application Exception: LocalAddress {getFilePath = \"/run/cardano-node/node.socket\"} Panic! applyHeaderTransition failed: [[NewEpochFailure (EpochFailure (NewPpFailure (UnexpectedDepositPot (Coin 831364000000) (Coin 831366000000))))]]\nCallStack (from HasCallStack):\n  error, called at src/Shelley/Spec/Ledger/API/Validation.hs:92:15 in shelley-spec-ledger-0.1.0.0-3QeazRqhkmeDSfJ73hDh1U:Shelley.Spec.Ledger.API.Validation",String "SubscriptionTrace"]
Nov 11 21:45:13 backend6 3wx77yqp4mndkxs6v5nwfw2l3245l0af-cardano-db-sync[23598]: [db-sync-node.ErrorPolicy:Error:4] [2020-11-11 21:45:13.63 UTC] [String "ErrorPolicyUnhandledApplicationException Panic! applyHeaderTransition failed: [[NewEpochFailure (EpochFailure (NewPpFailure (UnexpectedDepositPot (Coin 831364000000) (Coin 831366000000))))]]\nCallStack (from HasCallStack):\n  error, called at src/Shelley/Spec/Ledger/API/Validation.hs:92:15 in shelley-spec-ledger-0.1.0.0-3QeazRqhkmeDSfJ73hDh1U:Shelley.Spec.Ledger.API.Validation",String "ErrorPolicyTrace",String "LocalAddress {getFilePath = \"/run/cardano-node/node.socket\"}"]
Nov 11 21:45:14 backend6 3wx77yqp4mndkxs6v5nwfw2l3245l0af-cardano-db-sync[23598]: [db-sync-node.Handshake:Info:189900] [2020-11-11 21:45:14.79 UTC] [String "Send (ClientAgency TokPropose,MsgProposeVersions (fromList [(NodeToClientV_1,TInt 764824073),(NodeToClientV_2,TInt 764824073),(NodeToClientV_3,TInt 764824073)]))",String "LocalHandshakeTrace",String "ConnectionId {localAddress = LocalAddress {getFilePath = \"\"}, remoteAddress = LocalAddress {getFilePath = \"/run/cardano-node/node.socket\"}}"]
Nov 11 21:45:14 backend6 3wx77yqp4mndkxs6v5nwfw2l3245l0af-cardano-db-sync[23598]: [db-sync-node.Handshake:Info:189900] [2020-11-11 21:45:14.85 UTC] [String "Recv (ServerAgency TokConfirm,MsgAcceptVersion NodeToClientV_3 (TInt 764824073))",String "LocalHandshakeTrace",String "ConnectionId {localAddress = LocalAddress {getFilePath = \"\"}, remoteAddress = LocalAddress {getFilePath = \"/run/cardano-node/node.socket\"}}"]
Nov 11 21:45:14 backend6 3wx77yqp4mndkxs6v5nwfw2l3245l0af-cardano-db-sync[23598]: [db-sync-node:Info:189904] [2020-11-11 21:45:14.85 UTC] Starting chainSyncClient
Nov 11 21:45:14 backend6 3wx77yqp4mndkxs6v5nwfw2l3245l0af-cardano-db-sync[23598]: [db-sync-node:Info:189904] [2020-11-11 21:45:14.91 UTC] Cardano.Db tip is at slot 13564796, block 4938498
Nov 11 21:45:14 backend6 3wx77yqp4mndkxs6v5nwfw2l3245l0af-cardano-db-sync[23598]: [db-sync-node:Info:189909] [2020-11-11 21:45:14.91 UTC] Running DB thread
Nov 11 21:45:15 backend6 3wx77yqp4mndkxs6v5nwfw2l3245l0af-cardano-db-sync[23598]: [db-sync-node:Info:189909] [2020-11-11 21:45:15.13 UTC] Rolling back to slot 13564796, hash 5d8ea0d4cf2d4f46cc91aa48e83c029691f836d7200e11e26402f9a2bcb25987
Nov 11 21:45:15 backend6 3wx77yqp4mndkxs6v5nwfw2l3245l0af-cardano-db-sync[23598]: [db-sync-node:Info:189909] [2020-11-11 21:45:15.13 UTC] Deleting slots numbered: []
Nov 11 21:45:17 backend6 3wx77yqp4mndkxs6v5nwfw2l3245l0af-cardano-db-sync[23598]: [db-sync-node:Error:189909] [2020-11-11 21:45:17.70 UTC] runDBThread: Panic! applyHeaderTransition failed: [[NewEpochFailure (EpochFailure (NewPpFailure (UnexpectedDepositPot (Coin 831364000000) (Coin 831366000000))))]]
Nov 11 21:45:17 backend6 3wx77yqp4mndkxs6v5nwfw2l3245l0af-cardano-db-sync[23598]: CallStack (from HasCallStack):
Nov 11 21:45:17 backend6 3wx77yqp4mndkxs6v5nwfw2l3245l0af-cardano-db-sync[23598]:   error, called at src/Shelley/Spec/Ledger/API/Validation.hs:92:15 in shelley-spec-ledger-0.1.0.0-3QeazRqhkmeDSfJ73hDh1U:Shelley.Spec.Ledger.API.Validation
Nov 11 21:45:17 backend6 3wx77yqp4mndkxs6v5nwfw2l3245l0af-cardano-db-sync[23598]: [db-sync-node:Error:189904] [2020-11-11 21:45:17.70 UTC] ChainSyncWithBlocksPtcl: Panic! applyHeaderTransition failed: [[NewEpochFailure (EpochFailure (NewPpFailure (UnexpectedDepositPot (Coin 831364000000) (Coin 831366000000))))]]
Nov 11 21:45:17 backend6 3wx77yqp4mndkxs6v5nwfw2l3245l0af-cardano-db-sync[23598]: CallStack (from HasCallStack):
Nov 11 21:45:17 backend6 3wx77yqp4mndkxs6v5nwfw2l3245l0af-cardano-db-sync[23598]:   error, called at src/Shelley/Spec/Ledger/API/Validation.hs:92:15 in shelley-spec-ledger-0.1.0.0-3QeazRqhkmeDSfJ73hDh1U:Shelley.Spec.Ledger.API.Validation
Nov 11 21:45:17 backend6 3wx77yqp4mndkxs6v5nwfw2l3245l0af-cardano-db-sync[23598]: [db-sync-node.Subscription:Error:189900] [2020-11-11 21:45:17.70 UTC] [String "Application Exception: LocalAddress {getFilePath = \"/run/cardano-node/node.socket\"} Panic! applyHeaderTransition failed: [[NewEpochFailure (EpochFailure (NewPpFailure (UnexpectedDepositPot (Coin 831364000000) (Coin 831366000000))))]]\nCallStack (from HasCallStack):\n  error, called at src/Shelley/Spec/Ledger/API/Validation.hs:92:15 in shelley-spec-ledger-0.1.0.0-3QeazRqhkmeDSfJ73hDh1U:Shelley.Spec.Ledger.API.Validation",String "SubscriptionTrace"]
Nov 11 21:45:17 backend6 3wx77yqp4mndkxs6v5nwfw2l3245l0af-cardano-db-sync[23598]: [db-sync-node.ErrorPolicy:Error:4] [2020-11-11 21:45:17.70 UTC] [String "ErrorPolicyUnhandledApplicationException Panic! applyHeaderTransition failed: [[NewEpochFailure (EpochFailure (NewPpFailure (UnexpectedDepositPot (Coin 831364000000) (Coin 831366000000))))]]\nCallStack (from HasCallStack):\n  error, called at src/Shelley/Spec/Ledger/API/Validation.hs:92:15 in shelley-spec-ledger-0.1.0.0-3QeazRqhkmeDSfJ73hDh1U:Shelley.Spec.Ledger.API.Validation",String "ErrorPolicyTrace",String "LocalAddress {getFilePath = \"/run/cardano-node/node.socket\"}"]
Nov 11 21:45:24 backend6 3wx77yqp4mndkxs6v5nwfw2l3245l0af-cardano-db-sync[23598]: [db-sync-node.Handshake:Info:189929] [2020-11-11 21:45:24.80 UTC] [String "Send (ClientAgency TokPropose,MsgProposeVersions (fromList [(NodeToClientV_1,TInt 764824073),(NodeToClientV_2,TInt 764824073),(NodeToClientV_3,TInt 764824073)]))",String "LocalHandshakeTrace",String "ConnectionId {localAddress = LocalAddress {getFilePath = \"\"}, remoteAddress = LocalAddress {getFilePath = \"/run/cardano-node/node.socket\"}}"]
Nov 11 21:45:24 backend6 3wx77yqp4mndkxs6v5nwfw2l3245l0af-cardano-db-sync[23598]: [db-sync-node.Handshake:Info:189929] [2020-11-11 21:45:24.80 UTC] [String "Recv (ServerAgency TokConfirm,MsgAcceptVersion NodeToClientV_3 (TInt 764824073))",String "LocalHandshakeTrace",String "ConnectionId {localAddress = LocalAddress {getFilePath = \"\"}, remoteAddress = LocalAddress {getFilePath = \"/run/cardano-node/node.socket\"}}"]
Nov 11 21:45:24 backend6 3wx77yqp4mndkxs6v5nwfw2l3245l0af-cardano-db-sync[23598]: [db-sync-node:Info:189933] [2020-11-11 21:45:24.80 UTC] Starting chainSyncClient
Nov 11 21:45:24 backend6 3wx77yqp4mndkxs6v5nwfw2l3245l0af-cardano-db-sync[23598]: [db-sync-node:Info:189933] [2020-11-11 21:45:24.81 UTC] Cardano.Db tip is at slot 13564796, block 4938498
Nov 11 21:45:24 backend6 3wx77yqp4mndkxs6v5nwfw2l3245l0af-cardano-db-sync[23598]: [db-sync-node:Info:189938] [2020-11-11 21:45:24.81 UTC] Running DB thread

This did affect only one node of the cluster.
@mmahut
Copy link
Author

mmahut commented Nov 12, 2020

Duplicate of #398, need morning coffee.

@mmahut mmahut closed this as completed Nov 12, 2020
@erikd erikd mentioned this issue Nov 12, 2020
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@mmahut