Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rollback PoolOfflineData could not be written to db. #823

Closed
MarcelKlammer opened this issue Sep 15, 2021 · 8 comments
Closed

Rollback PoolOfflineData could not be written to db. #823

MarcelKlammer opened this issue Sep 15, 2021 · 8 comments
Labels
bug Something isn't working

Comments

@MarcelKlammer
Copy link

Linux/Ubuntu: 20.04
cardano-db-sync: v11.0.0 (see cardano-graphql)
Postgres: 11.5, docker image (see cardano-graphql)

We are running two db-sync servers, both had rollback of 10360 blocks at the same time:

Server 1

cardano-db-sync-extended_1  | [db-sync-node:Info:82] [2021-09-15 08:21:55.05 UTC] Offline pool metadata fetch: 43 results, 57 fetch errors
cardano-db-sync-extended_1  | [db-sync-node:Info:82] [2021-09-15 08:22:11.04 UTC] insertShelleyBlock(Alonzo): epoch 290, slot 40127838, block 6246411, hash 4c43818b951423dfd572b8b54e18f6982ea2ec2c923bac83fa548febb8193584
cardano-db-sync-extended_1  | [db-sync-node:Info:82] [2021-09-15 08:22:30.64 UTC] insertShelleyBlock(Alonzo): epoch 290, slot 40127857, block 6246412, hash 19e5d7c3d687090d98b50b2634b0c07751623abcd1f095be6c424baece094429
cardano-db-sync-extended_1  | [db-sync-node:Info:82] [2021-09-15 08:22:33.71 UTC] insertShelleyBlock(Alonzo): epoch 290, slot 40127861, block 6246413, hash 980a48ec01ea6122ed9c04a9c2f48054ceefae394e1315d85858d389bf045e16
cardano-db-sync-extended_1  | [db-sync-node:Info:82] [2021-09-15 08:23:38.65 UTC] insertShelleyBlock(Alonzo): epoch 290, slot 40127925, block 6246414, hash 378f58a6a65936f4fb049875b553e31585cbcc491d40181ac371ba9e88a119dd
cardano-db-sync-extended_1  | [db-sync-node:Info:82] [2021-09-15 08:23:57.81 UTC] insertShelleyBlock(Alonzo): epoch 290, slot 40127944, block 6246415, hash 86ebe9fd7835046e5447b60430d7ec870d318615ca808060f99852e451bea2f7
cardano-db-sync-extended_1  | [db-sync-node:Info:82] [2021-09-15 08:23:59.79 UTC] insertShelleyBlock(Alonzo): epoch 290, slot 40127947, block 6246416, hash b16e3bdaf6488a689fb3ddd372fd491b7cef744ad6f1da8e364a7b841cf74d58
cardano-db-sync-extended_1  | [db-sync-node:Info:82] [2021-09-15 08:24:04.48 UTC] insertShelleyBlock(Alonzo): epoch 290, slot 40127951, block 6246417, hash 2cd12a17ee9efeb042b44c6a271eab8a074dc9fa1dce19af888880d7f981303f
cardano-db-sync-extended_1  | [db-sync-node:Info:82] [2021-09-15 08:24:12.31 UTC] insertShelleyBlock(Alonzo): epoch 290, slot 40127959, block 6246418, hash 4d283e59f915e64721714f131f0b5c6bf8c2c4a7621db10ff9c5fd718d59aa12
cardano-db-sync-extended_1  | [db-sync-node:Info:82] [2021-09-15 08:24:28.83 UTC] insertShelleyBlock(Alonzo): epoch 290, slot 40127976, block 6246419, hash 6d0fe286fde5dc60253e2b0e1cbcffc0b22c71d8c765d5c2cc64eb874dbc4390
cardano-db-sync-extended_1  | [db-sync-node:Info:82] [2021-09-15 08:24:36.26 UTC] insertShelleyBlock(Alonzo): continuing epoch 290 (slot 211183/432000)
cardano-db-sync-extended_1  | [db-sync-node:Info:82] [2021-09-15 08:24:36.26 UTC] insertShelleyBlock(Alonzo): epoch 290, slot 40127983, block 6246420, hash d15131b76ea1e818e208e58cb4fa9d1b8c1d56d8642eb2552e1154cb3af29b97
cardano-db-sync-extended_1  | [db-sync-node:Info:82] [2021-09-15 08:24:36.26 UTC] Offline pool metadata fetch: 44 results, 56 fetch errors
cardano-db-sync-extended_1  | [db-sync-node:Error:82] [2021-09-15 08:24:36.27 UTC] runDBThread: DbInsertException "PoolOfflineData" (SqlError {sqlState = "23503", sqlExecStatus = FatalError, sqlErrorMsg = "insert or update on table \"pool_offline_data\" violates foreign key constraint \"pool_offline_data_pmr_id_fkey\"", sqlErrorDetail = "Key (pmr_id)=(23322) is not present in table \"pool_metadata_ref\".", sqlErrorHint = ""})
cardano-db-sync-extended_1  | [db-sync-node:Error:77] [2021-09-15 08:24:36.27 UTC] ChainSyncWithBlocksPtcl: DbInsertException "PoolOfflineData" (SqlError {sqlState = "23503", sqlExecStatus = FatalError, sqlErrorMsg = "insert or update on table \"pool_offline_data\" violates foreign key constraint \"pool_offline_data_pmr_id_fkey\"", sqlErrorDetail = "Key (pmr_id)=(23322) is not present in table \"pool_metadata_ref\".", sqlErrorHint = ""})
cardano-db-sync-extended_1  | [db-sync-node.Mux:Info:74] [2021-09-15 08:24:36.27 UTC] Bearer on ConnectionId {localAddress = LocalAddress "", remoteAddress = LocalAddress "/ipc/node.socket"} event: State: Dead
cardano-db-sync-extended_1  | [db-sync-node.Mux:Notice:74] [2021-09-15 08:24:36.27 UTC] Bearer on ConnectionId {localAddress = LocalAddress "", remoteAddress = LocalAddress "/ipc/node.socket"} event: Miniprotocol MiniProtocolNum 5 InitiatorDir terminated with exception DbInsertException "PoolOfflineData" (SqlError {sqlState = "23503", sqlExecStatus = FatalError, sqlErrorMsg = "insert or update on table \"pool_offline_data\" violates foreign key constraint \"pool_offline_data_pmr_id_fkey\"", sqlErrorDetail = "Key (pmr_id)=(23322) is not present in table \"pool_metadata_ref\".", sqlErrorHint = ""})
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Error:73] [2021-09-15 08:24:36.27 UTC] Identity Application Exception: LocalAddress "/node-ipc/node.socket" DbInsertException "PoolOfflineData" (SqlError {sqlState = "23503", sqlExecStatus = FatalError, sqlErrorMsg = "insert or update on table \"pool_offline_data\" violates foreign key constraint \"pool_offline_data_pmr_id_fkey\"", sqlErrorDetail = "Key (pmr_id)=(23322) is not present in table \"pool_metadata_ref\".", sqlErrorHint = ""})
cardano-db-sync-extended_1  | [db-sync-node.ErrorPolicy:Error:6] [2021-09-15 08:24:36.27 UTC] IP LocalAddress "/node-ipc/node.socket" ErrorPolicyUnhandledApplicationException (DbInsertException "PoolOfflineData" (SqlError {sqlState = "23503", sqlExecStatus = FatalError, sqlErrorMsg = "insert or update on table \"pool_offline_data\" violates foreign key constraint \"pool_offline_data_pmr_id_fkey\"", sqlErrorDetail = "Key (pmr_id)=(23322) is not present in table \"pool_metadata_ref\".", sqlErrorHint = ""}))
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:64] [2021-09-15 08:24:37.27 UTC] Identity Restarting Subscription after 254130.647440086s desired valency 1 current valency 0
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:64] [2021-09-15 08:24:37.27 UTC] Identity Starting Subscription Worker, valency 1
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:150362] [2021-09-15 08:24:37.27 UTC] Identity Connection Attempt Start, destination LocalAddress "/node-ipc/node.socket"
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:150362] [2021-09-15 08:24:37.27 UTC] Identity Connection Attempt End, destination LocalAddress "/node-ipc/node.socket" outcome: ConnectSuccessLast
cardano-db-sync-extended_1  | [db-sync-node.Handshake:Info:150362] [2021-09-15 08:24:37.27 UTC] WithMuxBearer (ConnectionId {localAddress = LocalAddress "", remoteAddress = LocalAddress "/ipc/node.socket"}) Send (ClientAgency TokPropose,MsgProposeVersions (fromList [(NodeToClientV_1,TInt 764824073),(NodeToClientV_2,TInt 764824073),(NodeToClientV_3,TInt 764824073),(NodeToClientV_4,TInt 764824073),(NodeToClientV_5,TInt 764824073),(NodeToClientV_6,TInt 764824073),(NodeToClientV_7,TInt 764824073),(NodeToClientV_8,TInt 764824073),(NodeToClientV_9,TInt 764824073)]))
cardano-db-sync-extended_1  | [db-sync-node.Handshake:Info:150362] [2021-09-15 08:24:37.27 UTC] WithMuxBearer (ConnectionId {localAddress = LocalAddress "", remoteAddress = LocalAddress "/ipc/node.socket"}) Recv (ServerAgency TokConfirm,MsgAcceptVersion NodeToClientV_9 (TInt 764824073))
cardano-db-sync-extended_1  | [db-sync-node.Mux:Info:150362] [2021-09-15 08:24:37.27 UTC] Bearer on ConnectionId {localAddress = LocalAddress "", remoteAddress = LocalAddress "/ipc/node.socket"} event: Handshake Client end, duration 0.000517092s
cardano-db-sync-extended_1  | [db-sync-node.Mux:Info:150363] [2021-09-15 08:24:37.27 UTC] Bearer on ConnectionId {localAddress = LocalAddress "", remoteAddress = LocalAddress "/ipc/node.socket"} event: State: Mature
cardano-db-sync-extended_1  | [db-sync-node:Info:150366] [2021-09-15 08:24:37.27 UTC] Starting chainSyncClient
cardano-db-sync-extended_1  | [db-sync-node:Info:150366] [2021-09-15 08:24:37.29 UTC] Cardano.Db tip is at slot 40127976, block 6246419
cardano-db-sync-extended_1  | [db-sync-node:Info:150371] [2021-09-15 08:24:37.29 UTC] Running DB thread
cardano-db-sync-extended_1  | [db-sync-node:Info:150371] [2021-09-15 08:24:37.29 UTC] Rolling back to slot 39916796, hash e72579ff89dc9ed325b723a33624b596c08141c7bd573ecfff56a1f7229e4d09
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:64] [2021-09-15 08:24:37.29 UTC] Identity Required subscriptions started
cardano-db-sync-extended_1  | [db-sync-node:Info:150371] [2021-09-15 08:24:37.33 UTC] Deleting 10360 blocks up to slot 40127976

Server 2

cardano-db-sync-extended_1  | [db-sync-node:Info:1207456] [2021-09-15 08:21:54.94 UTC] Offline pool metadata fetch: 47 results, 53 fetch errors
cardano-db-sync-extended_1  | [db-sync-node:Info:1207456] [2021-09-15 08:22:10.85 UTC] insertShelleyBlock(Alonzo): epoch 290, slot 40127838, block 6246411, hash 4c43818b951423dfd572b8b54e18f6982ea2ec2c923bac83fa548febb8193584
cardano-db-sync-extended_1  | [db-sync-node:Info:1207456] [2021-09-15 08:22:30.43 UTC] insertShelleyBlock(Alonzo): epoch 290, slot 40127857, block 6246412, hash 19e5d7c3d687090d98b50b2634b0c07751623abcd1f095be6c424baece094429
cardano-db-sync-extended_1  | [db-sync-node:Info:1207456] [2021-09-15 08:22:33.78 UTC] insertShelleyBlock(Alonzo): epoch 290, slot 40127861, block 6246413, hash 980a48ec01ea6122ed9c04a9c2f48054ceefae394e1315d85858d389bf045e16
cardano-db-sync-extended_1  | [db-sync-node:Info:1207456] [2021-09-15 08:23:38.15 UTC] insertShelleyBlock(Alonzo): epoch 290, slot 40127925, block 6246414, hash 378f58a6a65936f4fb049875b553e31585cbcc491d40181ac371ba9e88a119dd
cardano-db-sync-extended_1  | [db-sync-node:Info:1207456] [2021-09-15 08:23:57.41 UTC] insertShelleyBlock(Alonzo): epoch 290, slot 40127944, block 6246415, hash 86ebe9fd7835046e5447b60430d7ec870d318615ca808060f99852e451bea2f7
cardano-db-sync-extended_1  | [db-sync-node:Info:1207456] [2021-09-15 08:24:00.64 UTC] insertShelleyBlock(Alonzo): epoch 290, slot 40127947, block 6246416, hash b16e3bdaf6488a689fb3ddd372fd491b7cef744ad6f1da8e364a7b841cf74d58
cardano-db-sync-extended_1  | [db-sync-node:Info:1207456] [2021-09-15 08:24:03.89 UTC] insertShelleyBlock(Alonzo): epoch 290, slot 40127951, block 6246417, hash 2cd12a17ee9efeb042b44c6a271eab8a074dc9fa1dce19af888880d7f981303f
cardano-db-sync-extended_1  | [db-sync-node:Info:1207456] [2021-09-15 08:24:11.94 UTC] insertShelleyBlock(Alonzo): epoch 290, slot 40127959, block 6246418, hash 4d283e59f915e64721714f131f0b5c6bf8c2c4a7621db10ff9c5fd718d59aa12
cardano-db-sync-extended_1  | [db-sync-node:Info:1207456] [2021-09-15 08:24:28.68 UTC] insertShelleyBlock(Alonzo): epoch 290, slot 40127976, block 6246419, hash 6d0fe286fde5dc60253e2b0e1cbcffc0b22c71d8c765d5c2cc64eb874dbc4390
cardano-db-sync-extended_1  | [db-sync-node:Info:1207456] [2021-09-15 08:24:36.00 UTC] insertShelleyBlock(Alonzo): continuing epoch 290 (slot 211183/432000)
cardano-db-sync-extended_1  | [db-sync-node:Info:1207456] [2021-09-15 08:24:36.00 UTC] insertShelleyBlock(Alonzo): epoch 290, slot 40127983, block 6246420, hash d15131b76ea1e818e208e58cb4fa9d1b8c1d56d8642eb2552e1154cb3af29b97
cardano-db-sync-extended_1  | [db-sync-node:Info:1207456] [2021-09-15 08:24:36.00 UTC] Offline pool metadata fetch: 48 results, 52 fetch errors
cardano-db-sync-extended_1  | [db-sync-node:Error:1207456] [2021-09-15 08:24:36.00 UTC] runDBThread: DbInsertException "PoolOfflineData" (SqlError {sqlState = "23503", sqlExecStatus = FatalError, sqlErrorMsg = "insert or update on table \"pool_offline_data\" violates foreign key constraint \"pool_offline_data_pmr_id_fkey\"", sqlErrorDetail = "Key (pmr_id)=(22233) is not present in table \"pool_metadata_ref\".", sqlErrorHint = ""})
cardano-db-sync-extended_1  | [db-sync-node:Error:1207451] [2021-09-15 08:24:36.00 UTC] ChainSyncWithBlocksPtcl: DbInsertException "PoolOfflineData" (SqlError {sqlState = "23503", sqlExecStatus = FatalError, sqlErrorMsg = "insert or update on table \"pool_offline_data\" violates foreign key constraint \"pool_offline_data_pmr_id_fkey\"", sqlErrorDetail = "Key (pmr_id)=(22233) is not present in table \"pool_metadata_ref\".", sqlErrorHint = ""})
cardano-db-sync-extended_1  | [db-sync-node.Mux:Info:1207448] [2021-09-15 08:24:36.00 UTC] Bearer on ConnectionId {localAddress = LocalAddress "", remoteAddress = LocalAddress "/ipc/node.socket"} event: State: Dead
cardano-db-sync-extended_1  | [db-sync-node.Mux:Notice:1207448] [2021-09-15 08:24:36.00 UTC] Bearer on ConnectionId {localAddress = LocalAddress "", remoteAddress = LocalAddress "/ipc/node.socket"} event: Miniprotocol MiniProtocolNum 5 InitiatorDir terminated with exception DbInsertException "PoolOfflineData" (SqlError {sqlState = "23503", sqlExecStatus = FatalError, sqlErrorMsg = "insert or update on table \"pool_offline_data\" violates foreign key constraint \"pool_offline_data_pmr_id_fkey\"", sqlErrorDetail = "Key (pmr_id)=(22233) is not present in table \"pool_metadata_ref\".", sqlErrorHint = ""})
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Error:1207447] [2021-09-15 08:24:36.00 UTC] Identity Application Exception: LocalAddress "/node-ipc/node.socket" DbInsertException "PoolOfflineData" (SqlError {sqlState = "23503", sqlExecStatus = FatalError, sqlErrorMsg = "insert or update on table \"pool_offline_data\" violates foreign key constraint \"pool_offline_data_pmr_id_fkey\"", sqlErrorDetail = "Key (pmr_id)=(22233) is not present in table \"pool_metadata_ref\".", sqlErrorHint = ""})
cardano-db-sync-extended_1  | [db-sync-node.ErrorPolicy:Error:6] [2021-09-15 08:24:36.00 UTC] IP LocalAddress "/node-ipc/node.socket" ErrorPolicyUnhandledApplicationException (DbInsertException "PoolOfflineData" (SqlError {sqlState = "23503", sqlExecStatus = FatalError, sqlErrorMsg = "insert or update on table \"pool_offline_data\" violates foreign key constraint \"pool_offline_data_pmr_id_fkey\"", sqlErrorDetail = "Key (pmr_id)=(22233) is not present in table \"pool_metadata_ref\".", sqlErrorHint = ""}))
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:65] [2021-09-15 08:24:37.00 UTC] Identity Restarting Subscription after 123729.013160334s desired valency 1 current valency 0
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:65] [2021-09-15 08:24:37.00 UTC] Identity Starting Subscription Worker, valency 1
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:1289069] [2021-09-15 08:24:37.00 UTC] Identity Connection Attempt Start, destination LocalAddress "/node-ipc/node.socket"
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:1289069] [2021-09-15 08:24:37.00 UTC] Identity Connection Attempt End, destination LocalAddress "/node-ipc/node.socket" outcome: ConnectSuccessLast
cardano-db-sync-extended_1  | [db-sync-node.Handshake:Info:1289069] [2021-09-15 08:24:37.00 UTC] WithMuxBearer (ConnectionId {localAddress = LocalAddress "", remoteAddress = LocalAddress "/ipc/node.socket"}) Send (ClientAgency TokPropose,MsgProposeVersions (fromList [(NodeToClientV_1,TInt 764824073),(NodeToClientV_2,TInt 764824073),(NodeToClientV_3,TInt 764824073),(NodeToClientV_4,TInt 764824073),(NodeToClientV_5,TInt 764824073),(NodeToClientV_6,TInt 764824073),(NodeToClientV_7,TInt 764824073),(NodeToClientV_8,TInt 764824073),(NodeToClientV_9,TInt 764824073)]))
cardano-db-sync-extended_1  | [db-sync-node.Handshake:Info:1289069] [2021-09-15 08:24:37.00 UTC] WithMuxBearer (ConnectionId {localAddress = LocalAddress "", remoteAddress = LocalAddress "/ipc/node.socket"}) Recv (ServerAgency TokConfirm,MsgAcceptVersion NodeToClientV_9 (TInt 764824073))
cardano-db-sync-extended_1  | [db-sync-node.Mux:Info:1289069] [2021-09-15 08:24:37.00 UTC] Bearer on ConnectionId {localAddress = LocalAddress "", remoteAddress = LocalAddress "/ipc/node.socket"} event: Handshake Client end, duration 0.000346705s
cardano-db-sync-extended_1  | [db-sync-node.Mux:Info:1289070] [2021-09-15 08:24:37.00 UTC] Bearer on ConnectionId {localAddress = LocalAddress "", remoteAddress = LocalAddress "/ipc/node.socket"} event: State: Mature
cardano-db-sync-extended_1  | [db-sync-node:Info:1289073] [2021-09-15 08:24:37.00 UTC] Starting chainSyncClient
cardano-db-sync-extended_1  | [db-sync-node:Info:1289073] [2021-09-15 08:24:37.01 UTC] Cardano.Db tip is at slot 40127976, block 6246419
cardano-db-sync-extended_1  | [db-sync-node:Info:1289078] [2021-09-15 08:24:37.01 UTC] Running DB thread
cardano-db-sync-extended_1  | [db-sync-node:Info:1289078] [2021-09-15 08:24:37.02 UTC] Rolling back to slot 39916796, hash e72579ff89dc9ed325b723a33624b596c08141c7bd573ecfff56a1f7229e4d09
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:65] [2021-09-15 08:24:37.03 UTC] Identity Required subscriptions started
cardano-db-sync-extended_1  | [db-sync-node:Info:1289078] [2021-09-15 08:24:37.03 UTC] Deleting 10360 blocks up to slot 40127976

db-sync now needs 30 mins to catch up with the tip.
@MarcelKlammer MarcelKlammer added the bug Something isn't working label Sep 15, 2021
@Crypto2099
Copy link

Crypto2099 commented Sep 15, 2021 •
edited

Adding my 2 Lovelace...
db-sync-get-rekt.txt

Dedicated server, Ubuntu 20.04, not in docker.

@kingli-crypto
Copy link

same issue

@mmahut
Copy link

mmahut commented Sep 15, 2021 •
edited

Bare-metal, not in docker.

cardano-db-sync 11.0.0

Sep 15 08:21:54 dbsync2 638knhvcxaif6aszcvr9mrdrxq3iva6l-cardano-db-sync[1623381]: [db-sync-node:Info:4211] [2021-09-15 08:21:54.17 UTC] Offline pool metadata fetch: 36 results, 64 fetch errors
Sep 15 08:21:54 dbsync2 638knhvcxaif6aszcvr9mrdrxq3iva6l-cardano-db-sync[1623381]: [db-sync-node:Error:4211] [2021-09-15 08:21:54.18 UTC] runDBThread: DbInsertException "PoolOfflineData" (SqlError {sqlState = "23503", sqlExecStatus = FatalError, sqlErrorMsg = "insert or update on table \"pool_offline_data\" violates foreign key constraint \"pool_offline_data_pmr_id_fkey\"", sqlErrorDetail = "Key (pmr_id)=(22822) is not present in table \"pool_metadata_ref\".", sqlErrorHint = ""})
Sep 15 08:21:54 dbsync2 638knhvcxaif6aszcvr9mrdrxq3iva6l-cardano-db-sync[1623381]: [db-sync-node:Error:4206] [2021-09-15 08:21:54.18 UTC] ChainSyncWithBlocksPtcl: DbInsertException "PoolOfflineData" (SqlError {sqlState = "23503", sqlExecStatus = FatalError, sqlErrorMsg = "insert or update on table \"pool_offline_data\" violates foreign key constraint \"pool_offline_data_pmr_id_fkey\"", sqlErrorDetail = "Key (pmr_id)=(22822) is not present in table \"pool_metadata_ref\".", sqlErrorHint = ""})
Sep 15 08:21:54 dbsync2 638knhvcxaif6aszcvr9mrdrxq3iva6l-cardano-db-sync[1623381]: [db-sync-node.Mux:Info:4203] [2021-09-15 08:21:54.18 UTC] Bearer on ConnectionId {localAddress = LocalAddress "", remoteAddress = LocalAddress "/run/cardano-node/node.socket"} event: State: Dead
Sep 15 08:21:54 dbsync2 638knhvcxaif6aszcvr9mrdrxq3iva6l-cardano-db-sync[1623381]: [db-sync-node.Mux:Notice:4203] [2021-09-15 08:21:54.18 UTC] Bearer on ConnectionId {localAddress = LocalAddress "", remoteAddress = LocalAddress "/run/cardano-node/node.socket"} event: Miniprotocol MiniProtocolNum 5 InitiatorDir terminated with exception DbInsertException "PoolOfflineData" (SqlError {sqlState = "23503", sqlExecStatus = FatalError, sqlErrorMsg = "insert or update on table \"pool_offline_data\" violates foreign key constraint \"pool_offline_data_pmr_id_fkey\"", sqlErrorDetail = "Key (pmr_id)=(22822) is not present in table \"pool_metadata_ref\".", sqlErrorHint = ""})
Sep 15 08:21:54 dbsync2 638knhvcxaif6aszcvr9mrdrxq3iva6l-cardano-db-sync[1623381]: [db-sync-node.Subscription:Error:4202] [2021-09-15 08:21:54.18 UTC] Identity Application Exception: LocalAddress "/run/cardano-node/node.socket" DbInsertException "PoolOfflineData" (SqlError {sqlState = "23503", sqlExecStatus = FatalError, sqlErrorMsg = "insert or update on table \"pool_offline_data\" violates foreign key constraint \"pool_offline_data_pmr_id_fkey\"", sqlErrorDetail = "Key (pmr_id)=(22822) is not present in table \"pool_metadata_ref\".", sqlErrorHint = ""})
Sep 15 08:21:54 dbsync2 638knhvcxaif6aszcvr9mrdrxq3iva6l-cardano-db-sync[1623381]: [db-sync-node.ErrorPolicy:Error:6] [2021-09-15 08:21:54.18 UTC] IP LocalAddress "/run/cardano-node/node.socket" ErrorPolicyUnhandledApplicationException (DbInsertException "PoolOfflineData" (SqlError {sqlState = "23503", sqlExecStatus = FatalError, sqlErrorMsg = "insert or update on table \"pool_offline_data\" violates foreign key constraint \"pool_offline_data_pmr_id_fkey\"", sqlErrorDetail = "Key (pmr_id)=(22822) is not present in table \"pool_metadata_ref\".", sqlErrorHint = ""}))
Sep 15 08:21:55 dbsync2 638knhvcxaif6aszcvr9mrdrxq3iva6l-cardano-db-sync[1623381]: [db-sync-node.Subscription:Notice:64] [2021-09-15 08:21:55.18 UTC] Identity Restarting Subscription after 208968.57303432s desired valency 1 current valency 0
Sep 15 08:21:55 dbsync2 638knhvcxaif6aszcvr9mrdrxq3iva6l-cardano-db-sync[1623381]: [db-sync-node.Subscription:Notice:64] [2021-09-15 08:21:55.18 UTC] Identity Starting Subscription Worker, valency 1
Sep 15 08:21:55 dbsync2 638knhvcxaif6aszcvr9mrdrxq3iva6l-cardano-db-sync[1623381]: [db-sync-node.Subscription:Notice:115249] [2021-09-15 08:21:55.18 UTC] Identity Connection Attempt Start, destination LocalAddress "/run/cardano-node/node.socket"
Sep 15 08:21:55 dbsync2 638knhvcxaif6aszcvr9mrdrxq3iva6l-cardano-db-sync[1623381]: [db-sync-node.Subscription:Notice:115249] [2021-09-15 08:21:55.18 UTC] Identity Connection Attempt End, destination LocalAddress "/run/cardano-node/node.socket" outcome: ConnectSuccessLast
Sep 15 08:21:55 dbsync2 638knhvcxaif6aszcvr9mrdrxq3iva6l-cardano-db-sync[1623381]: [db-sync-node.Handshake:Info:115249] [2021-09-15 08:21:55.18 UTC] WithMuxBearer (ConnectionId {localAddress = LocalAddress "", remoteAddress = LocalAddress "/run/cardano-node/node.socket"}) Send (ClientAgency TokPropose,MsgProposeVersions (fromList [(NodeToClientV_1,TInt 764824073),(NodeToClientV_2,TInt 764824073),(NodeToClientV_3,TInt 764824073),(NodeToClientV_4,TInt 764824073),(NodeToClientV_5,TInt 764824073),(NodeToClientV_6,TInt 764824073),(NodeToClientV_7,TInt 764824073),(NodeToClientV_8,TInt 764824073),(NodeToClientV_9,TInt 764824073)]))
Sep 15 08:21:55 dbsync2 638knhvcxaif6aszcvr9mrdrxq3iva6l-cardano-db-sync[1623381]: [db-sync-node.Handshake:Info:115249] [2021-09-15 08:21:55.18 UTC] WithMuxBearer (ConnectionId {localAddress = LocalAddress "", remoteAddress = LocalAddress "/run/cardano-node/node.socket"}) Recv (ServerAgency TokConfirm,MsgAcceptVersion NodeToClientV_9 (TInt 764824073))
Sep 15 08:21:55 dbsync2 638knhvcxaif6aszcvr9mrdrxq3iva6l-cardano-db-sync[1623381]: [db-sync-node.Mux:Info:115249] [2021-09-15 08:21:55.18 UTC] Bearer on ConnectionId {localAddress = LocalAddress "", remoteAddress = LocalAddress "/run/cardano-node/node.socket"} event: Handshake Client end, duration 0.000297497s
Sep 15 08:21:55 dbsync2 638knhvcxaif6aszcvr9mrdrxq3iva6l-cardano-db-sync[1623381]: [db-sync-node.Mux:Info:115250] [2021-09-15 08:21:55.18 UTC] Bearer on ConnectionId {localAddress = LocalAddress "", remoteAddress = LocalAddress "/run/cardano-node/node.socket"} event: State: Mature
Sep 15 08:21:55 dbsync2 638knhvcxaif6aszcvr9mrdrxq3iva6l-cardano-db-sync[1623381]: [db-sync-node:Info:115253] [2021-09-15 08:21:55.18 UTC] Starting chainSyncClient
Sep 15 08:21:55 dbsync2 638knhvcxaif6aszcvr9mrdrxq3iva6l-cardano-db-sync[1623381]: [db-sync-node:Info:115253] [2021-09-15 08:21:55.19 UTC] Cardano.Db tip is at slot 40127795, block 6246409
Sep 15 08:21:55 dbsync2 638knhvcxaif6aszcvr9mrdrxq3iva6l-cardano-db-sync[1623381]: [db-sync-node:Info:115258] [2021-09-15 08:21:55.19 UTC] Running DB thread
Sep 15 08:21:55 dbsync2 638knhvcxaif6aszcvr9mrdrxq3iva6l-cardano-db-sync[1623381]: [db-sync-node:Info:115258] [2021-09-15 08:21:55.19 UTC] Rolling back to slot 39916796, hash e72579ff89dc9ed325b723a33624b596c08141c7bd573ecfff56a1f7229e4d09
Sep 15 08:21:55 dbsync2 638knhvcxaif6aszcvr9mrdrxq3iva6l-cardano-db-sync[1623381]: [db-sync-node.Subscription:Notice:64] [2021-09-15 08:21:55.20 UTC] Identity Required subscriptions started
Sep 15 08:21:55 dbsync2 638knhvcxaif6aszcvr9mrdrxq3iva6l-cardano-db-sync[1623381]: [db-sync-node:Info:115258] [2021-09-15 08:21:55.20 UTC] Deleting 10350 blocks up to slot 40127795
Sep 15 08:33:25 dbsync2 638knhvcxaif6aszcvr9mrdrxq3iva6l-cardano-db-sync[1623381]: [db-sync-node:Info:115258] [2021-09-15 08:33:25.76 UTC] Blocks deleted

@cardanians
Copy link
Contributor

same here (no docker)

@xdzurman
Copy link

Same here. Docker.

@erikd
Copy link
Contributor

erikd commented Sep 15, 2021 •
edited

The rollback issue is fixed in a105cb5 . The fix will be in the 11.0.1 release.

Its basically a race condition as explained here #806 (comment) .

@erikd
Copy link
Contributor

erikd commented Sep 15, 2021

Closing this as a duplicate of #806 .

@erikd erikd closed this as completed Sep 15, 2021
@kderme
Copy link
Contributor

kderme commented Sep 15, 2021

The rollback issue is fixed in a105cb5 . The fix will be in the 11.0.1 release.

I think it will be in 11.0.2 release

@erikd erikd mentioned this issue Sep 16, 2021
Closed
erikd pushed a commit that referenced this issue Sep 21, 2021
@kderme @erikd
Check foreign key exists before insertion
f3df1b7 
Closes: #806
Closes: #823
erikd pushed a commit that referenced this issue Sep 21, 2021
@kderme @erikd
Check foreign key exists before insertion
6cd822c 
This avoids the race condition described in the ticket #806 below. It
does so by ensuring the required foreign keys exist before the insert.
If the required foreign keys do not exist the data is just dropped and
will be refetched later.

Closes: #806
Closes: #823
erikd pushed a commit that referenced this issue Sep 21, 2021
@kderme @erikd
Check foreign key exists before insertion
d658f38 
This avoids the race condition described in the ticket #806 below. It
does so by ensuring the required foreign keys exist before the insert.
If the required foreign keys do not exist the data is just dropped and
will be refetched later.

Closes: #806
Closes: #823
Closes: #831
kderme added a commit that referenced this issue Sep 21, 2021
@kderme @erikd
Check foreign key exists before insertion
71988e2 
This avoids the race condition described in the ticket #806 below. It
does so by ensuring the required foreign keys exist before the insert.
If the required foreign keys do not exist the data is just dropped and
will be refetched later.

Cherry pick of d658f38 from master.

Closes: #806
Closes: #823
Closes: #831
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@mmahut @erikd @MarcelKlammer @kderme @xdzurman @cardanians @Crypto2099 @kingli-crypto