Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Node network error followed by schema migration failure #809

Closed
erikd opened this issue Sep 12, 2021 · 2 comments
Closed

Node network error followed by schema migration failure #809

erikd opened this issue Sep 12, 2021 · 2 comments

Comments

@erikd
Copy link
Contributor

erikd commented Sep 12, 2021 •
edited
Loading

From @MarcelKlammer

cardano-db-sync-extended_1  | [db-sync-node:Info:69791] [2021-09-12 09:10:32.35 UTC] insertShelleyBlock(Mary): epoch 289, slot 39871541, block 6233753, hash fcc8db27fd348fd44d817a0f075e42ad12754310dbca84f58082002256541a6b
cardano-db-sync-extended_1  | [db-sync-node:Info:69791] [2021-09-12 09:10:55.37 UTC] insertShelleyBlock(Mary): epoch 289, slot 39871563, block 6233754, hash 25078225a1b7693b4e11df588f3d9aeda0ed1b8ccada4ddeeaabddb990832626
cardano-db-sync-extended_1  | [db-sync-node:Info:69791] [2021-09-12 09:11:33.38 UTC] insertShelleyBlock(Mary): epoch 289, slot 39871602, block 6233755, hash 74c8d544699187c33a62bbe4aeb3ddd6ef46d02a2c5f0296ca1d2745d0c515c9
cardano-db-sync-extended_1  | [db-sync-node:Info:69791] [2021-09-12 09:11:38.75 UTC] insertShelleyBlock(Mary): epoch 289, slot 39871607, block 6233756, hash 17fc5bd96e76f119b725754c063f42fe110744dff17b45dc2c18d9b9d6890f54
cardano-db-sync-extended_1  | [db-sync-node:Info:69791] [2021-09-12 09:11:49.52 UTC] insertShelleyBlock(Mary): epoch 289, slot 39871618, block 6233757, hash 512c07ffca101437fdc55bbc8c004452e9d6755018076768388ffa4d9a46cf4d
cardano-db-sync-extended_1  | [db-sync-node.Mux:Info:69783] [2021-09-12 09:12:26.64 UTC] Bearer on ConnectionId {localAddress = LocalAddress "", remoteAddress = LocalAddress "/ipc/node.socket"} event: State: Dead
cardano-db-sync-extended_1  | [db-sync-node:Error:69791] [2021-09-12 09:12:26.64 UTC] runDBThread: AsyncCancelled
cardano-db-sync-extended_1  | [db-sync-node:Error:69786] [2021-09-12 09:12:26.64 UTC] ChainSyncWithBlocksPtcl: AsyncCancelled
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Error:69782] [2021-09-12 09:12:26.64 UTC] Identity Application Exception: LocalAddress "/node-ipc/node.socket" MuxError MuxBearerClosed "<socket: 22> closed when reading data, waiting on next header True"
cardano-db-sync-extended_1  | [db-sync-node.ErrorPolicy:Warning:6] [2021-09-12 09:12:26.64 UTC] IP LocalAddress "/node-ipc/node.socket" ErrorPolicySuspendPeer (Just (ApplicationExceptionTrace (MuxError MuxBearerClosed "<socket: 22> closed when reading data, waiting on next header True"))) 20s 20s
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:64] [2021-09-12 09:12:28.09 UTC] Identity Restarting Subscription after 143410.725933692s desired valency 1 current valency 0
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:64] [2021-09-12 09:12:28.09 UTC] Identity Starting Subscription Worker, valency 1
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:161806] [2021-09-12 09:12:28.09 UTC] Identity Connection Attempt Start, destination LocalAddress "/node-ipc/node.socket"
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Error:161806] [2021-09-12 09:12:28.09 UTC] Identity Connection Attempt Exception, destination LocalAddress "/node-ipc/node.socket" exception: Network.Socket.connect: <socket: 22>: does not exist (Connection refused)
cardano-db-sync-extended_1  | [db-sync-node.ErrorPolicy:Warning:6] [2021-09-12 09:12:28.09 UTC] IP LocalAddress "/node-ipc/node.socket" ErrorPolicySuspendPeer (Just (ConnectionExceptionTrace Network.Socket.connect: <socket: 22>: does not exist (Connection refused))) 20s 20s
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Warning:64] [2021-09-12 09:12:28.11 UTC] Identity Failed to start all required subscriptions
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:64] [2021-09-12 09:12:29.48 UTC] Identity Restarting Subscription after 1.391498382s desired valency 1 current valency 0
cardano-db-sync-extended_1  | Connecting to network specified in configuration.yaml
cardano-db-sync-extended_1  | [db-sync-node:Info:6] [2021-09-12 09:12:37.03 UTC] Schema migration files validated
cardano-db-sync-extended_1  | [db-sync-node:Info:6] [2021-09-12 09:12:37.03 UTC] Running database migrations
cardano-db-sync-extended_1  | ExitFailure 2
cardano-db-sync-extended_1  | 
cardano-db-sync-extended_1  | Errors in file: /tmp/migrate-2021-09-12T091237.log
cardano-db-sync-extended_1  | 
cardano-db-sync-extended_1  | Connecting to network specified in configuration.yaml
cardano-db-sync-extended_1  | [db-sync-node:Info:6] [2021-09-12 09:12:37.79 UTC] Schema migration files validated
cardano-db-sync-extended_1  | [db-sync-node:Info:6] [2021-09-12 09:12:37.79 UTC] Running database migrations
cardano-db-sync-extended_1  | ExitFailure 2
cardano-db-sync-extended_1  | 
cardano-db-sync-extended_1  | Errors in file: /tmp/migrate-2021-09-12T091237.log
cardano-db-sync-extended_1  | 
cardano-db-sync-extended_1  | Connecting to network specified in configuration.yaml
cardano-db-sync-extended_1  | [db-sync-node:Info:6] [2021-09-12 09:12:38.65 UTC] Schema migration files validated
cardano-db-sync-extended_1  | [db-sync-node:Info:6] [2021-09-12 09:12:38.65 UTC] Running database migrations
cardano-db-sync-extended_1  | [db-sync-node:Info:6] [2021-09-12 09:12:39.22 UTC] Using byron genesis file from: "/config/cardano-db-sync/../cardano-node/../genesis/byron.json"
cardano-db-sync-extended_1  | [db-sync-node:Info:6] [2021-09-12 09:12:39.22 UTC] Using shelley genesis file from: "/config/cardano-db-sync/../cardano-node/../genesis/shelley.json"
cardano-db-sync-extended_1  | [db-sync-node:Info:6] [2021-09-12 09:12:39.22 UTC] Using alonzo genesis file from: "/config/cardano-db-sync/../cardano-node/../genesis/alonzo.json"
cardano-db-sync-extended_1  | [db-sync-node:Info:6] [2021-09-12 09:12:39.95 UTC] NetworkMagic: 764824073
cardano-db-sync-extended_1  | [db-sync-node:Info:6] [2021-09-12 09:12:40.51 UTC] Initial genesis distribution present and correct
cardano-db-sync-extended_1  | [db-sync-node:Info:6] [2021-09-12 09:12:40.51 UTC] Total genesis supply of Ada: 31112484745.000000
cardano-db-sync-extended_1  | [db-sync-node:Info:6] [2021-09-12 09:12:40.51 UTC] Inserting Shelley Genesis distribution
cardano-db-sync-extended_1  | [db-sync-node:Info:6] [2021-09-12 09:12:40.51 UTC] epochPluginOnStartup: Checking
cardano-db-sync-extended_1  | [db-sync-node:Info:6] [2021-09-12 09:12:40.51 UTC] localInitiatorNetworkApplication: connecting to node via "/node-ipc/node.socket"
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:64] [2021-09-12 09:12:40.51 UTC] Identity Starting Subscription Worker, valency 1
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:65] [2021-09-12 09:12:40.51 UTC] Identity Connection Attempt Start, destination LocalAddress "/node-ipc/node.socket"
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Error:65] [2021-09-12 09:12:40.51 UTC] Identity Connection Attempt Exception, destination LocalAddress "/node-ipc/node.socket" exception: Network.Socket.connect: <socket: 22>: does not exist (No such file or directory)
cardano-db-sync-extended_1  | [db-sync-node.ErrorPolicy:Warning:6] [2021-09-12 09:12:40.51 UTC] IP LocalAddress "/node-ipc/node.socket" ErrorPolicySuspendPeer (Just (ConnectionExceptionTrace Network.Socket.connect: <socket: 22>: does not exist (No such file or directory))) 20s 20s
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Warning:64] [2021-09-12 09:12:40.54 UTC] Identity Failed to start all required subscriptions
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:64] [2021-09-12 09:12:41.54 UTC] Identity Restarting Subscription after 1.026788277s desired valency 1 current valency 0
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:64] [2021-09-12 09:12:50.52 UTC] Identity Starting Subscription Worker, valency 1
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:67] [2021-09-12 09:12:50.52 UTC] Identity Connection Attempt Start, destination LocalAddress "/node-ipc/node.socket"
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Error:67] [2021-09-12 09:12:50.52 UTC] Identity Connection Attempt Exception, destination LocalAddress "/node-ipc/node.socket" exception: Network.Socket.connect: <socket: 22>: does not exist (No such file or directory)
cardano-db-sync-extended_1  | [db-sync-node.ErrorPolicy:Warning:6] [2021-09-12 09:12:50.52 UTC] IP LocalAddress "/node-ipc/node.socket" ErrorPolicySuspendPeer (Just (ConnectionExceptionTrace Network.Socket.connect: <socket: 22>: does not exist (No such file or directory))) 20s 20s
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Warning:64] [2021-09-12 09:12:50.55 UTC] Identity Failed to start all required subscriptions
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:64] [2021-09-12 09:12:51.55 UTC] Identity Restarting Subscription after 1.02633279s desired valency 1 current valency 0
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:64] [2021-09-12 09:13:00.53 UTC] Identity Starting Subscription Worker, valency 1
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:69] [2021-09-12 09:13:00.53 UTC] Identity Connection Attempt Start, destination LocalAddress "/node-ipc/node.socket"
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Error:69] [2021-09-12 09:13:00.53 UTC] Identity Connection Attempt Exception, destination LocalAddress "/node-ipc/node.socket" exception: Network.Socket.connect: <socket: 22>: does not exist (No such file or directory)
cardano-db-sync-extended_1  | [db-sync-node.ErrorPolicy:Warning:6] [2021-09-12 09:13:00.53 UTC] IP LocalAddress "/node-ipc/node.socket" ErrorPolicySuspendPeer (Just (ConnectionExceptionTrace Network.Socket.connect: <socket: 22>: does not exist (No such file or directory))) 20s 20s
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Warning:64] [2021-09-12 09:13:00.56 UTC] Identity Failed to start all required subscriptions
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:64] [2021-09-12 09:13:01.56 UTC] Identity Restarting Subscription after 1.02667043s desired valency 1 current valency 0
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:64] [2021-09-12 09:13:10.54 UTC] Identity Starting Subscription Worker, valency 1
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:71] [2021-09-12 09:13:10.54 UTC] Identity Connection Attempt Start, destination LocalAddress "/node-ipc/node.socket"
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Error:71] [2021-09-12 09:13:10.54 UTC] Identity Connection Attempt Exception, destination LocalAddress "/node-ipc/node.socket" exception: Network.Socket.connect: <socket: 22>: does not exist (No such file or directory)
cardano-db-sync-extended_1  | [db-sync-node.ErrorPolicy:Warning:6] [2021-09-12 09:13:10.54 UTC] IP LocalAddress "/node-ipc/node.socket" ErrorPolicySuspendPeer (Just (ConnectionExceptionTrace Network.Socket.connect: <socket: 22>: does not exist (No such file or directory))) 20s 20s
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Warning:64] [2021-09-12 09:13:10.57 UTC] Identity Failed to start all required subscriptions
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:64] [2021-09-12 09:13:11.57 UTC] Identity Restarting Subscription after 1.02649241s desired valency 1 current valency 0
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:64] [2021-09-12 09:13:20.55 UTC] Identity Starting Subscription Worker, valency 1
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:73] [2021-09-12 09:13:20.55 UTC] Identity Connection Attempt Start, destination LocalAddress "/node-ipc/node.socket"
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Error:73] [2021-09-12 09:13:20.55 UTC] Identity Connection Attempt Exception, destination LocalAddress "/node-ipc/node.socket" exception: Network.Socket.connect: <socket: 22>: does not exist (No such file or directory)
cardano-db-sync-extended_1  | [db-sync-node.ErrorPolicy:Warning:6] [2021-09-12 09:13:20.55 UTC] IP LocalAddress "/node-ipc/node.socket" ErrorPolicySuspendPeer (Just (ConnectionExceptionTrace Network.Socket.connect: <socket: 22>: does not exist (No such file or directory))) 20s 20s
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Warning:64] [2021-09-12 09:13:20.57 UTC] Identity Failed to start all required subscriptions
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:64] [2021-09-12 09:13:21.58 UTC] Identity Restarting Subscription after 1.026461715s desired valency 1 current valency 0
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:64] [2021-09-12 09:13:30.56 UTC] Identity Starting Subscription Worker, valency 1
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:75] [2021-09-12 09:13:30.56 UTC] Identity Connection Attempt Start, destination LocalAddress "/node-ipc/node.socket"
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:75] [2021-09-12 09:13:30.56 UTC] Identity Connection Attempt End, destination LocalAddress "/node-ipc/node.socket" outcome: ConnectSuccessLast
cardano-db-sync-extended_1  | [db-sync-node.Handshake:Info:75] [2021-09-12 09:13:30.56 UTC] WithMuxBearer (ConnectionId {localAddress = LocalAddress "", remoteAddress = LocalAddress "/ipc/node.socket"}) Send (ClientAgency TokPropose,MsgProposeVersions (fromList [(NodeToClientV_1,TInt 764824073),(NodeToClientV_2,TInt 764824073),(NodeToClientV_3,TInt 764824073),(NodeToClientV_4,TInt 764824073),(NodeToClientV_5,TInt 764824073),(NodeToClientV_6,TInt 764824073),(NodeToClientV_7,TInt 764824073),(NodeToClientV_8,TInt 764824073),(NodeToClientV_9,TInt 764824073)]))
cardano-db-sync-extended_1  | [db-sync-node.Handshake:Info:75] [2021-09-12 09:13:30.56 UTC] WithMuxBearer (ConnectionId {localAddress = LocalAddress "", remoteAddress = LocalAddress "/ipc/node.socket"}) Recv (ServerAgency TokConfirm,MsgAcceptVersion NodeToClientV_9 (TInt 764824073))
cardano-db-sync-extended_1  | [db-sync-node.Mux:Info:75] [2021-09-12 09:13:30.56 UTC] Bearer on ConnectionId {localAddress = LocalAddress "", remoteAddress = LocalAddress "/ipc/node.socket"} event: Handshake Client end, duration 0.000673874s
cardano-db-sync-extended_1  | [db-sync-node.Mux:Info:76] [2021-09-12 09:13:30.56 UTC] Bearer on ConnectionId {localAddress = LocalAddress "", remoteAddress = LocalAddress "/ipc/node.socket"} event: State: Mature
cardano-db-sync-extended_1  | [db-sync-node:Info:79] [2021-09-12 09:13:30.56 UTC] Starting chainSyncClient
cardano-db-sync-extended_1  | [db-sync-node:Info:79] [2021-09-12 09:13:30.58 UTC] Cardano.Db tip is at slot 39871618, block 6233757
cardano-db-sync-extended_1  | [db-sync-node:Info:84] [2021-09-12 09:13:30.58 UTC] Running DB thread
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:64] [2021-09-12 09:13:30.58 UTC] Identity Required subscriptions started
cardano-db-sync-extended_1  | [db-sync-node:Info:84] [2021-09-12 09:13:30.60 UTC] Rolling back to slot 39484797, hash 86ad35fd0bf316a036dd8794400ead9d4431d6df28eab986864bd5ef65e97f87
cardano-db-sync-extended_1  | [db-sync-node:Info:84] [2021-09-12 09:13:30.66 UTC] Deleting 18893 blocks up to slot 39871618
cardano-db-sync-extended_1  | [db-sync-node:Info:84] [2021-09-12 09:15:42.44 UTC] Blocks deleted
cardano-db-sync-extended_1  | [db-sync-node:Info:84] [2021-09-12 09:16:04.34 UTC] Found snapshot file for slot 39484797, hash 86ad35fd0bf316a036dd8794400ead9d4431d6df28eab986864bd5ef65e97f87
cardano-db-sync-extended_1  | [db-sync-node:Info:87] [2021-09-12 09:16:14.44 UTC] getHistoryInterpreter: acquired
cardano-db-sync-extended_1  | [db-sync-node:Info:98] [2021-09-12 09:16:14.44 UTC] Running Offline fetch thread
cardano-db-sync-extended_1  | [db-sync-node:Info:84] [2021-09-12 09:16:14.44 UTC] thisIsAnUglyHack: Main thead
cardano-db-sync-extended_1  | [db-sync-node:Info:84] [2021-09-12 09:16:49.25 UTC] File /data/39484797-86ad35fd0b-288.lstate exists
cardano-db-sync-extended_1  | [db-sync-node:Info:84] [2021-09-12 09:16:49.25 UTC] Starting at epoch 289
cardano-db-sync-extended_1  | [db-sync-node:Info:84] [2021-09-12 09:16:51.71 UTC] Handling 801247 stakes for epoch 289 slot 39484840, hash faf4ceb70f91e6ed3d1baa584780f6d3e7a29a23bb17287f5bb1283ae5bdf4bb
cardano-db-sync-extended_1  | [db-sync-node:Info:84] [2021-09-12 09:19:51.91 UTC] insertShelleyBlock(Mary): epoch 289, slot 39488197, block 6215000, hash 9daef3159978f9da08cf02f726f68ac01e7d5883cebf96f418d6dcdc9ee2056a
cardano-db-sync-extended_1  | [db-sync-node:Info:84] [2021-09-12 09:22:10.90 UTC] insertEpochInterleaved: Epoch 289, 801247 stake addresses
cardano-db-sync-extended_1  | [db-sync-node:Info:84] [2021-09-12 09:27:24.74 UTC] Offline pool metadata fetch: 51 results, 49 fetch errors
cardano-db-sync-extended_1  | [db-sync-node:Info:84] [2021-09-12 09:34:18.79 UTC] insertShelleyBlock(Mary): epoch 289, slot 39590888, block 6220000, hash 8ad4e3bfad0490812ed62501c2f3872c5cb5d369f063f4f083fd22684ebc13e0
cardano-db-sync-extended_1  | [db-sync-node:Info:84] [2021-09-12 09:34:18.79 UTC] Offline pool metadata fetch: 55 results, 45 fetch errors
cardano-db-sync-extended_1  | [db-sync-node:Info:84] [2021-09-12 09:38:03.23 UTC] Offline pool metadata fetch: 50 results, 50 fetch errors
cardano-db-sync-extended_1  | [db-sync-node:Info:84] [2021-09-12 09:47:36.16 UTC] Offline pool metadata fetch: 53 results, 47 fetch errors
cardano-db-sync-extended_1  | [db-sync-node.Mux:Info:76] [2021-09-12 09:47:56.55 UTC] Bearer on ConnectionId {localAddress = LocalAddress "", remoteAddress = LocalAddress "/ipc/node.socket"} event: State: Dead
cardano-db-sync-extended_1  | [db-sync-node:Error:87] [2021-09-12 09:47:56.55 UTC] recvMsgRollForward: AsyncCancelled
cardano-db-sync-extended_1  | [db-sync-node:Error:84] [2021-09-12 09:47:56.55 UTC] runDBThread: AsyncCancelled
cardano-db-sync-extended_1  | [db-sync-node:Error:79] [2021-09-12 09:47:56.55 UTC] ChainSyncWithBlocksPtcl: AsyncCancelled
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Error:75] [2021-09-12 09:47:56.55 UTC] Identity Application Exception: LocalAddress "/node-ipc/node.socket" MuxError MuxBearerClosed "<socket: 22> closed when reading data, waiting on next header True"
cardano-db-sync-extended_1  | [db-sync-node.ErrorPolicy:Warning:6] [2021-09-12 09:47:56.55 UTC] IP LocalAddress "/node-ipc/node.socket" ErrorPolicySuspendPeer (Just (ApplicationExceptionTrace (MuxError MuxBearerClosed "<socket: 22> closed when reading data, waiting on next header True"))) 20s 20s
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:64] [2021-09-12 09:47:58.96 UTC] Identity Restarting Subscription after 2068.398953455s desired valency 1 current valency 0
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:64] [2021-09-12 09:47:58.96 UTC] Identity Starting Subscription Worker, valency 1
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:1796] [2021-09-12 09:47:58.96 UTC] Identity Connection Attempt Start, destination LocalAddress "/node-ipc/node.socket"
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Error:1796] [2021-09-12 09:47:58.96 UTC] Identity Connection Attempt Exception, destination LocalAddress "/node-ipc/node.socket" exception: Network.Socket.connect: <socket: 22>: does not exist (No such file or directory)
cardano-db-sync-extended_1  | [db-sync-node.ErrorPolicy:Warning:6] [2021-09-12 09:47:58.96 UTC] IP LocalAddress "/node-ipc/node.socket" ErrorPolicySuspendPeer (Just (ConnectionExceptionTrace Network.Socket.connect: <socket: 22>: does not exist (No such file or directory))) 20s 20s
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Warning:64] [2021-09-12 09:47:58.98 UTC] Identity Failed to start all required subscriptions
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:64] [2021-09-12 09:48:00.82 UTC] Identity Restarting Subscription after 1.865099839s desired valency 1 current valency 0
cardano-db-sync-extended_1  | Connecting to network specified in configuration.yaml
cardano-db-sync-extended_1  | [db-sync-node:Info:6] [2021-09-12 09:48:07.50 UTC] Schema migration files validated
cardano-db-sync-extended_1  | [db-sync-node:Info:6] [2021-09-12 09:48:07.50 UTC] Running database migrations
cardano-db-sync-extended_1  | ExitFailure 2
cardano-db-sync-extended_1  | 
cardano-db-sync-extended_1  | Errors in file: /tmp/migrate-2021-09-12T094807.log
cardano-db-sync-extended_1  | 
cardano-db-sync-extended_1  | Connecting to network specified in configuration.yaml
cardano-db-sync-extended_1  | [db-sync-node:Info:6] [2021-09-12 09:48:08.62 UTC] Schema migration files validated
cardano-db-sync-extended_1  | [db-sync-node:Info:6] [2021-09-12 09:48:08.62 UTC] Running database migrations
cardano-db-sync-extended_1  | ExitFailure 2
cardano-db-sync-extended_1  | 
cardano-db-sync-extended_1  | Errors in file: /tmp/migrate-2021-09-12T094808.log
cardano-db-sync-extended_1  | 
cardano-db-sync-extended_1  | Connecting to network specified in configuration.yaml
cardano-db-sync-extended_1  | [db-sync-node:Info:6] [2021-09-12 09:48:09.48 UTC] Schema migration files validated
cardano-db-sync-extended_1  | [db-sync-node:Info:6] [2021-09-12 09:48:09.48 UTC] Running database migrations
cardano-db-sync-extended_1  | ExitFailure 2
cardano-db-sync-extended_1  | 
cardano-db-sync-extended_1  | Errors in file: /tmp/migrate-2021-09-12T094809.log
cardano-db-sync-extended_1  | 
cardano-db-sync-extended_1  | Connecting to network specified in configuration.yaml
cardano-db-sync-extended_1  | [db-sync-node:Info:6] [2021-09-12 09:48:10.45 UTC] Schema migration files validated
cardano-db-sync-extended_1  | [db-sync-node:Info:6] [2021-09-12 09:48:10.45 UTC] Running database migrations
cardano-db-sync-extended_1  | ExitFailure 2
cardano-db-sync-extended_1  | 
cardano-db-sync-extended_1  | Errors in file: /tmp/migrate-2021-09-12T094810.log
cardano-db-sync-extended_1  | 
cardano-db-sync-extended_1  | Connecting to network specified in configuration.yaml
cardano-db-sync-extended_1  | [db-sync-node:Info:6] [2021-09-12 09:48:11.82 UTC] Schema migration files validated
cardano-db-sync-extended_1  | [db-sync-node:Info:6] [2021-09-12 09:48:11.82 UTC] Running database migrations
cardano-db-sync-extended_1  | ExitFailure 2
cardano-db-sync-extended_1  | 
cardano-db-sync-extended_1  | Errors in file: /tmp/migrate-2021-09-12T094811.log
cardano-db-sync-extended_1  | 
cardano-db-sync-extended_1  | Connecting to network specified in configuration.yaml
cardano-db-sync-extended_1  | [db-sync-node:Info:6] [2021-09-12 09:48:14.03 UTC] Schema migration files validated
cardano-db-sync-extended_1  | [db-sync-node:Info:6] [2021-09-12 09:48:14.03 UTC] Running database migrations
cardano-db-sync-extended_1  | ExitFailure 2
cardano-db-sync-extended_1  | 
cardano-db-sync-extended_1  | Errors in file: /tmp/migrate-2021-09-12T094814.log
cardano-db-sync-extended_1  | 
cardano-db-sync-extended_1  | Connecting to network specified in configuration.yaml
cardano-db-sync-extended_1  | [db-sync-node:Info:6] [2021-09-12 09:48:17.82 UTC] Schema migration files validated
cardano-db-sync-extended_1  | [db-sync-node:Info:6] [2021-09-12 09:48:17.82 UTC] Running database migrations
cardano-db-sync-extended_1  | ExitFailure 2
cardano-db-sync-extended_1  | 
cardano-db-sync-extended_1  | Errors in file: /tmp/migrate-2021-09-12T094817.log
cardano-db-sync-extended_1  | 
cardano-db-sync-extended_1  | Connecting to network specified in configuration.yaml
cardano-db-sync-extended_1  | [db-sync-node:Info:6] [2021-09-12 09:48:24.79 UTC] Schema migration files validated
cardano-db-sync-extended_1  | [db-sync-node:Info:6] [2021-09-12 09:48:24.79 UTC] Running database migrations
cardano-db-sync-extended_1  | [db-sync-node:Info:6] [2021-09-12 09:48:25.38 UTC] Using byron genesis file from: "/config/cardano-db-sync/../cardano-node/../genesis/byron.json"
cardano-db-sync-extended_1  | [db-sync-node:Info:6] [2021-09-12 09:48:25.38 UTC] Using shelley genesis file from: "/config/cardano-db-sync/../cardano-node/../genesis/shelley.json"
cardano-db-sync-extended_1  | [db-sync-node:Info:6] [2021-09-12 09:48:25.38 UTC] Using alonzo genesis file from: "/config/cardano-db-sync/../cardano-node/../genesis/alonzo.json"
cardano-db-sync-extended_1  | [db-sync-node:Info:6] [2021-09-12 09:48:25.98 UTC] NetworkMagic: 764824073
cardano-db-sync-extended_1  | [db-sync-node:Info:6] [2021-09-12 09:48:26.58 UTC] Initial genesis distribution present and correct
cardano-db-sync-extended_1  | [db-sync-node:Info:6] [2021-09-12 09:48:26.58 UTC] Total genesis supply of Ada: 31112484745.000000
cardano-db-sync-extended_1  | [db-sync-node:Info:6] [2021-09-12 09:48:26.58 UTC] Inserting Shelley Genesis distribution
cardano-db-sync-extended_1  | [db-sync-node:Info:6] [2021-09-12 09:48:26.58 UTC] epochPluginOnStartup: Checking
cardano-db-sync-extended_1  | [db-sync-node:Info:6] [2021-09-12 09:48:26.59 UTC] localInitiatorNetworkApplication: connecting to node via "/node-ipc/node.socket"
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:64] [2021-09-12 09:48:26.59 UTC] Identity Starting Subscription Worker, valency 1
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:65] [2021-09-12 09:48:26.59 UTC] Identity Connection Attempt Start, destination LocalAddress "/node-ipc/node.socket"
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Error:65] [2021-09-12 09:48:26.59 UTC] Identity Connection Attempt Exception, destination LocalAddress "/node-ipc/node.socket" exception: Network.Socket.connect: <socket: 22>: does not exist (No such file or directory)
cardano-db-sync-extended_1  | [db-sync-node.ErrorPolicy:Warning:6] [2021-09-12 09:48:26.59 UTC] IP LocalAddress "/node-ipc/node.socket" ErrorPolicySuspendPeer (Just (ConnectionExceptionTrace Network.Socket.connect: <socket: 22>: does not exist (No such file or directory))) 20s 20s
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Warning:64] [2021-09-12 09:48:26.61 UTC] Identity Failed to start all required subscriptions
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:64] [2021-09-12 09:48:27.61 UTC] Identity Restarting Subscription after 1.027143753s desired valency 1 current valency 0
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:64] [2021-09-12 09:48:36.59 UTC] Identity Starting Subscription Worker, valency 1
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:67] [2021-09-12 09:48:36.59 UTC] Identity Connection Attempt Start, destination LocalAddress "/node-ipc/node.socket"
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Error:67] [2021-09-12 09:48:36.59 UTC] Identity Connection Attempt Exception, destination LocalAddress "/node-ipc/node.socket" exception: Network.Socket.connect: <socket: 22>: does not exist (No such file or directory)
cardano-db-sync-extended_1  | [db-sync-node.ErrorPolicy:Warning:6] [2021-09-12 09:48:36.59 UTC] IP LocalAddress "/node-ipc/node.socket" ErrorPolicySuspendPeer (Just (ConnectionExceptionTrace Network.Socket.connect: <socket: 22>: does not exist (No such file or directory))) 20s 20s
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Warning:64] [2021-09-12 09:48:36.62 UTC] Identity Failed to start all required subscriptions
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:64] [2021-09-12 09:48:37.62 UTC] Identity Restarting Subscription after 1.026427393s desired valency 1 current valency 0
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:64] [2021-09-12 09:48:46.60 UTC] Identity Starting Subscription Worker, valency 1
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:69] [2021-09-12 09:48:46.60 UTC] Identity Connection Attempt Start, destination LocalAddress "/node-ipc/node.socket"
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Error:69] [2021-09-12 09:48:46.60 UTC] Identity Connection Attempt Exception, destination LocalAddress "/node-ipc/node.socket" exception: Network.Socket.connect: <socket: 22>: does not exist (No such file or directory)
cardano-db-sync-extended_1  | [db-sync-node.ErrorPolicy:Warning:6] [2021-09-12 09:48:46.60 UTC] IP LocalAddress "/node-ipc/node.socket" ErrorPolicySuspendPeer (Just (ConnectionExceptionTrace Network.Socket.connect: <socket: 22>: does not exist (No such file or directory))) 20s 20s
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Warning:64] [2021-09-12 09:48:46.63 UTC] Identity Failed to start all required subscriptions
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:64] [2021-09-12 09:48:47.63 UTC] Identity Restarting Subscription after 1.026610635s desired valency 1 current valency 0
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:64] [2021-09-12 09:48:56.61 UTC] Identity Starting Subscription Worker, valency 1
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:71] [2021-09-12 09:48:56.61 UTC] Identity Connection Attempt Start, destination LocalAddress "/node-ipc/node.socket"
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Error:71] [2021-09-12 09:48:56.61 UTC] Identity Connection Attempt Exception, destination LocalAddress "/node-ipc/node.socket" exception: Network.Socket.connect: <socket: 22>: does not exist (No such file or directory)
cardano-db-sync-extended_1  | [db-sync-node.ErrorPolicy:Warning:6] [2021-09-12 09:48:56.61 UTC] IP LocalAddress "/node-ipc/node.socket" ErrorPolicySuspendPeer (Just (ConnectionExceptionTrace Network.Socket.connect: <socket: 22>: does not exist (No such file or directory))) 20s 20s
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Warning:64] [2021-09-12 09:48:56.64 UTC] Identity Failed to start all required subscriptions
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:64] [2021-09-12 09:48:57.64 UTC] Identity Restarting Subscription after 1.026638047s desired valency 1 current valency 0
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:64] [2021-09-12 09:49:06.62 UTC] Identity Starting Subscription Worker, valency 1
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:73] [2021-09-12 09:49:06.62 UTC] Identity Connection Attempt Start, destination LocalAddress "/node-ipc/node.socket"
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:73] [2021-09-12 09:49:06.62 UTC] Identity Connection Attempt End, destination LocalAddress "/node-ipc/node.socket" outcome: ConnectSuccessLast
cardano-db-sync-extended_1  | [db-sync-node.Handshake:Info:73] [2021-09-12 09:49:06.62 UTC] WithMuxBearer (ConnectionId {localAddress = LocalAddress "", remoteAddress = LocalAddress "/ipc/node.socket"}) Send (ClientAgency TokPropose,MsgProposeVersions (fromList [(NodeToClientV_1,TInt 764824073),(NodeToClientV_2,TInt 764824073),(NodeToClientV_3,TInt 764824073),(NodeToClientV_4,TInt 764824073),(NodeToClientV_5,TInt 764824073),(NodeToClientV_6,TInt 764824073),(NodeToClientV_7,TInt 764824073),(NodeToClientV_8,TInt 764824073),(NodeToClientV_9,TInt 764824073)]))
cardano-db-sync-extended_1  | [db-sync-node.Handshake:Info:73] [2021-09-12 09:49:06.62 UTC] WithMuxBearer (ConnectionId {localAddress = LocalAddress "", remoteAddress = LocalAddress "/ipc/node.socket"}) Recv (ServerAgency TokConfirm,MsgAcceptVersion NodeToClientV_9 (TInt 764824073))
cardano-db-sync-extended_1  | [db-sync-node.Mux:Info:73] [2021-09-12 09:49:06.62 UTC] Bearer on ConnectionId {localAddress = LocalAddress "", remoteAddress = LocalAddress "/ipc/node.socket"} event: Handshake Client end, duration 0.000561375s
cardano-db-sync-extended_1  | [db-sync-node.Mux:Info:74] [2021-09-12 09:49:06.62 UTC] Bearer on ConnectionId {localAddress = LocalAddress "", remoteAddress = LocalAddress "/ipc/node.socket"} event: State: Mature
cardano-db-sync-extended_1  | [db-sync-node:Info:77] [2021-09-12 09:49:06.62 UTC] Starting chainSyncClient
cardano-db-sync-extended_1  | [db-sync-node.Subscription:Notice:64] [2021-09-12 09:49:06.65 UTC] Identity Required subscriptions started
cardano-db-sync-extended_1  | [db-sync-node:Info:77] [2021-09-12 09:49:06.65 UTC] Cardano.Db tip is at slot 39657384, block 6223340
cardano-db-sync-extended_1  | [db-sync-node:Info:82] [2021-09-12 09:49:06.65 UTC] Running DB thread
cardano-db-sync-extended_1  | [db-sync-node:Info:82] [2021-09-12 09:49:06.66 UTC] Rolling back to slot 39484797, hash 86ad35fd0bf316a036dd8794400ead9d4431d6df28eab986864bd5ef65e97f87
cardano-db-sync-extended_1  | [db-sync-node:Info:82] [2021-09-12 09:49:06.69 UTC] Deleting 8476 blocks up to slot 39657384
cardano-db-sync-extended_1  | [db-sync-node:Info:82] [2021-09-12 09:50:14.50 UTC] Blocks deleted
cardano-db-sync-extended_1  | [db-sync-node:Info:82] [2021-09-12 09:50:37.84 UTC] Found snapshot file for slot 39484797, hash 86ad35fd0bf316a036dd8794400ead9d4431d6df28eab986864bd5ef65e97f87
cardano-db-sync-extended_1  | [db-sync-node:Info:85] [2021-09-12 09:50:46.55 UTC] getHistoryInterpreter: acquired
cardano-db-sync-extended_1  | [db-sync-node:Info:82] [2021-09-12 09:50:46.55 UTC] thisIsAnUglyHack: Main thead
cardano-db-sync-extended_1  | [db-sync-node:Info:95] [2021-09-12 09:50:46.55 UTC] Running Offline fetch thread
cardano-db-sync-extended_1  | [db-sync-node:Info:82] [2021-09-12 09:51:10.90 UTC] File /data/39484797-86ad35fd0b-288.lstate exists
cardano-db-sync-extended_1  | [db-sync-node:Info:82] [2021-09-12 09:51:10.90 UTC] Starting at epoch 289
cardano-db-sync-extended_1  | [db-sync-node:Info:82] [2021-09-12 09:51:12.76 UTC] Handling 801247 stakes for epoch 289 slot 39484840, hash faf4ceb70f91e6ed3d1baa584780f6d3e7a29a23bb17287f5bb1283ae5bdf4bb
cardano-db-sync-extended_1  | [db-sync-node:Info:82] [2021-09-12 09:54:16.88 UTC] insertShelleyBlock(Mary): epoch 289, slot 39488197, block 6215000, hash 9daef3159978f9da08cf02f726f68ac01e7d5883cebf96f418d6dcdc9ee2056a
cardano-db-sync-extended_1  | [db-sync-node:Info:82] [2021-09-12 09:56:34.71 UTC] insertEpochInterleaved: Epoch 289, 801247 stake addresses
cardano-db-sync-extended_1  | [db-sync-node:Info:82] [2021-09-12 10:01:55.21 UTC] Offline pool metadata fetch: 51 results, 49 fetch errors
cardano-db-sync-extended_1  | [db-sync-node:Info:82] [2021-09-12 10:08:57.14 UTC] insertShelleyBlock(Mary): epoch 289, slot 39590888, block 6220000, hash 8ad4e3bfad0490812ed62501c2f3872c5cb5d369f063f4f083fd22684ebc13e0
cardano-db-sync-extended_1  | [db-sync-node:Info:82] [2021-09-12 10:08:57.14 UTC] Offline pool metadata fetch: 55 results, 45 fetch errors

@MarcelKlammer We need the contents of at least one of those migration failure files (I suspect they are all mostly the same).
@erikd erikd mentioned this issue Sep 12, 2021
Closed
@MarcelKlammer
Copy link

@erikd Our posts overlapped: #810

@erikd
Copy link
Contributor Author

erikd commented Sep 12, 2021

Closing this in favor of #810 .

@erikd erikd closed this as completed Sep 12, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@erikd @MarcelKlammer