network-history: hangs when loading post Protocol Upgrade

[fkondej]

Problem encountered

The loading process hung when I tried to load all network-history segments on Stagnet 3.

Info about loaded data:

• data from the past nine days
• 925k blocks
• 3092 network history segments
• ~120GB in database on running node
• ~280kk rows (based on the logs)

The network has undergone several Protocol Upgrades, so different versions of vega created the loaded segments.

The load was done by the network's current (latest segment) version: v0.69.0-dev-11688-3b05c262 (commit 3b05c262).

context: https://vegaprotocol.slack.com/archives/CE5L3T60J/p1677244146755909

Observed behaviour

The loading process hung.

Expected behaviour

The loading process should finish successfully and in a timely manner.

Evidence

Logs

023-02-24 14:14:24 decompressing vega-stagnet3-202302151135-887872-888048-historysnapshot.tar.gz
2023-02-24 14:14:24 copying vega-stagnet3-202302151135-887872-888048-historysnapshot into database
2023-02-24 14:14:24 copied 1440 rows from vega-stagnet3-202302151135-887872-888048-historysnapshot into database in 97.491501ms
2023-02-24T14:14:24.546Z        INFO    root    networkhistory/load.go:248      migrating database      {"current database version": 4, "target database version": 5}
2023-02-24 14:14:24 restoring all indexes prior to schema migration
2023-02-24 14:14:24 creating index balances_account_id_vega_time_balance_idx
2023-02-24 14:15:42 creating index balances_vega_time_idx
2023-02-24 14:16:12 creating index blocks_height_idx
2023-02-24 14:16:13 creating index checkpoints_vega_time_idx
2023-02-24 14:16:13 creating index current_liquidity_provisions_party_id_idx
2023-02-24 14:16:13 creating index current_liquidity_provisions_reference_idx
2023-02-24 14:16:13 creating index current_liquidity_provisions_market_id_party_id_idx
2023-02-24 14:16:13 creating index delegations_vega_time_idx
2023-02-24 14:16:13 creating index delegations_party_id_node_id_epoch_id_idx
2023-02-24 14:16:13 creating index delegations_current_node_id_epoch_id_idx
2023-02-24 14:16:13 creating index delegations_current_epoch_id_idx
2023-02-24 14:16:13 creating index deposits_party_id_idx
2023-02-24 14:16:13 creating index deposits_vega_time_idx
2023-02-24 14:16:13 creating index ledger_account_to_id_vega_time_idx
2023-02-24 14:17:28 creating index ledger_type_vega_time_idx
2023-02-24 14:18:39 creating index ledger_account_from_id_vega_time_idx
2023-02-24 14:19:46 creating index liquidity_provisions_vega_time_idx
2023-02-24 14:19:46 creating index margin_levels_vega_time_idx
2023-02-24 14:20:41 creating index market_data_market_vega_time_idx
2023-02-24 14:21:12 creating index markets_vega_time_idx
2023-02-24 14:21:12 creating index idx_oracle_data_matched_spec_ids
2023-02-24 14:21:12 creating index orders_vega_time_idx
2023-02-24 14:22:54 creating index orders_market_id_vega_time_idx
2023-02-24 14:24:45 creating index orders_party_id_vega_time_idx
2023-02-24 14:26:51 creating index orders_reference_vega_time_idx
2023-02-24 14:28:33 creating index orders_id_current_idx
2023-02-24 14:31:26 creating index orders_live_party_id_vega_time_idx
2023-02-24 14:31:26 creating index orders_live_reference_vega_time_idx
2023-02-24 14:31:26 creating index orders_live_id_idx
2023-02-24 14:31:26 creating index orders_live_market_id_vega_time_idx
2023-02-24 14:31:26 creating index positions_vega_time_idx
2023-02-24 14:32:31 creating index positions_current_market_id_idx
2023-02-24 14:32:31 creating index rewards_vega_time_idx
2023-02-24 14:32:32 creating index rewards_epoch_id_idx
2023-02-24 14:32:32 creating index rewards_asset_id_idx
2023-02-24 14:32:32 creating index rewards_party_id_asset_id_idx
2023-02-24 14:32:33 creating index stake_linking_current_party_id_idx
2023-02-24 14:32:33 creating index trades_buyer_synthetic_time_idx
2023-02-24 14:32:55 creating index trades_seller_synthetic_time_idx
2023-02-24 14:33:06 creating index trades_market_id_synthetic_time_idx
2023-02-24 14:33:17 creating index transfers_from_account_id_idx
2023-02-24 14:33:17 creating index transfers_to_account_id_idx
2023-02-24 14:33:17 creating index votes_party_id_idx
2023-02-24 14:33:17 creating index withdrawals_vega_time_idx
2023-02-24 14:33:17 creating index withdrawals_party_id_idx
2023-02-24 14:33:17 restoring all constraints prior to schema migration
2023-02-24 14:41:10 migrating schema to version 5
2023-02-24 14:41:10 removing constraints after schema migration
2023-02-24 14:41:10 dropping all constraints
# Now I am checking this 29 hours later

[guoguojin]

Off the top of my head, I think this might be an issue related to database connection pool. Perhaps something is acquiring connections and not releasing them. I will check to be sure and fix ASAP