Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hack to skip cleanup_dead_slots upon snapshot load #8561

Merged
merged 1 commit into from Mar 2, 2020

Conversation

@ryoqun
Copy link
Member

ryoqun commented Mar 2, 2020

Problem

A validator started from snapshot may create bad snapshots, which cause bank hash mismatch error when consumed by other validators. This is because the validator internally removes too much AccountsStorage and bad snapshots don't include some necessary ones anymore.

Detail of problem

generate_index has been broken for long time about maintaining the correct count of storage's counts, which confuses purge_zero_lamport_accounts() (demonstrating assertion are currently disabled in this pr, real fix #8337 is pending).
This causes too many slots to be incorrectly marked as dead.
However, it hadn't caused problems because cleanup_dead_slots() had been broken too when called immediately after snapshot restoration. That function didn't remove dead_slots at all in that case.
In short, the caller wrongly requested too many slots to remove and the callee wrongly didn't anything. So, there had been a peace...

But it changed since #8148. This PR is needed as one of fixes for the enlarged snapshot issue #8168. And that PR was thought rather harmless..

After #8148, the callee started to remove actually. But things don't break immediately. Snapshots load successfully. And validator just works.
Its rather subtle ramification of unbalanced fix is at the future snapshot creation: Because purge_zero_lamport_accounts() incorrectly removes too many slots as dead in some corner case, newly-created snapshots silently start to omit to include that removed storages backing that slots, causing the deadly BankHashMismatch error because of obvious incorrect view into the accounts set.

Summary of Changes

Until #8337 lands, just disable erroneous too eager removing behavior as an ugly hack...
This PR should have the same effect as reverting #8148, with better merge-conflict-free characteristics. #8337 should revert most of changes in this PR.

Fixes #8560

Backport only to v1.0 because v0.23 was branched before #8148.

@ryoqun ryoqun added the v1.0 label Mar 2, 2020
@ryoqun ryoqun requested review from sakridge and mvines Mar 2, 2020
@mvines

This comment has been minimized.

Copy link
Member

mvines commented Mar 2, 2020

Fixes #8157

Do you mean #8560 :)

@ryoqun

This comment has been minimized.

Copy link
Member Author

ryoqun commented Mar 2, 2020

Fixes #8157
Do you mean #8560 :)

Err, fixed!

@mvines mvines added the CI label Mar 2, 2020
@solana-grimes solana-grimes removed the CI label Mar 2, 2020
@ryoqun

This comment has been minimized.

Copy link
Member Author

ryoqun commented Mar 2, 2020

This PR is first worked on the v1.0 branch. And I confirmed that it fixes the bankhashmismatch for the TdS cluster as follows:

Bad:

$ LD_LIBRARY_PATH=. ROOT_AT_ONCE=1 ../solana-ledger-tool --ledger ledger/ create-snapshot 2197060 newnew
[2020-03-02T13:43:45.705345012Z INFO  solana_ledger::blockstore] Maximum open file descriptors: 65000
[2020-03-02T13:43:45.811575716Z INFO  solana_ledger::blockstore] "/mnt/solana-data/tds-with-1.0/try5/ledger/rocksdb" open took 106ms
[2020-03-02T13:43:45.811609117Z INFO  solana_ledger::bank_forks_utils] Initializing snapshot path: "/mnt/solana-data/tds-with-1.0/try5/ledger/snapshot"
[2020-03-02T13:43:45.812618124Z INFO  solana_ledger::bank_forks_utils] Loading snapshot package: "/mnt/solana-data/tds-with-1.0/try5/ledger/snapshot-2196960-BKVp1Jh8u23QnPoGPdYW82oXoirW65YWejSNYYwPEbWj.tar.bz2"
[2020-03-02T13:46:18.338022518Z INFO  solana_ledger::snapshot_utils] snapshot untar took 152.5s
[2020-03-02T13:46:18.338353221Z INFO  solana_ledger::snapshot_utils] snapshot version: 1.0.0
[2020-03-02T13:46:18.338416021Z INFO  solana_ledger::snapshot_utils] Loading bank from "/mnt/solana-data/tds-with-1.0/try5/ledger/snapshot/.tmpaNkCQy/snapshots/2196960/2196960"
[2020-03-02T13:46:18.346609278Z INFO  solana_ledger::snapshot_utils] Rebuilding accounts...
[2020-03-02T13:46:50.732872144Z INFO  solana_ledger::snapshot_utils] Rebuilding status cache...
[2020-03-02T13:46:50.742541373Z INFO  solana_ledger::snapshot_utils] Loaded bank for slot: 2196960
[2020-03-02T13:46:50.742710095Z INFO  solana_runtime::accounts_db] total_stores: 8358, newest_slot: 2196960, oldest_slot: 0, max_slot: 2127312 (num=9), min_slot: 1157346 (num=1)
[2020-03-02T13:46:50.742780075Z INFO  solana_metrics::metrics] metrics disabled: SOLANA_METRICS_CONFIG: environment variable not found
[2020-03-02T13:46:50.742987107Z INFO  solana_metrics::metrics] datapoint: accounts_db-stores total_count=8358i
[2020-03-02T13:46:50.772892920Z INFO  solana_runtime::accounts_db] scan took 53us merge took 540ns accumulate took 7us
[2020-03-02T13:46:50.772931710Z INFO  solana_runtime::bank] hard fork at bank 2196960
[2020-03-02T13:46:50.772950710Z INFO  solana_runtime::bank] bank frozen: 2196960 hash: EKu56JJW5cgiMufuZAorwTNW33YAeuX3suXsGKvQh3ja accounts_delta: 2M4rDVkGMcPsZspHfXxW1A26kU6d6bf3XRe1hqzBpvco signature_count: 4 last_blockhash: 9xGYFG7WfBK4S9BDtiJbA7UfQ4BVqiWK8nCu7P9hZgbx
[2020-03-02T13:46:50.772981680Z INFO  solana_runtime::bank] accounts hash slot: 2196960 stats: BankHashStats { num_removed_accounts: 2, num_added_accounts: 0, num_lamports_stored: 55429440, total_data_len: 7708, num_executable_accounts: 0 }
[2020-03-02T13:46:50.772990280Z INFO  solana_ledger::snapshot_utils] bank rebuild from snapshot took 32.4s
[2020-03-02T13:46:50.792703161Z INFO  solana_ledger::blockstore_processor] processing ledger from root slot 2196960...
[2020-03-02T13:46:50.875857797Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196961i block_height=1858552i
[2020-03-02T13:46:50.883564650Z INFO  solana_runtime::accounts_db] scan took 95us merge took 300ns accumulate took 7us
[2020-03-02T13:46:50.883597500Z INFO  solana_runtime::bank] bank frozen: 2196961 hash: A9HXzimMyoipVNHpAC4h9ibcGBEzAnEFHnCjVZb5e41X accounts_delta: DSrGH9Z8wx4XouK2y5CrabogdhmkxsVmxE2fbiuRAdor signature_count: 0 last_blockhash: BmDCRmcUHFt8DhN2nTDMNC9FXVYpHbWF3xs5ebbAMQLq
[2020-03-02T13:46:50.883627251Z INFO  solana_runtime::bank] accounts hash slot: 2196961 stats: BankHashStats { num_removed_accounts: 5, num_added_accounts: 0, num_lamports_stored: 5, total_data_len: 152698, num_executable_accounts: 0 }
[2020-03-02T13:46:50.885891779Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196962i block_height=1858553i
[2020-03-02T13:46:50.889390667Z INFO  solana_runtime::accounts_db] scan took 105us merge took 150ns accumulate took 6us
[2020-03-02T13:46:50.889420287Z INFO  solana_runtime::bank] bank frozen: 2196962 hash: 9hvAvjLjW1pmB7tqzoEsV44xS7d2Zvmdwjv7WGAYAEkB accounts_delta: CmH8NvizArg32GgNy67PNcHEyT8UHSM6MrscTn7FguGo signature_count: 0 last_blockhash: 5rYTG65GCWMS3s7pwFjy5o4WinqucQQ65DATjx4b2eV4
[2020-03-02T13:46:50.889448307Z INFO  solana_runtime::bank] accounts hash slot: 2196962 stats: BankHashStats { num_removed_accounts: 5, num_added_accounts: 0, num_lamports_stored: 5, total_data_len: 152698, num_executable_accounts: 0 }
[2020-03-02T13:46:50.890341874Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196963i block_height=1858554i
[2020-03-02T13:46:50.899727021Z INFO  solana_metrics::metrics] metrics disabled: SOLANA_METRICS_CONFIG: environment variable not found
[2020-03-02T13:46:50.914025167Z INFO  solana_runtime::accounts_db] scan took 115us merge took 21us accumulate took 231us
[2020-03-02T13:46:50.914062187Z INFO  solana_runtime::bank] bank frozen: 2196963 hash: CPT1oc6khN8ZP7bVG3DCU2TQYLcv5wzudJ7QYRnTqZVW accounts_delta: CVkTZJHyiWmHdeTDQsh6Z8zGsHqUb3nLPd8MPoMrRhWM signature_count: 98 last_blockhash: F9LamkFWQyaCZkgMbDRA3bRHbaVJTqwBNoHxp2kK5o8E
[2020-03-02T13:46:50.914095978Z INFO  solana_runtime::bank] accounts hash slot: 2196963 stats: BankHashStats { num_removed_accounts: 103, num_added_accounts: 0, num_lamports_stored: 5816864488496, total_data_len: 341544, num_executable_accounts: 0 }
[2020-03-02T13:46:50.915104846Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196964i block_height=1858555i
[2020-03-02T13:46:50.970189774Z INFO  solana_runtime::accounts_db] scan took 223us merge took 30us accumulate took 401us
[2020-03-02T13:46:50.970225274Z INFO  solana_runtime::bank] bank frozen: 2196964 hash: Cxxk485kPxTtjWZumBPoaDf7tsF5CoMJ4XMVCrrp5my4 accounts_delta: 45tMhGEGACG88jj8A78GF11xQVBUoEBYR4Af3hESWv1U signature_count: 176 last_blockhash: 4m1k6UAoFk67RNqMYMmByvsT9d7XryDc1w2qjDLNyXDZ
[2020-03-02T13:46:50.970252794Z INFO  solana_runtime::bank] accounts hash slot: 2196964 stats: BankHashStats { num_removed_accounts: 165, num_added_accounts: 0, num_lamports_stored: 8463120937743, total_data_len: 430186, num_executable_accounts: 0 }
[2020-03-02T13:46:50.971299842Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196965i block_height=1858556i
[2020-03-02T13:46:50.987033860Z INFO  solana_runtime::accounts_db] scan took 133us merge took 24us accumulate took 210us
[2020-03-02T13:46:50.987071010Z INFO  solana_runtime::bank] bank frozen: 2196965 hash: 5eNySJmPxpLf4rgyt3kHypxo43MnYiSKUU7qgWJdZ4VK accounts_delta: CPVZpUTzwmCE8uJzrBWHhdgeNKR21XhWmDpktQB9GRok signature_count: 74 last_blockhash: E7UVwcTWFxZiQ8S1m4XF5pu2QSARsUZ4fDyR4etSdtBk
[2020-03-02T13:46:50.987101571Z INFO  solana_runtime::bank] accounts hash slot: 2196965 stats: BankHashStats { num_removed_accounts: 75, num_added_accounts: 0, num_lamports_stored: 3636499297344, total_data_len: 279880, num_executable_accounts: 0 }
[2020-03-02T13:46:50.988162389Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196966i block_height=1858557i
[2020-03-02T13:46:51.006199527Z INFO  solana_runtime::accounts_db] scan took 131us merge took 27us accumulate took 169us
[2020-03-02T13:46:51.006234646Z INFO  solana_runtime::bank] bank frozen: 2196966 hash: 2rTdQCMXNHED4vDrqbkJXAwmdSuamDDv8uE4HDqr7Ymx accounts_delta: AgpYNvxMyERDHbRpPB5HWTDKxmgSfmFSKLUggtGnB4VR signature_count: 86 last_blockhash: A5Z32SHXruPh1zwAYvchBNXvyqJS7C9N866MQ1RFk1mz
[2020-03-02T13:46:51.006265667Z INFO  solana_runtime::bank] accounts hash slot: 2196966 stats: BankHashStats { num_removed_accounts: 90, num_added_accounts: 0, num_lamports_stored: 3371546611186, total_data_len: 314566, num_executable_accounts: 0 }
[2020-03-02T13:46:51.007344576Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196967i block_height=1858558i
[2020-03-02T13:46:51.024648336Z INFO  solana_runtime::accounts_db] scan took 121us merge took 24us accumulate took 222us
[2020-03-02T13:46:51.024685026Z INFO  solana_runtime::bank] bank frozen: 2196967 hash: 9pwdFEGUcyppPWvhbLiiVWH6M1tCedUnKE1d1Ncag2W8 accounts_delta: G2ZWgaZLtxdCpFsPXRjpFFa2yTvAq1KS6urBSGqbBb27 signature_count: 94 last_blockhash: FggJXeoXit5sU7VbFKur2yNjUjnNV8kZ9BSWAydo4xyN
[2020-03-02T13:46:51.024717557Z INFO  solana_runtime::bank] accounts hash slot: 2196967 stats: BankHashStats { num_removed_accounts: 97, num_added_accounts: 0, num_lamports_stored: 4165610156231, total_data_len: 326128, num_executable_accounts: 0 }
[2020-03-02T13:46:51.025768465Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196968i block_height=1858559i
[2020-03-02T13:46:51.065062745Z INFO  solana_runtime::accounts_db] scan took 137us merge took 20us accumulate took 278us
[2020-03-02T13:46:51.065089325Z INFO  solana_runtime::bank] bank frozen: 2196968 hash: Bqiz6gJFQ5mQiXd8mKkZwZSuy29Bh2trDjJnY9y5YiVk accounts_delta: GPPVyYfjvBXpaipRAZqBo9J2fb6Hhkd517ux4FJxZgaK signature_count: 270 last_blockhash: E2zDUHNCL7Wy4BKHhskrgYogYhBVmLtBf85JsXzxETNw
[2020-03-02T13:46:51.065110806Z INFO  solana_runtime::bank] accounts hash slot: 2196968 stats: BankHashStats { num_removed_accounts: 253, num_added_accounts: 0, num_lamports_stored: 13483662649264, total_data_len: 588200, num_executable_accounts: 0 }
[2020-03-02T13:46:51.066111743Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196969i block_height=1858560i
[2020-03-02T13:46:51.073881557Z INFO  solana_runtime::accounts_db] scan took 107us merge took 9us accumulate took 133us
[2020-03-02T13:46:51.073917367Z INFO  solana_runtime::bank] bank frozen: 2196969 hash: 8gMM8riA6jcsRE8acM2PjcZimJgJ4XQmoc9Mb6XzRhZF accounts_delta: 5otiXDKyA3dADZixRRHUx5JET88jXVRRkwJJkew83A9 signature_count: 20 last_blockhash: HknEZ3h24MWk14fW1pyYaWdR4m5LpGZLtsuR2wsJVrcL
[2020-03-02T13:46:51.073948357Z INFO  solana_runtime::bank] accounts hash slot: 2196969 stats: BankHashStats { num_removed_accounts: 24, num_added_accounts: 0, num_lamports_stored: 543962485587, total_data_len: 187384, num_executable_accounts: 0 }
[2020-03-02T13:46:51.074977875Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196970i block_height=1858561i
[2020-03-02T13:46:51.086615240Z INFO  solana_runtime::accounts_db] scan took 122us merge took 26us accumulate took 271us
[2020-03-02T13:46:51.086647180Z INFO  solana_runtime::bank] bank frozen: 2196970 hash: DcPAeJRmfCVEr1WNkrs6zTXuyhxvwEo7gzaJ1hRSMR4X accounts_delta: 2LSwf8PHftx2bJMyUAMU8xXX39CZS1bqUb16ySthAf7s signature_count: 90 last_blockhash: F9JSksL45iJSefoVhSaGrj68q1D4RQ7cQzqEFvRda8pc
[2020-03-02T13:46:51.086682500Z INFO  solana_runtime::bank] accounts hash slot: 2196970 stats: BankHashStats { num_removed_accounts: 95, num_added_accounts: 0, num_lamports_stored: 4317846573484, total_data_len: 326128, num_executable_accounts: 0 }
[2020-03-02T13:46:51.087663129Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196971i block_height=1858562i
[2020-03-02T13:46:51.115377394Z INFO  solana_runtime::accounts_db] scan took 143us merge took 8us accumulate took 187us
[2020-03-02T13:46:51.115416764Z INFO  solana_runtime::bank] bank frozen: 2196971 hash: BUBQypHUBj3JdHTDBhr9vynSN44mrmKFC7CmDV82KJ6M accounts_delta: J4Yo9AR2cT6bf9BNzGgQZkQxKoWwJkY3P7NWa8LYFcsf signature_count: 152 last_blockhash: 811Bkq6NA3j7qmsxRHhic7Wbwv1eC78fvaXY31PPwrri
[2020-03-02T13:46:51.115449595Z INFO  solana_runtime::bank] accounts hash slot: 2196971 stats: BankHashStats { num_removed_accounts: 155, num_added_accounts: 0, num_lamports_stored: 6748570957219, total_data_len: 437894, num_executable_accounts: 0 }
[2020-03-02T13:46:51.116523763Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196972i block_height=1858563i
[2020-03-02T13:46:51.142853907Z INFO  solana_runtime::accounts_db] scan took 141us merge took 28us accumulate took 194us
[2020-03-02T13:46:51.142892207Z INFO  solana_runtime::bank] bank frozen: 2196972 hash: 2BnraaRi4FjzbymGxQ5Rm5P6FVhexq1RVokCENo2k4hV accounts_delta: 2vsAzDdPNeYa5z6riQUoL7XwUQg19ctSghuPnQzAsyHh signature_count: 158 last_blockhash: GDSZW88WppK5asiyspSWgKe14SCNNAei4WRS3EhA1KMU
[2020-03-02T13:46:51.142925388Z INFO  solana_runtime::bank] accounts hash slot: 2196972 stats: BankHashStats { num_removed_accounts: 158, num_added_accounts: 0, num_lamports_stored: 8616171467346, total_data_len: 437894, num_executable_accounts: 0 }
[2020-03-02T13:46:51.144055107Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196973i block_height=1858564i
[2020-03-02T13:46:51.151233415Z INFO  solana_runtime::accounts_db] scan took 108us merge took 4us accumulate took 8us
[2020-03-02T13:46:51.151267935Z INFO  solana_runtime::bank] bank frozen: 2196973 hash: G9ojeqnCcMVGxg2oydyf3CTYvxZZ84gPriE6TQRjMroj accounts_delta: FidRqEYMSqNSx3A6iwdMc21Nyzakb2157t3sdUck47c3 signature_count: 6 last_blockhash: Bid3hgEHwZVRjyfpbp9JWMD83goAH58cgLENWsideEsE
[2020-03-02T13:46:51.151293336Z INFO  solana_runtime::bank] accounts hash slot: 2196973 stats: BankHashStats { num_removed_accounts: 11, num_added_accounts: 0, num_lamports_stored: 11995100165, total_data_len: 164260, num_executable_accounts: 0 }
[2020-03-02T13:46:51.152358234Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196974i block_height=1858565i
[2020-03-02T13:46:51.157456756Z INFO  solana_runtime::accounts_db] scan took 101us merge took 180ns accumulate took 10us
[2020-03-02T13:46:51.157487126Z INFO  solana_runtime::bank] bank frozen: 2196974 hash: 7yqxa92NPLgCmiDCwHKSVtu3vAvLi8VWwXTaRpyWYLii accounts_delta: EwnuSP7a8Gh8RJ1DmgY5ieXmmJDvxQaDuMG6PpwiVKY5 signature_count: 10 last_blockhash: 9QSGg4WgsAGKDtA2d5G1yRhSRDx2L7ziM7KPzhSUGFPs
[2020-03-02T13:46:51.157513727Z INFO  solana_runtime::bank] accounts hash slot: 2196974 stats: BankHashStats { num_removed_accounts: 14, num_added_accounts: 0, num_lamports_stored: 274962485599, total_data_len: 168114, num_executable_accounts: 0 }
[2020-03-02T13:46:51.158533495Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196975i block_height=1858566i
[2020-03-02T13:46:51.170966075Z INFO  solana_runtime::accounts_db] scan took 119us merge took 8us accumulate took 154us
[2020-03-02T13:46:51.171000396Z INFO  solana_runtime::bank] bank frozen: 2196975 hash: 38PkpL28feUgd3Cbdjk13X4pckyYuBHeRcmUx3mVtEyW accounts_delta: 5zfKcDaNfqEYo83gaLVAWz8F7zfapnrixK5AoMKjTyBe signature_count: 54 last_blockhash: 8kFSR3ydBwqDjaqVmQHYgdrUD7hqFtz5gbD1WzwHDxyp
[2020-03-02T13:46:51.171035636Z INFO  solana_runtime::bank] accounts hash slot: 2196975 stats: BankHashStats { num_removed_accounts: 54, num_added_accounts: 0, num_lamports_stored: 673812427777, total_data_len: 237486, num_executable_accounts: 0 }
[2020-03-02T13:46:51.171846783Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196976i block_height=1858567i
[2020-03-02T13:46:51.219594961Z INFO  solana_runtime::accounts_db] scan took 149us merge took 29us accumulate took 226us
[2020-03-02T13:46:51.219634011Z INFO  solana_runtime::bank] bank frozen: 2196976 hash: AUhoVhKgMcsqnHv5wcBukRLCWEdPYQfwNvZyD5q7TM3D accounts_delta: 3FAJ3gQwJFsUioLq4aZaA3z75k7en9jYKuS6uW19V1vx signature_count: 318 last_blockhash: 6gdp5sTreo73J11N4jZFQU7TxmmduH8Y32AWeC3mF1Xn
[2020-03-02T13:46:51.219667182Z INFO  solana_runtime::bank] accounts hash slot: 2196976 stats: BankHashStats { num_removed_accounts: 296, num_added_accounts: 0, num_lamports_stored: 17631241875235, total_data_len: 661426, num_executable_accounts: 0 }
[2020-03-02T13:46:51.220696050Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196977i block_height=1858568i
[2020-03-02T13:46:51.233116741Z INFO  solana_runtime::accounts_db] scan took 125us merge took 250ns accumulate took 270us
[2020-03-02T13:46:51.233153671Z INFO  solana_runtime::bank] bank frozen: 2196977 hash: FhcBR1UEQRWNphmS8wso35qqPvddXS7LYcR3dC7j7TrY accounts_delta: G9GaEASZiYUm8Vpr42Er5pY1MPZevHXNu8RmxufaVdGM signature_count: 64 last_blockhash: 6soYEgCgrFgcVT5GZCnTVJNquvyEc2m13AJwC8r9ivdN
[2020-03-02T13:46:51.233182081Z INFO  solana_runtime::bank] accounts hash slot: 2196977 stats: BankHashStats { num_removed_accounts: 69, num_added_accounts: 0, num_lamports_stored: 2350787928790, total_data_len: 276026, num_executable_accounts: 0 }
[2020-03-02T13:46:51.234176090Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196978i block_height=1858569i
[2020-03-02T13:46:51.252897552Z INFO  solana_runtime::accounts_db] scan took 128us merge took 29us accumulate took 279us
[2020-03-02T13:46:51.252937382Z INFO  solana_runtime::bank] bank frozen: 2196978 hash: 9Ha6MXpv1784aoMqLFV1AXMNbTgNSrM3GFZtMWLar2kk accounts_delta: FgousJRUciSNk1nh8bPkT7RUkggK9avakogsyc2YqUis signature_count: 76 last_blockhash: 464UAoorQhUFKAsNgZDX3Gy5z5v6Wrcv1c5n48qqD8Zs
[2020-03-02T13:46:51.252964902Z INFO  solana_runtime::bank] accounts hash slot: 2196978 stats: BankHashStats { num_removed_accounts: 80, num_added_accounts: 0, num_lamports_stored: 4292644455071, total_data_len: 295296, num_executable_accounts: 0 }
[2020-03-02T13:46:51.253996701Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196979i block_height=1858570i
[2020-03-02T13:46:51.272245789Z INFO  solana_runtime::accounts_db] scan took 137us merge took 70us accumulate took 284us
[2020-03-02T13:46:51.272278230Z INFO  solana_runtime::bank] bank frozen: 2196979 hash: BxeXxRDomkrLvvp2sAMvozvjmqrUEyxCTeacvAGbAUF4 accounts_delta: 34ocJytyHtB5yssHBZvwqjZQPzvzUaQpfMwWfxuEwZkY signature_count: 128 last_blockhash: 4Qf1Ns1thymUz4qz1LthpMbTvV5ZUCZ8QYLdcB1SNB7H
[2020-03-02T13:46:51.272312969Z INFO  solana_runtime::bank] accounts hash slot: 2196979 stats: BankHashStats { num_removed_accounts: 133, num_added_accounts: 0, num_lamports_stored: 6278621487050, total_data_len: 399354, num_executable_accounts: 0 }
[2020-03-02T13:46:51.273309077Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196980i block_height=1858571i
[2020-03-02T13:46:51.287185860Z INFO  solana_metrics::counter] COUNTER:{"name": "vote-native", "counts": 1001, "samples": 1000,  "now": 1583156811287, "events": 1}
[2020-03-02T13:46:51.293864214Z INFO  solana_runtime::accounts_db] scan took 137us merge took 40us accumulate took 214us
[2020-03-02T13:46:51.293900545Z INFO  solana_runtime::bank] bank frozen: 2196980 hash: D3v4ptHjT5TZLsXVGV7mrcDUNT9MgEKhPckWUAvNtzDs accounts_delta: YqDD4Eh17FgEQkoYSpVN2zxG4PZByr5rbHdAq5d7i3M signature_count: 160 last_blockhash: Hpkw9QywjrxERRYECKiFGjdQb6cJNR3UffmVnNTT9dwY
[2020-03-02T13:46:51.293927615Z INFO  solana_runtime::bank] accounts hash slot: 2196980 stats: BankHashStats { num_removed_accounts: 163, num_added_accounts: 0, num_lamports_stored: 7452482913203, total_data_len: 453310, num_executable_accounts: 0 }
[2020-03-02T13:46:51.294984854Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196981i block_height=1858572i
[2020-03-02T13:46:51.304027657Z INFO  solana_runtime::accounts_db] scan took 118us merge took 300ns accumulate took 203us
[2020-03-02T13:46:51.304060797Z INFO  solana_runtime::bank] bank frozen: 2196981 hash: 2ssbZmpHemAg6Kxak1ZtpNZREYQSGaTdx3gB6gatAmwx accounts_delta: 9dxTnsSmkgAotg6EYpwVuDPYbbmfG5gzReiNxytrXxBB signature_count: 40 last_blockhash: DnHA9PefYWejormELVubK4p9NGWqwQpMGtx38gZ86trX
[2020-03-02T13:46:51.304082998Z INFO  solana_runtime::bank] accounts hash slot: 2196981 stats: BankHashStats { num_removed_accounts: 45, num_added_accounts: 0, num_lamports_stored: 1490928186470, total_data_len: 229778, num_executable_accounts: 0 }
[2020-03-02T13:46:51.305095176Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196982i block_height=1858573i
[2020-03-02T13:46:51.319403332Z INFO  solana_runtime::accounts_db] scan took 134us merge took 24us accumulate took 346us
[2020-03-02T13:46:51.319435403Z INFO  solana_runtime::bank] bank frozen: 2196982 hash: 5XZbrGn2rWnMBbE1uqtBPBX9rG7R5ZR573edHmLoiwBR accounts_delta: Bm1aBRfzvZc4SXRFN7q9sLaqNHdaAAa9F6pBKVdRV6Ww signature_count: 72 last_blockhash: 6tKfQwT1NL29HSRAhZdEY81QGvE7tNYQvWv3w6d7qqRo
[2020-03-02T13:46:51.319466643Z INFO  solana_runtime::bank] accounts hash slot: 2196982 stats: BankHashStats { num_removed_accounts: 77, num_added_accounts: 0, num_lamports_stored: 4001755314207, total_data_len: 291442, num_executable_accounts: 0 }
[2020-03-02T13:46:51.320495671Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196983i block_height=1858574i
[2020-03-02T13:46:51.333327755Z INFO  solana_runtime::accounts_db] scan took 124us merge took 14us accumulate took 264us
[2020-03-02T13:46:51.333360276Z INFO  solana_runtime::bank] bank frozen: 2196983 hash: F1UqRxiPyTjKUWpkeufq6313ZcM2XpLafgFotEqvU57C accounts_delta: 3zsdNkirUPawGTKRndfsZ1bSbKjeYtf8kNRk15SNutVW signature_count: 90 last_blockhash: 3siecsDFZC39pR8ureqsjMgrRSypZJZH88pAfQJrmwBN
[2020-03-02T13:46:51.333386766Z INFO  solana_runtime::bank] accounts hash slot: 2196983 stats: BankHashStats { num_removed_accounts: 95, num_added_accounts: 0, num_lamports_stored: 4335846573486, total_data_len: 326128, num_executable_accounts: 0 }
[2020-03-02T13:46:51.334425485Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196984i block_height=1858575i
[2020-03-02T13:46:51.364797252Z INFO  solana_runtime::accounts_db] scan took 148us merge took 24us accumulate took 181us
[2020-03-02T13:46:51.364835372Z INFO  solana_runtime::bank] bank frozen: 2196984 hash: 8EcGriUcj3bXwbMWrjdw6GACCWXoYMibX9PQ9oot9xbK accounts_delta: GUpRCFCrMERTzgTUxUzKQWMvadzMXRCuMqnhPaTEyPW5 signature_count: 208 last_blockhash: dptWqU3NoGde51y18qRpMqsVKpRfLe3b9Q11W1TEZzV
[2020-03-02T13:46:51.364876771Z INFO  solana_runtime::bank] accounts hash slot: 2196984 stats: BankHashStats { num_removed_accounts: 193, num_added_accounts: 0, num_lamports_stored: 9907688832859, total_data_len: 476434, num_executable_accounts: 0 }
[2020-03-02T13:46:51.365969831Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196985i block_height=1858576i
[2020-03-02T13:46:51.370833910Z INFO  solana_runtime::accounts_db] scan took 73us merge took 170ns accumulate took 8us
[2020-03-02T13:46:51.370864070Z INFO  solana_runtime::bank] bank frozen: 2196985 hash: 9JuFfrkoo4gWMLcogrNP3QuQhy5ehbAijvMVjYb3q4Vx accounts_delta: 2uVtNh2vYVNhkvhjYUN8oKQCmgapvLfTJ3V46wQTAthJ signature_count: 6 last_blockhash: CrmMYW46g7d1YGA6JbiRabkp4svmx8hRh9E4LVHzGQ6B
[2020-03-02T13:46:51.370897811Z INFO  solana_runtime::bank] accounts hash slot: 2196985 stats: BankHashStats { num_removed_accounts: 10, num_added_accounts: 0, num_lamports_stored: 989967385431, total_data_len: 160406, num_executable_accounts: 0 }
[2020-03-02T13:46:51.371939280Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196986i block_height=1858577i
[2020-03-02T13:46:51.379459301Z INFO  solana_runtime::accounts_db] scan took 119us merge took 13us accumulate took 168us
[2020-03-02T13:46:51.379491901Z INFO  solana_runtime::bank] bank frozen: 2196986 hash: 7sXDM9tBtqQhJregvYmcAeu2mDV1V25SkV9vJcuf8kYf accounts_delta: J5ipoZXwL6UCphNYNEEGoWAfnCNHirE4xZQTJU6BdJJZ signature_count: 28 last_blockhash: GUY48zmkBir6BkKMMYpyfVLh8kYzh42hB541ap5Gwr4Y
[2020-03-02T13:46:51.379522981Z INFO  solana_runtime::bank] accounts hash slot: 2196986 stats: BankHashStats { num_removed_accounts: 33, num_added_accounts: 0, num_lamports_stored: 1701970600940, total_data_len: 206654, num_executable_accounts: 0 }
[2020-03-02T13:46:51.380570940Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196987i block_height=1858578i
[2020-03-02T13:46:51.401307968Z INFO  solana_runtime::accounts_db] scan took 121us merge took 27us accumulate took 175us
[2020-03-02T13:46:51.401438389Z INFO  solana_runtime::bank] bank frozen: 2196987 hash: wn6At4Ev3dNHKvfpXe5zDkTfFnWEh9dwWvdwxdZstzm accounts_delta: C7G2eLKF6Ko3okp7C9xpA5G7nXPEB14kMfU1t869btQi signature_count: 118 last_blockhash: 78z9QYBYb1KLcW9p28Ub4twFQ4vsqAcPQ78vQQxnBQN1
[2020-03-02T13:46:51.401458160Z INFO  solana_runtime::bank] accounts hash slot: 2196987 stats: BankHashStats { num_removed_accounts: 121, num_added_accounts: 0, num_lamports_stored: 4681721014863, total_data_len: 372376, num_executable_accounts: 0 }
[2020-03-02T13:46:51.402271566Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196988i block_height=1858579i
[2020-03-02T13:46:51.438459990Z INFO  solana_runtime::accounts_db] scan took 132us merge took 41us accumulate took 255us
[2020-03-02T13:46:51.438494951Z INFO  solana_runtime::bank] bank frozen: 2196988 hash: FwuKtLyPu9D8A9bLF9moJ2Uajwd2r1YeEb3ExyDt91Y7 accounts_delta: 5E2FpKbTn1xcVA6MHdr9o4riXFDk7KRxAm8eYV1p8c4n signature_count: 238 last_blockhash: FV4uitNDGsHqWEFjLuf5i4UddMkGRqFFfbkGLX5WV99v
[2020-03-02T13:46:51.438523240Z INFO  solana_runtime::bank] accounts hash slot: 2196988 stats: BankHashStats { num_removed_accounts: 239, num_added_accounts: 0, num_lamports_stored: 12702073470258, total_data_len: 595908, num_executable_accounts: 0 }
[2020-03-02T13:46:51.439564789Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196989i block_height=1858580i
[2020-03-02T13:46:51.449019446Z INFO  solana_runtime::accounts_db] scan took 114us merge took 630ns accumulate took 151us
[2020-03-02T13:46:51.449058856Z INFO  solana_runtime::bank] bank frozen: 2196989 hash: H9X4G7oSr2fUHN4BxUCvbDkdX2Q1UwrPsTV9Zpu83PGP accounts_delta: CgYquuHxTsYYd4hPgbrqhLPiHcmKqHcU9xw5xGaH32gT signature_count: 40 last_blockhash: XLVtG3KX4VExh7VjvhNNwLAjkr64J2PSEBMWTFrs23X
[2020-03-02T13:46:51.449083606Z INFO  solana_runtime::bank] accounts hash slot: 2196989 stats: BankHashStats { num_removed_accounts: 44, num_added_accounts: 0, num_lamports_stored: 275766798205, total_data_len: 225924, num_executable_accounts: 0 }
[2020-03-02T13:46:51.450196726Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196990i block_height=1858581i
[2020-03-02T13:46:51.468511324Z INFO  solana_runtime::accounts_db] scan took 174us merge took 1us accumulate took 175us
[2020-03-02T13:46:51.468550565Z INFO  solana_runtime::bank] bank frozen: 2196990 hash: G6BehQHWEi8AGSyAWe9MiRkF53WVVcj72qGkyfMbh4X6 accounts_delta: 7jC581DCqwnKRfeEqZA12rxdx1EM1BjoSQL1GQa87Mz5 signature_count: 116 last_blockhash: 3eAA8g3op5Prpu3fNKtu9vGchv5tZ6ZGo1P2zyoRcyUT
[2020-03-02T13:46:51.468598915Z INFO  solana_runtime::bank] accounts hash slot: 2196990 stats: BankHashStats { num_removed_accounts: 121, num_added_accounts: 0, num_lamports_stored: 6316663901307, total_data_len: 376230, num_executable_accounts: 0 }
[2020-03-02T13:46:51.469667094Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196991i block_height=1858582i
[2020-03-02T13:46:51.481682102Z INFO  solana_runtime::accounts_db] scan took 149us merge took 24us accumulate took 215us
[2020-03-02T13:46:51.481717332Z INFO  solana_runtime::bank] bank frozen: 2196991 hash: 4CYH4N5sCLWDxfVZVGjtFCYZyZGfP7U3C2417WKMejtx accounts_delta: 23pWhVjQFLP7yfaHPAebDiSu7B1yAUvn9QxAoWtE7eTW signature_count: 74 last_blockhash: 6LRRFqAERPGjYZ163cCkx5mDSxireaVEoTN9R6oW7Vcc
[2020-03-02T13:46:51.481739352Z INFO  solana_runtime::bank] accounts hash slot: 2196991 stats: BankHashStats { num_removed_accounts: 79, num_added_accounts: 0, num_lamports_stored: 3918760213797, total_data_len: 295296, num_executable_accounts: 0 }
[2020-03-02T13:46:51.482789501Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196992i block_height=1858583i
[2020-03-02T13:46:51.511741037Z INFO  solana_runtime::accounts_db] scan took 140us merge took 38us accumulate took 280us
[2020-03-02T13:46:51.511776976Z INFO  solana_runtime::bank] bank frozen: 2196992 hash: DGofYZgj5VXyvuct9vCk6UsNiT7CSgH2uXrUy5dQxaF8 accounts_delta: 5GKKZ7QVaG5PhmXBhKQM1UuRFw8bgsHGuaGHytPZKMSt signature_count: 192 last_blockhash: 9j5uNz6YXy9NMbUTm1DN1DNMBhDKBXfWkktHHgFD6YaM
[2020-03-02T13:46:51.511804956Z INFO  solana_runtime::bank] accounts hash slot: 2196992 stats: BankHashStats { num_removed_accounts: 181, num_added_accounts: 0, num_lamports_stored: 8842962764567, total_data_len: 461018, num_executable_accounts: 0 }
[2020-03-02T13:46:51.512694963Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196993i block_height=1858584i
[2020-03-02T13:46:51.518546461Z INFO  solana_runtime::accounts_db] scan took 111us merge took 280ns accumulate took 11us
[2020-03-02T13:46:51.518580151Z INFO  solana_runtime::bank] bank frozen: 2196993 hash: Etyz6VoEreB6oss4cbXpFwGQrABkKAC8gMVnJAPGdLzS accounts_delta: BAwBtYKTqPynbRfcRDoJjcS6ecgk45bn5MFCLV37WLFK signature_count: 10 last_blockhash: 287nSUckKgse4md57rWUpHShK2SDkhTKXR8P5V3dbzUe
[2020-03-02T13:46:51.518595322Z INFO  solana_runtime::bank] accounts hash slot: 2196993 stats: BankHashStats { num_removed_accounts: 15, num_added_accounts: 0, num_lamports_stored: 514990200319, total_data_len: 171968, num_executable_accounts: 0 }
[2020-03-02T13:46:51.519583609Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196994i block_height=1858585i
[2020-03-02T13:46:51.532816477Z INFO  solana_runtime::accounts_db] scan took 126us merge took 29us accumulate took 327us
[2020-03-02T13:46:51.532856487Z INFO  solana_runtime::bank] bank frozen: 2196994 hash: CQjBKcKRDci6iabNcrmbNWkTdqW7JNSXy7pRT4jfT7fb accounts_delta: E8jY9jDhkEviu4vBWSHsgCaNA3HhvdZ7KPWE73ozVeNR signature_count: 74 last_blockhash: 5aXBDEMCUwBw27w3wF5qtYyiee5tXLEsdTRqk5g6HSuZ
[2020-03-02T13:46:51.532890178Z INFO  solana_runtime::bank] accounts hash slot: 2196994 stats: BankHashStats { num_removed_accounts: 79, num_added_accounts: 0, num_lamports_stored: 3282866172854, total_data_len: 295296, num_executable_accounts: 0 }
[2020-03-02T13:46:51.533938646Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196995i block_height=1858586i
[2020-03-02T13:46:51.550537971Z INFO  solana_runtime::accounts_db] scan took 120us merge took 34us accumulate took 174us
[2020-03-02T13:46:51.550562961Z INFO  solana_runtime::bank] bank frozen: 2196995 hash: 8noFMDHr3HnutTMkTgu7fbsU4CeBN5TyYLkAhJSrc47B accounts_delta: CGBWxXV8Y98Zg4HStAnE4igpNLu6DAaUjQBwozQctFFg signature_count: 94 last_blockhash: EYD5E3GrjGSkaTdANwRWdXD1h2JGbTQUHAWsgj4rYXhT
[2020-03-02T13:46:51.550590552Z INFO  solana_runtime::bank] accounts hash slot: 2196995 stats: BankHashStats { num_removed_accounts: 99, num_added_accounts: 0, num_lamports_stored: 4431693300372, total_data_len: 333836, num_executable_accounts: 0 }
[2020-03-02T13:46:51.551407648Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196996i block_height=1858587i
[2020-03-02T13:46:51.588590100Z INFO  solana_runtime::accounts_db] scan took 162us merge took 38us accumulate took 366us
[2020-03-02T13:46:51.588626971Z INFO  solana_runtime::bank] bank frozen: 2196996 hash: NyPmifAkHbmPWUW6GhBfMqbagQdLST95ScR79ZNHT72 accounts_delta: 5P5ayaeKNmYzVaQzJ7GtumFNdwpdVCyCLK1oJrwozuC9 signature_count: 226 last_blockhash: FjV88M5MR1gV4XKQtQ5KVEDMuWtWNugSdts9AFvsqb5Z
[2020-03-02T13:46:51.588662131Z INFO  solana_runtime::bank] accounts hash slot: 2196996 stats: BankHashStats { num_removed_accounts: 216, num_added_accounts: 0, num_lamports_stored: 11984807807069, total_data_len: 530390, num_executable_accounts: 0 }
[2020-03-02T13:46:51.589705419Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196997i block_height=1858588i
[2020-03-02T13:46:51.598718553Z INFO  solana_runtime::accounts_db] scan took 91us merge took 17us accumulate took 163us
[2020-03-02T13:46:51.598753603Z INFO  solana_runtime::bank] bank frozen: 2196997 hash: AqdiBvLMGiqSdknh2vABgZtDM7n7cGfqtEUZn3hB3Xd8 accounts_delta: F6JTqfM16EFZbpjdg734ZLsUq9twrtUxjFbsScKrHLdH signature_count: 42 last_blockhash: Bvn1ZZRbp4MMeQrb5jWnZna4ceYwtN2sjmGvZi4nWTQJ
[2020-03-02T13:46:51.598783324Z INFO  solana_runtime::bank] accounts hash slot: 2196997 stats: BankHashStats { num_removed_accounts: 47, num_added_accounts: 0, num_lamports_stored: 1921960801266, total_data_len: 233632, num_executable_accounts: 0 }
[2020-03-02T13:46:51.599826692Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196998i block_height=1858589i
[2020-03-02T13:46:51.613943867Z INFO  solana_runtime::accounts_db] scan took 151us merge took 24us accumulate took 277us
[2020-03-02T13:46:51.613982347Z INFO  solana_runtime::bank] bank frozen: 2196998 hash: 2ePphHQmksn1y4wj2raaaqwvf3fi93Tejv43vcxNiAs7 accounts_delta: 8LW8DZY6UBRXYAfp3BcLD7CwjUKyGM6HgpTnXSjg6E2A signature_count: 76 last_blockhash: Dtsnepmr8iBUbznJ45C38LfbkQsTSGJkERKJiF6SbxQX
[2020-03-02T13:46:51.614018207Z INFO  solana_runtime::bank] accounts hash slot: 2196998 stats: BankHashStats { num_removed_accounts: 81, num_added_accounts: 0, num_lamports_stored: 2654717799567, total_data_len: 299150, num_executable_accounts: 0 }
[2020-03-02T13:46:51.615045006Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196999i block_height=1858590i
[2020-03-02T13:46:51.630730294Z INFO  solana_runtime::accounts_db] scan took 152us merge took 26us accumulate took 178us
[2020-03-02T13:46:51.630766624Z INFO  solana_runtime::bank] bank frozen: 2196999 hash: 6KZzd4NjSXLFGLr7kqbbXhSb5kpvNMPuAva1WCWyssGg accounts_delta: B67MPF47TuSL4quL3hU5MgA8183JGsDhsRXm8wWsV4mM signature_count: 94 last_blockhash: BVw6juZ8X6yWtfnXTtQuBcsWhhFQkMtmo2TfyAVWkC1j
[2020-03-02T13:46:51.630800053Z INFO  solana_runtime::bank] accounts hash slot: 2196999 stats: BankHashStats { num_removed_accounts: 99, num_added_accounts: 0, num_lamports_stored: 4812637870904, total_data_len: 333836, num_executable_accounts: 0 }
[2020-03-02T13:46:51.631839582Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197000i block_height=1858591i
[2020-03-02T13:46:51.651316571Z INFO  solana_metrics::counter] COUNTER:{"name": "vote-native", "counts": 2001, "samples": 2000,  "now": 1583156811651, "events": 1}
[2020-03-02T13:46:51.659283106Z INFO  solana_runtime::accounts_db] scan took 135us merge took 33us accumulate took 251us
[2020-03-02T13:46:51.659325995Z INFO  solana_runtime::bank] bank frozen: 2197000 hash: 25nRZTNmKZ8ekezqwt1HExSUdhdHuya6puxoS3k5hATM accounts_delta: GoWEirbxgzoyaLdGejC8BJcHr4Nz7HukBk5AtkUeX914 signature_count: 178 last_blockhash: WLjoME218PzTKRCFtUZ3tVtQ1YtDYqXZjh3qnffLo78
[2020-03-02T13:46:51.659357776Z INFO  solana_runtime::bank] accounts hash slot: 2197000 stats: BankHashStats { num_removed_accounts: 181, num_added_accounts: 0, num_lamports_stored: 8727554573361, total_data_len: 487996, num_executable_accounts: 0 }
[2020-03-02T13:46:51.660329953Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197001i block_height=1858592i
[2020-03-02T13:46:51.669179625Z INFO  solana_runtime::accounts_db] scan took 151us merge took 9us accumulate took 177us
[2020-03-02T13:46:51.669216096Z INFO  solana_runtime::bank] bank frozen: 2197001 hash: HuefqXsHtbTGnu5nt3bsYoGZE9tGT5qSrrK4xCnWzNB2 accounts_delta: 52ZLR8RdMHYbVgbEptMUBTJMJ5tmHbSTesdMJCh7iFjT signature_count: 28 last_blockhash: Ca7F7nr3BZSFZPCGQRAHsEQkMRc5USRgTgFkVzDDFV9A
[2020-03-02T13:46:51.669245596Z INFO  solana_runtime::bank] accounts hash slot: 2197001 stats: BankHashStats { num_removed_accounts: 32, num_added_accounts: 0, num_lamports_stored: 1136809212463, total_data_len: 202800, num_executable_accounts: 0 }
[2020-03-02T13:46:51.670251374Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197002i block_height=1858593i
[2020-03-02T13:46:51.683701254Z INFO  solana_runtime::accounts_db] scan took 258us merge took 2us accumulate took 284us
[2020-03-02T13:46:51.683738614Z INFO  solana_runtime::bank] bank frozen: 2197002 hash: 4ZoXYqXHmgJcnpKPAqdLCkYRALbDZkgfmX8k1mRa8EBb accounts_delta: CBceq2tJjhwhHjEdudqdd9fuiLZzjRCbdjcB6rYKXMtr signature_count: 88 last_blockhash: 2P8wDEUgmfw8df4XxpJLTLHz53v3rEjJJz8nbnePKbu7
[2020-03-02T13:46:51.683765745Z INFO  solana_runtime::bank] accounts hash slot: 2197002 stats: BankHashStats { num_removed_accounts: 93, num_added_accounts: 0, num_lamports_stored: 4899707999906, total_data_len: 322274, num_executable_accounts: 0 }
[2020-03-02T13:46:51.684838053Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197003i block_height=1858594i
[2020-03-02T13:46:51.702831009Z INFO  solana_runtime::accounts_db] scan took 89us merge took 32us accumulate took 350us
[2020-03-02T13:46:51.702865300Z INFO  solana_runtime::bank] bank frozen: 2197003 hash: 49sbi6kdDWr324odp8YckhWNpMxkkoXd8nYa6TzDwADy accounts_delta: HZWsDxz3dMLYyrvbkXmVWLB746EWvJfAh9GneP7td7fa signature_count: 104 last_blockhash: CpxKVnQTJCAzDKb4A2f6eyBbr9PG9TmFuL6CT3Mbe58G
[2020-03-02T13:46:51.702889779Z INFO  solana_runtime::bank] accounts hash slot: 2197003 stats: BankHashStats { num_removed_accounts: 109, num_added_accounts: 0, num_lamports_stored: 4372836773800, total_data_len: 353106, num_executable_accounts: 0 }
[2020-03-02T13:46:51.703922428Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197004i block_height=1858595i
[2020-03-02T13:46:51.725283552Z INFO  solana_runtime::accounts_db] scan took 125us merge took 31us accumulate took 334us
[2020-03-02T13:46:51.725319013Z INFO  solana_runtime::bank] bank frozen: 2197004 hash: 5q5T51Rg6ECWy5MuBy93pvtyLaaHsyXzp2he5AUTSj5u accounts_delta: 4fFCVes4rPsa8nMyS6QGb5ZPM6dYrqKMrg5zhn6T9WpS signature_count: 112 last_blockhash: VWrbehV6b3YuXTuHCWvEbV5Pfk4vE9VCQK29jNqvP8Q
[2020-03-02T13:46:51.725346142Z INFO  solana_runtime::bank] accounts hash slot: 2197004 stats: BankHashStats { num_removed_accounts: 117, num_added_accounts: 0, num_lamports_stored: 6412540027217, total_data_len: 368522, num_executable_accounts: 0 }
[2020-03-02T13:46:51.726391131Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197005i block_height=1858596i
[2020-03-02T13:46:51.739426907Z INFO  solana_runtime::accounts_db] scan took 139us merge took 29us accumulate took 192us
[2020-03-02T13:46:51.739465897Z INFO  solana_runtime::bank] bank frozen: 2197005 hash: UeRvkLZkNa4jwmpm5oY815ypCVjWfQD4E1jGqPfxnaS accounts_delta: 4ZxjtPdvXmHLAaknjKLQA6XKUni5rLSh5z3UJvL3URME signature_count: 78 last_blockhash: EjW8ar1k1ZmFRm7qYTEo6rBmss7Me8ve8A8CDx21FT5k
[2020-03-02T13:46:51.739500878Z INFO  solana_runtime::bank] accounts hash slot: 2197005 stats: BankHashStats { num_removed_accounts: 83, num_added_accounts: 0, num_lamports_stored: 3307871072686, total_data_len: 303004, num_executable_accounts: 0 }
[2020-03-02T13:46:51.740615167Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197006i block_height=1858597i
[2020-03-02T13:46:51.748763162Z INFO  solana_runtime::accounts_db] scan took 111us merge took 11us accumulate took 149us
[2020-03-02T13:46:51.748801753Z INFO  solana_runtime::bank] bank frozen: 2197006 hash: AfSQ81sc7KnoQfS5yRJk3bxsZNY6mhCPSchJxZem8EzX accounts_delta: 2Uob4iMyzZqjbB2N2Voym6hWr6BoeHcmkHK1jYfADJ4Z signature_count: 20 last_blockhash: 8nzM2eoAH63DVessTKaAtPt9ywz2RRQozAyy6XpV9mHu
[2020-03-02T13:46:51.748838873Z INFO  solana_runtime::bank] accounts hash slot: 2197006 stats: BankHashStats { num_removed_accounts: 25, num_added_accounts: 0, num_lamports_stored: 1013827127504, total_data_len: 191238, num_executable_accounts: 0 }
[2020-03-02T13:46:51.749910832Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197007i block_height=1858598i
[2020-03-02T13:46:51.755774310Z INFO  solana_runtime::accounts_db] scan took 97us merge took 350ns accumulate took 10us
[2020-03-02T13:46:51.755808860Z INFO  solana_runtime::bank] bank frozen: 2197007 hash: 98uq9dDZn5p7zijBN7Aip68cpCpB71KQczrGjADFxPkM accounts_delta: 3wTheAvdZ3rDNG4u3PfamYjXM8duJEcANkkNaRxCh6Rk signature_count: 8 last_blockhash: 2tgV47zoBJ1F3Mg7c4TksXGVXggu6fwRX2fCj2vH8hqB
[2020-03-02T13:46:51.755827040Z INFO  solana_runtime::bank] accounts hash slot: 2197007 stats: BankHashStats { num_removed_accounts: 13, num_added_accounts: 0, num_lamports_stored: 910000000004, total_data_len: 168114, num_executable_accounts: 0 }
[2020-03-02T13:46:51.756875409Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197008i block_height=1858599i
[2020-03-02T13:46:51.797557619Z INFO  solana_runtime::accounts_db] scan took 135us merge took 25us accumulate took 367us
[2020-03-02T13:46:51.797591120Z INFO  solana_runtime::bank] bank frozen: 2197008 hash: EouDH23iU5jPLYK9Jy1mPNP5v4Hr3vVE6ToY5v1avrjy accounts_delta: CL3k7zK3C2dder11GsmXSUAaN3YG5yzmT5oeoP6LQpEB signature_count: 302 last_blockhash: BNCsd1c4mLbCfs8VCtEziaBhzf5FPMkaE1uPdPWmvUyo
[2020-03-02T13:46:51.797617290Z INFO  solana_runtime::bank] accounts hash slot: 2197008 stats: BankHashStats { num_removed_accounts: 278, num_added_accounts: 0, num_lamports_stored: 13430294088953, total_data_len: 622886, num_executable_accounts: 0 }
[2020-03-02T13:46:51.798615838Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197009i block_height=1858600i
[2020-03-02T13:46:51.805160451Z INFO  solana_runtime::accounts_db] scan took 103us merge took 9us accumulate took 149us
[2020-03-02T13:46:51.805201962Z INFO  solana_runtime::bank] bank frozen: 2197009 hash: 5SQ259RQeUxz5amo7dA2TNUFow6xiy9cyLfG6GYWuDe5 accounts_delta: HPo4VYBAVaMo5YrDu741DnewRnNB96wHvi22pLpgfYaq signature_count: 22 last_blockhash: vzg3zqwgrUsumBqeFwFSzSjZz2iiTbsM9R2RzxPLJdc
[2020-03-02T13:46:51.805228211Z INFO  solana_runtime::bank] accounts hash slot: 2197009 stats: BankHashStats { num_removed_accounts: 27, num_added_accounts: 0, num_lamports_stored: 1125846726871, total_data_len: 195092, num_executable_accounts: 0 }
[2020-03-02T13:46:51.806266020Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197010i block_height=1858601i
[2020-03-02T13:46:51.823451160Z INFO  solana_runtime::accounts_db] scan took 127us merge took 3us accumulate took 250us
[2020-03-02T13:46:51.823492390Z INFO  solana_runtime::bank] bank frozen: 2197010 hash: 4FgxWADe1zcdStfdP368M3f4E6kmmMH1hu5NWhhtUhB4 accounts_delta: BNgT63SqUMS8yaa1sJL8LTgHzCrFj2ftmVUep1EGeGUq signature_count: 78 last_blockhash: Z5RjyFfuFy1KTcDbWtGLszAGyiiuhS2RJbysdiQxxvB
[2020-03-02T13:46:51.823525120Z INFO  solana_runtime::bank] accounts hash slot: 2197010 stats: BankHashStats { num_removed_accounts: 82, num_added_accounts: 0, num_lamports_stored: 2326866173067, total_data_len: 299150, num_executable_accounts: 0 }
[2020-03-02T13:46:51.824642260Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197011i block_height=1858602i
[2020-03-02T13:46:51.844760803Z INFO  solana_runtime::accounts_db] scan took 130us merge took 28us accumulate took 345us
[2020-03-02T13:46:51.844787624Z INFO  solana_runtime::bank] bank frozen: 2197011 hash: 4L9D2YQd3ZiLdnbT6B4YiqHD3SzRWNJS7qhdGmAvzm5H accounts_delta: Gzn27LGqRAinfKb6ta6m6EU8ihe5ptDKjdDboxxc3nM4 signature_count: 86 last_blockhash: BpmTdVYSpizqW67rdJH16SV7JYVPaPPYiRqWZREDemZa
[2020-03-02T13:46:51.844823814Z INFO  solana_runtime::bank] accounts hash slot: 2197011 stats: BankHashStats { num_removed_accounts: 90, num_added_accounts: 0, num_lamports_stored: 5357796044138, total_data_len: 314566, num_executable_accounts: 0 }
[2020-03-02T13:46:51.845881842Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197016i block_height=1858603i
[2020-03-02T13:46:51.877027606Z INFO  solana_runtime::accounts_db] scan took 135us merge took 13us accumulate took 234us
[2020-03-02T13:46:51.877069556Z INFO  solana_runtime::bank] bank frozen: 2197016 hash: AxyCNnUAKBynEWNDKguy2VBRyXd2v8yDHVzF276mj4Qe accounts_delta: 4qCRYvMfW4bK3LsQt7eXzQthsTALHYViD93FWFNNKPzR signature_count: 202 last_blockhash: 59LFYroQapr4mFDsxPX5xUaFgVYFJ8RZNshPckQrC4W7
[2020-03-02T13:46:51.877093346Z INFO  solana_runtime::bank] accounts hash slot: 2197016 stats: BankHashStats { num_removed_accounts: 195, num_added_accounts: 0, num_lamports_stored: 9862054024059, total_data_len: 495704, num_executable_accounts: 0 }
[2020-03-02T13:46:51.878143984Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197017i block_height=1858604i
[2020-03-02T13:46:51.888453688Z INFO  solana_runtime::accounts_db] scan took 104us merge took 23us accumulate took 157us
[2020-03-02T13:46:51.888487539Z INFO  solana_runtime::bank] bank frozen: 2197017 hash: 4nJzFaGSPb53z4KYkTS9EXLmgMdQjxqL9tzpTp5Rwtmm accounts_delta: 42R8vUXdiMfR3QBWyxUqabjFyrGwWjpG2kipYaDq9uGJ signature_count: 54 last_blockhash: 6XjarN3jD2SCpbxJTF1LWHnTEgKZ8n4GUTwQiHfbx75A
[2020-03-02T13:46:51.888518529Z INFO  solana_runtime::bank] accounts hash slot: 2197017 stats: BankHashStats { num_removed_accounts: 59, num_added_accounts: 0, num_lamports_stored: 2527875972778, total_data_len: 256756, num_executable_accounts: 0 }
[2020-03-02T13:46:51.889524407Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197018i block_height=1858605i
[2020-03-02T13:46:51.901465704Z INFO  solana_runtime::accounts_db] scan took 126us merge took 26us accumulate took 332us
[2020-03-02T13:46:51.901498494Z INFO  solana_runtime::bank] bank frozen: 2197018 hash: DRL9Ms8eaQNGKWfma64FEWae9rZYFFGu5ZuVkrsTMgf2 accounts_delta: FWLVJCNUBt6DVLKAKrMnbAiCQm1SZuARqGAbi9UdYMDS signature_count: 76 last_blockhash: 4smSwAtRrnz3bV23kRwqCvKkAnSf4apgjvQmApf4Y6LX
[2020-03-02T13:46:51.901539634Z INFO  solana_runtime::bank] accounts hash slot: 2197018 stats: BankHashStats { num_removed_accounts: 81, num_added_accounts: 0, num_lamports_stored: 3619717799574, total_data_len: 299150, num_executable_accounts: 0 }
[2020-03-02T13:46:51.902530702Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197019i block_height=1858606i
[2020-03-02T13:46:51.921411896Z INFO  solana_runtime::accounts_db] scan took 132us merge took 27us accumulate took 345us
[2020-03-02T13:46:51.921443517Z INFO  solana_runtime::bank] bank frozen: 2197019 hash: GFGoywFayUc4kWaPndxaT8nSyE6z7Uxkb45VLcbawGXX accounts_delta: DGGbRhycwKmGQH1pb5R1RYGGAATUmQKAEfWsXxgbwZez signature_count: 94 last_blockhash: 5sq18reiMd8L6xeNMNi5GDahsrAraBoE3jKsc8B4u7z
[2020-03-02T13:46:51.921458127Z INFO  solana_runtime::bank] accounts hash slot: 2197019 stats: BankHashStats { num_removed_accounts: 99, num_added_accounts: 0, num_lamports_stored: 4423740614412, total_data_len: 333836, num_executable_accounts: 0 }
[2020-03-02T13:46:51.922267463Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197020i block_height=1858607i
[2020-03-02T13:46:51.953166844Z INFO  solana_runtime::accounts_db] scan took 133us merge took 15us accumulate took 333us
[2020-03-02T13:46:51.953190184Z INFO  solana_runtime::bank] bank frozen: 2197020 hash: YT3WjW88VMu6bjRoZ12nAAZXUXBB8mFu9xo2vsUJg4r accounts_delta: 4m4MH5syCwj6QVZAbHsJBvN2qXWvfZn8UAAUzsQUMLyu signature_count: 182 last_blockhash: C7vvAjnqvFNDMKcVTpShG5mAzfu1FT5eYBab48oQ8Nya
[2020-03-02T13:46:51.953218145Z INFO  solana_runtime::bank] accounts hash slot: 2197020 stats: BankHashStats { num_removed_accounts: 184, num_added_accounts: 0, num_lamports_stored: 9154355670484, total_data_len: 491850, num_executable_accounts: 0 }
[2020-03-02T13:46:51.954263874Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197021i block_height=1858608i
[2020-03-02T13:46:51.964766759Z INFO  solana_runtime::accounts_db] scan took 111us merge took 22us accumulate took 285us
[2020-03-02T13:46:51.964804439Z INFO  solana_runtime::bank] bank frozen: 2197021 hash: APrvak2DpS5JPKF3s71hXUcDxpCNS9X6pixbXfRyUf2Z accounts_delta: BLDV7ZKGh712psEZnFv8MtDEirddbxLnd5CGHZyfvdHu signature_count: 54 last_blockhash: CnPF38mNuxgHE2PTQBTtcmr6gePrf5ALhMnNkyiEKBfU
[2020-03-02T13:46:51.964833989Z INFO  solana_runtime::bank] accounts hash slot: 2197021 stats: BankHashStats { num_removed_accounts: 59, num_added_accounts: 0, num_lamports_stored: 2712890672307, total_data_len: 256756, num_executable_accounts: 0 }
[2020-03-02T13:46:51.965932249Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197022i block_height=1858609i
[2020-03-02T13:46:51.985433367Z INFO  solana_runtime::accounts_db] scan took 141us merge took 31us accumulate took 185us
[2020-03-02T13:46:51.985472777Z INFO  solana_runtime::bank] bank frozen: 2197022 hash: Atn9tYLkWTqjfnY2B8f4K1j3MfJPTnrJqtBXSQZZ2cqG accounts_delta: ENtcimhKUufUnXJXM23LcBn1hakG42yjYDss8pn9vB5L signature_count: 130 last_blockhash: F2czD5cPf3Y87bdJSUV9s5CDkKLsqMMeYVGBCyv5Z3Q2
[2020-03-02T13:46:51.985495577Z INFO  solana_runtime::bank] accounts hash slot: 2197022 stats: BankHashStats { num_removed_accounts: 135, num_added_accounts: 0, num_lamports_stored: 5366663901073, total_data_len: 403208, num_executable_accounts: 0 }
[2020-03-02T13:46:51.986569806Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197023i block_height=1858610i
[2020-03-02T13:46:52.004063389Z INFO  solana_runtime::accounts_db] scan took 160us merge took 29us accumulate took 202us
[2020-03-02T13:46:52.004100349Z INFO  solana_runtime::bank] bank frozen: 2197023 hash: 9tPHgnqLpx12aFGkqNhxBvJCSxVbVMpF3WYWzXj6BFeT accounts_delta: GkiCaBWh3y586dJWaN3a4SRPnBezocB87tjVn4354UWd signature_count: 92 last_blockhash: G3zSTyNbFodzUEw1iqYn2YpNXUqjr1hTREgAHLcHnnXt
[2020-03-02T13:46:52.004130449Z INFO  solana_runtime::bank] accounts hash slot: 2197023 stats: BankHashStats { num_removed_accounts: 97, num_added_accounts: 0, num_lamports_stored: 4190703100019, total_data_len: 329982, num_executable_accounts: 0 }
[2020-03-02T13:46:52.005211567Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197028i block_height=1858611i
[2020-03-02T13:46:52.032612830Z INFO  solana_runtime::accounts_db] scan took 148us merge took 31us accumulate took 330us
[2020-03-02T13:46:52.032648040Z INFO  solana_runtime::bank] bank frozen: 2197028 hash: HxDTsBNKXVkhvBZeVq4RQYS7Tbkr93NQ98GDdozMCveH accounts_delta: 58HRhrrnQV1u6YodRC2TpJQPbw74P15C6GGcEBHc2oUM signature_count: 134 last_blockhash: AXTYF5viUxwTCQHTmxz6uCzQTNL19CG5EgZu7bNkMw8y
[2020-03-02T13:46:52.032677661Z INFO  solana_runtime::bank] accounts hash slot: 2197028 stats: BankHashStats { num_removed_accounts: 134, num_added_accounts: 0, num_lamports_stored: 7457376954409, total_data_len: 391646, num_executable_accounts: 0 }
[2020-03-02T13:46:52.033480667Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197029i block_height=1858612i
[2020-03-02T13:46:52.038971152Z INFO  solana_runtime::accounts_db] scan took 92us merge took 190ns accumulate took 7us
[2020-03-02T13:46:52.039003513Z INFO  solana_runtime::bank] bank frozen: 2197029 hash: Aq7kbyCNZPT6kW6fjC95QuPsemfNDfuciqGRZPQqJSkH accounts_delta: 3aDH8hGZxHDteomd11jysVLRAcnRrCzAf43mKjjUjCu4 signature_count: 6 last_blockhash: 2cPnSSZBnMDfH4jhjZeTdj6fLmd61FQTVQYUWr8UreAb
[2020-03-02T13:46:52.039030902Z INFO  solana_runtime::bank] accounts hash slot: 2197029 stats: BankHashStats { num_removed_accounts: 11, num_added_accounts: 0, num_lamports_stored: 19934770883, total_data_len: 164260, num_executable_accounts: 0 }
[2020-03-02T13:46:52.040060071Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197030i block_height=1858613i
[2020-03-02T13:46:52.043667980Z INFO  solana_metrics::counter] COUNTER:{"name": "vote-native", "counts": 3001, "samples": 3000,  "now": 1583156812043, "events": 1}
[2020-03-02T13:46:52.056131681Z INFO  solana_runtime::accounts_db] scan took 165us merge took 26us accumulate took 164us
[2020-03-02T13:46:52.056158842Z INFO  solana_runtime::bank] bank frozen: 2197030 hash: 9X26ka3MpiF7LwAcyZFNAibjrWsAqWcxyRKAMcmW3Vae accounts_delta: 2pey3MNhwEw9EhLsYMv6FpnPDvEXDWgQKSkgaRydWk6Z signature_count: 86 last_blockhash: 3oX7F2ojf2DdMoeWLDodABPqRw6wJ5P3QDHB9e13xXGE
[2020-03-02T13:46:52.056178441Z INFO  solana_runtime::bank] accounts hash slot: 2197030 stats: BankHashStats { num_removed_accounts: 91, num_added_accounts: 0, num_lamports_stored: 4543712899726, total_data_len: 318420, num_executable_accounts: 0 }
[2020-03-02T13:46:52.057241430Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197031i block_height=1858614i
[2020-03-02T13:46:52.074389510Z INFO  solana_runtime::accounts_db] scan took 120us merge took 30us accumulate took 296us
[2020-03-02T13:46:52.074424210Z INFO  solana_runtime::bank] bank frozen: 2197031 hash: EA5dpjFxQveX4RMmFaJb3uEcfmfM8Eon4CQVvN3jqhMz accounts_delta: 2v3xpVsw6o5N1pk9VBL7Qmwi69ZqKwhVnK2mGx1yADbq signature_count: 112 last_blockhash: Dy5Yfx14ZcFtbp1vNTHuFzaxganYoGasMaP3EpZ151Su
[2020-03-02T13:46:52.074452110Z INFO  solana_runtime::bank] accounts hash slot: 2197031 stats: BankHashStats { num_removed_accounts: 114, num_added_accounts: 0, num_lamports_stored: 5531580757138, total_data_len: 356960, num_executable_accounts: 0 }
[2020-03-02T13:46:52.075546880Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197036i block_height=1858615i
[2020-03-02T13:46:52.107534049Z INFO  solana_runtime::accounts_db] scan took 147us merge took 34us accumulate took 200us
[2020-03-02T13:46:52.107573710Z INFO  solana_runtime::bank] bank frozen: 2197036 hash: 2wB1usUx9m8CKWtqUJSXBCxwpFNHt9kpNfeN9YMVrgfa accounts_delta: 3AZUW3rncsMZBTkx46dSCyLXhzty1WB5qHqZhfbJpNLx signature_count: 188 last_blockhash: 9e1cHj3Psw99cEAXTTcuAzVJoeeUTqbmYcBVDZcJMr1q
[2020-03-02T13:46:52.107617131Z INFO  solana_runtime::bank] accounts hash slot: 2197036 stats: BankHashStats { num_removed_accounts: 183, num_added_accounts: 0, num_lamports_stored: 9575184482301, total_data_len: 476434, num_executable_accounts: 0 }
[2020-03-02T13:46:52.108709159Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197037i block_height=1858616i
[2020-03-02T13:46:52.113585089Z INFO  solana_runtime::accounts_db] scan took 82us merge took 260ns accumulate took 6us
[2020-03-02T13:46:52.113606899Z INFO  solana_runtime::bank] bank frozen: 2197037 hash: 3vHGUW1hSNn3t1zTSbMFErNgzzN2yQXczKmqqCMj6Rm7 accounts_delta: 9AVQjTTCVt5iwuyUhx2fEf9KkHch8jaFJSP5f7nNy5j7 signature_count: 2 last_blockhash: qiuKu8iZuk9KYHkCeocCCz8ECDVGXHo7xqP2CrXZcMN
[2020-03-02T13:46:52.113624959Z INFO  solana_runtime::bank] accounts hash slot: 2197037 stats: BankHashStats { num_removed_accounts: 7, num_added_accounts: 0, num_lamports_stored: 9995100165, total_data_len: 156552, num_executable_accounts: 0 }
[2020-03-02T13:46:52.114680747Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197038i block_height=1858617i
[2020-03-02T13:46:52.119735128Z INFO  solana_runtime::accounts_db] scan took 102us merge took 290ns accumulate took 7us
[2020-03-02T13:46:52.119766268Z INFO  solana_runtime::bank] bank frozen: 2197038 hash: 2CiNQNuPvurfen9yHkmfsExsyfv8QGQt3DGt5c4xf5n6 accounts_delta: 9e6SRYa5EnZ8u2zDMM7RATMpSjwLLw6Gv1kE21GcUYJK signature_count: 2 last_blockhash: 9xTLMPFHWcH2rASoFHLcHX4fz9Lfm9ZB18YN82hR8pzN
[2020-03-02T13:46:52.119787039Z INFO  solana_runtime::bank] accounts hash slot: 2197038 stats: BankHashStats { num_removed_accounts: 7, num_added_accounts: 0, num_lamports_stored: 9995100165, total_data_len: 156552, num_executable_accounts: 0 }
[2020-03-02T13:46:52.120628286Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197039i block_height=1858618i
[2020-03-02T13:46:52.132243780Z INFO  solana_runtime::accounts_db] scan took 162us merge took 250ns accumulate took 251us
[2020-03-02T13:46:52.132279090Z INFO  solana_runtime::bank] bank frozen: 2197039 hash: 32AhQKrsnF6C1getvZkrTpjXJ9nbbFkYAKqtXKkojJcH accounts_delta: 22bc3xd9vuS21aYJMAyLLtHeqYxYpR4G5W8Uyu9yY5AK signature_count: 58 last_blockhash: FqUmKy97oiijwaTEHyDz64W7n6M5Jfgqkt6e2gVKv5Zr
[2020-03-02T13:46:52.132308350Z INFO  solana_runtime::bank] accounts hash slot: 2197039 stats: BankHashStats { num_removed_accounts: 63, num_added_accounts: 0, num_lamports_stored: 3647871072718, total_data_len: 264464, num_executable_accounts: 0 }
[2020-03-02T13:46:52.133337570Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197040i block_height=1858619i
[2020-03-02T13:46:52.171250058Z INFO  solana_runtime::accounts_db] scan took 150us merge took 24us accumulate took 246us
[2020-03-02T13:46:52.171288648Z INFO  solana_runtime::bank] bank frozen: 2197040 hash: BVc2GjaJS9dmzs5kRbqupuxJnx66Pr4iMd7d8nQS3mFW accounts_delta: GDDc7HrgGeTnMHGww9R9XxgCsiuysZpMcAG61prEV2Ag signature_count: 214 last_blockhash: HD9yKoXdVhm6E7ni2NJ77RmM8BAz5ZcPS1fXB1KQQuLv
[2020-03-02T13:46:52.171318958Z INFO  solana_runtime::bank] accounts hash slot: 2197040 stats: BankHashStats { num_removed_accounts: 205, num_added_accounts: 0, num_lamports_stored: 9893003494459, total_data_len: 511120, num_executable_accounts: 0 }
[2020-03-02T13:46:52.172393157Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197041i block_height=1858620i
[2020-03-02T13:46:52.177832910Z INFO  solana_runtime::accounts_db] scan took 133us merge took 330ns accumulate took 8us
[2020-03-02T13:46:52.177864671Z INFO  solana_runtime::bank] bank frozen: 2197041 hash: 76sn4k9bKmv5nK8bBeMxHSK6Bjawj4kx8CwTmPi8FAwV accounts_delta: 7yJmcjLArm5h1h94Mn9YdPu6dELwhzeovGbC2CFuuUka signature_count: 10 last_blockhash: 9jypzrD9gVs63LZKQBFRo6yRPog5rtFiVzxDd3D19iGa
[2020-03-02T13:46:52.177894371Z INFO  solana_runtime::bank] accounts hash slot: 2197041 stats: BankHashStats { num_removed_accounts: 15, num_added_accounts: 0, num_lamports_stored: 27962485605, total_data_len: 171968, num_executable_accounts: 0 }
[2020-03-02T13:46:52.178906150Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197042i block_height=1858621i
[2020-03-02T13:46:52.194557267Z INFO  solana_runtime::accounts_db] scan took 182us merge took 7us accumulate took 196us
[2020-03-02T13:46:52.194591217Z INFO  solana_runtime::bank] bank frozen: 2197042 hash: D339oWRjwpyFaWWhsy7VLBeEnoLeNSe5hbeU1vWmdLh1 accounts_delta: 3BHCjzmbMgG6Tc91CBFCVh2GxYdr8HjNFNdoekde7wes signature_count: 108 last_blockhash: 3P1YNrXdbhsFRQfs756cKKpR7dgF9hVNzr1vyU7epVx9
[2020-03-02T13:46:52.194621987Z INFO  solana_runtime::bank] accounts hash slot: 2197042 stats: BankHashStats { num_removed_accounts: 113, num_added_accounts: 0, num_lamports_stored: 5691655785952, total_data_len: 360814, num_executable_accounts: 0 }
[2020-03-02T13:46:52.195693867Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197043i block_height=1858622i
[2020-03-02T13:46:52.217657405Z INFO  solana_runtime::accounts_db] scan took 206us merge took 31us accumulate took 329us
[2020-03-02T13:46:52.217694225Z INFO  solana_runtime::bank] bank frozen: 2197043 hash: 8CmmsBxqBGWwemAqJD7z6eP4hqB7vxWZZxLMAvzZy2eH accounts_delta: Hj2n2EhUv4zeqxXK9RjzfXWwLWLk1qFLPwWuzQGU5VEK signature_count: 128 last_blockhash: 2m7wgQwamJtiST5xFqHbAagdTxFqzy7Upwu9Ny8w3ykP
[2020-03-02T13:46:52.217720866Z INFO  solana_runtime::bank] accounts hash slot: 2197043 stats: BankHashStats { num_removed_accounts: 133, num_added_accounts: 0, num_lamports_stored: 5848729130425, total_data_len: 399354, num_executable_accounts: 0 }
[2020-03-02T13:46:52.218796304Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197052i block_height=1858623i
[2020-03-02T13:46:52.245049908Z INFO  solana_runtime::accounts_db] scan took 164us merge took 1us accumulate took 289us
[2020-03-02T13:46:52.245090138Z INFO  solana_runtime::bank] bank frozen: 2197052 hash: GXe5jJs5hnfhszUASei3fGe7mzLThgQAyf4RUJgq7fWT accounts_delta: GNQhQpUjPjtG1MUBw7YAJxkrTTBisECFatJezKhKP7tb signature_count: 160 last_blockhash: 7MQMpqJ3n5cwKENX8bG6AZZr4GNXgVKw9sCbTtN8GLc2
[2020-03-02T13:46:52.245121928Z INFO  solana_runtime::bank] accounts hash slot: 2197052 stats: BankHashStats { num_removed_accounts: 158, num_added_accounts: 0, num_lamports_stored: 8148231796386, total_data_len: 434040, num_executable_accounts: 0 }
[2020-03-02T13:46:52.246204067Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197053i block_height=1858624i
[2020-03-02T13:46:52.251500959Z INFO  solana_runtime::accounts_db] scan took 117us merge took 180ns accumulate took 6us
[2020-03-02T13:46:52.251529890Z INFO  solana_runtime::bank] bank frozen: 2197053 hash: Es59BSLAfvZQT97izcQhSBFGnyJHo5p2d4cLhhCUeiH5 accounts_delta: Ct2iaHupQ1wiN33Djr11MfpBA2sJ12recF9BtvxfAbW signature_count: 2 last_blockhash: H6umv1Rt7Snqq7RyiVT1y1fXTeUKKprH3GCRKBQLsTFW
[2020-03-02T13:46:52.251567821Z INFO  solana_runtime::bank] accounts hash slot: 2197053 stats: BankHashStats { num_removed_accounts: 7, num_added_accounts: 0, num_lamports_stored: 1000000005, total_data_len: 156552, num_executable_accounts: 0 }
[2020-03-02T13:46:52.252614839Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197054i block_height=1858625i
[2020-03-02T13:46:52.258455116Z INFO  solana_runtime::accounts_db] scan took 105us merge took 200ns accumulate took 161us
[2020-03-02T13:46:52.258489327Z INFO  solana_runtime::bank] bank frozen: 2197054 hash: 6fYgVRg9MxjF8rhnPUgcjsNSNJ4FVRsRdGMyBBBr6WUK accounts_delta: 74SJH5SQxoainFgydizMyHeK6chFvLG5KkGrviWsfDjF signature_count: 16 last_blockhash: cW9TFS4XaAh44hay14JudxSeYeQNRnZJ1p3jaMxWTob
[2020-03-02T13:46:52.258519427Z INFO  solana_runtime::bank] accounts hash slot: 2197054 stats: BankHashStats { num_removed_accounts: 21, num_added_accounts: 0, num_lamports_stored: 134942886014, total_data_len: 183530, num_executable_accounts: 0 }
[2020-03-02T13:46:52.259547775Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197055i block_height=1858626i
[2020-03-02T13:46:52.276739825Z INFO  solana_runtime::accounts_db] scan took 125us merge took 27us accumulate took 233us
[2020-03-02T13:46:52.276781265Z INFO  solana_runtime::bank] bank frozen: 2197055 hash: 5KxkPPpYruwRrPLz6mt7h7NmVfngEhZiCMEJ9ygjKTcJ accounts_delta: HEte6UJ1TmYFFA3VMVBFksnwm56zH5Piux149sUjArMg signature_count: 90 last_blockhash: G7Kre6dBrq7vVJwoXmXzXyQXfXaosdMr8hcjSvmcWvFT
[2020-03-02T13:46:52.276802606Z INFO  solana_runtime::bank] accounts hash slot: 2197055 stats: BankHashStats { num_removed_accounts: 92, num_added_accounts: 0, num_lamports_stored: 5901582441457, total_data_len: 314566, num_executable_accounts: 0 }
[2020-03-02T13:46:52.277890854Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197056i block_height=1858627i
[2020-03-02T13:46:52.312414085Z INFO  solana_runtime::accounts_db] scan took 165us merge took 40us accumulate took 365us
[2020-03-02T13:46:52.312451805Z INFO  solana_runtime::bank] bank frozen: 2197056 hash: DJbTdZs5JZNQsVQ9rNLvJfwXeRqyHsGMVSb4NoiUgqr9 accounts_delta: F2tPoUj67DU8nwAspP8AyVciT2W1HxG5RvT6jM28xqrw signature_count: 236 last_blockhash: 9BEebFoLdnzybsQWqhqqqe1zVQtGkB4xLRuZS6vGWVhF
[2020-03-02T13:46:52.312484746Z INFO  solana_runtime::bank] accounts hash slot: 2197056 stats: BankHashStats { num_removed_accounts: 224, num_added_accounts: 0, num_lamports_stored: 12079732778277, total_data_len: 541952, num_executable_accounts: 0 }
[2020-03-02T13:46:52.313521504Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197057i block_height=1858628i
[2020-03-02T13:46:52.322988651Z INFO  solana_runtime::accounts_db] scan took 119us merge took 11us accumulate took 149us
[2020-03-02T13:46:52.323023122Z INFO  solana_runtime::bank] bank frozen: 2197057 hash: 5op2NrrcGdGS13Eo63y7DkgpsqjAFd6fai21jGjbxXBr accounts_delta: SRUVeVWkPkX1HsmjZ1XVf54Bo533i3UXqb9fdAXrReh signature_count: 34 last_blockhash: 3iG47D2UEY9nF3qb4hj8LBSfWnVKk5KqJKvPL16wPKX4
[2020-03-02T13:46:52.323067152Z INFO  solana_runtime::bank] accounts hash slot: 2197057 stats: BankHashStats { num_removed_accounts: 39, num_added_accounts: 0, num_lamports_stored: 645905371601, total_data_len: 218216, num_executable_accounts: 0 }
[2020-03-02T13:46:52.324198021Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197058i block_height=1858629i
[2020-03-02T13:46:52.342589360Z INFO  solana_runtime::accounts_db] scan took 116us merge took 25us accumulate took 263us
[2020-03-02T13:46:52.342628830Z INFO  solana_runtime::bank] bank frozen: 2197058 hash: 2qnF9nc99CLsVQaKYL1WomKUVkZi1vVuzCj1JXXgRSZJ accounts_delta: 2RuK5HT5jYZrEWnHwy4JZHXrVmDpe8qkFNyZUSQnLdUw signature_count: 98 last_blockhash: 2mtVBYqE4jEVAQaCERk6ZSq4yUdbzAg8LFo5TU8SJ5nq
[2020-03-02T13:46:52.342662511Z INFO  solana_runtime::bank] accounts hash slot: 2197058 stats: BankHashStats { num_removed_accounts: 103, num_added_accounts: 0, num_lamports_stored: 4915758529472, total_data_len: 341544, num_executable_accounts: 0 }
[2020-03-02T13:46:52.343726419Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197059i block_height=1858630i
[2020-03-02T13:46:52.366057581Z INFO  solana_runtime::accounts_db] scan took 123us merge took 25us accumulate took 250us
[2020-03-02T13:46:52.366095731Z INFO  solana_runtime::bank] bank frozen: 2197059 hash: 3bmNTjYA6XsQ9zcJGNNNv4kKCy9vdUfUbNaRVu4kfRTo accounts_delta: FxJ8JDfvyyPomWQdcHFgBNPVsUGoqHqDUMqG2Ua6BvF5 signature_count: 98 last_blockhash: 3K8bFvFNHyGv1txWHfW8pUcpN1WVwMyc8t5V6KCfE8Vx
[2020-03-02T13:46:52.366132792Z INFO  solana_runtime::bank] accounts hash slot: 2197059 stats: BankHashStats { num_removed_accounts: 103, num_added_accounts: 0, num_lamports_stored: 4389703100034, total_data_len: 341544, num_executable_accounts: 0 }
[2020-03-02T13:46:52.367207051Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197060i block_height=1858631i
[2020-03-02T13:46:52.401986734Z INFO  solana_runtime::accounts_db] scan took 163us merge took 10us accumulate took 195us
[2020-03-02T13:46:52.402024344Z INFO  solana_runtime::bank] bank frozen: 2197060 hash: BDwmWnqkWeh4TcQwQUWmB9QZta9s73pQ9RgAkPbvAk6p accounts_delta: 8FLMLAJgMvpembgKg1PeGSJqK7rjUozxPLFyE5N5tkmC signature_count: 214 last_blockhash: 4xpY2b9aNytXX1o14drKg69bghYA3ivdGmdJd7JDviBE
[2020-03-02T13:46:52.402047714Z INFO  solana_runtime::bank] accounts hash slot: 2197060 stats: BankHashStats { num_removed_accounts: 214, num_added_accounts: 0, num_lamports_stored: 10398257827013, total_data_len: 545806, num_executable_accounts: 0 }
[2020-03-02T13:46:52.403092672Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197061i block_height=1858632i
[2020-03-02T13:46:52.405672424Z INFO  solana_ledger::blockstore_processor] ledger processed in 1612ms. 388 MB allocated. 1 fork at 2197060, with 1 frozen bank
Creating a snapshot of slot 2197060
[2020-03-02T13:46:52.410018898Z INFO  solana_runtime::accounts_db] total_stores: 8243, newest_slot: 2197060, oldest_slot: 0, max_slot: 2127312 (num=9), min_slot: 1157346 (num=1)
[2020-03-02T13:46:52.410086590Z INFO  solana_metrics::metrics] datapoint: accounts_db-stores total_count=8243i
[2020-03-02T13:46:52.412035185Z INFO  solana_ledger::snapshot_utils] Creating snapshot for slot 2197060, path: "/tmp/.tmpGUlwxT/2197060/2197060"
[2020-03-02T13:46:52.419535346Z INFO  solana_metrics::metrics] datapoint: serialize_account_storage_ms duration=2i num_entries=8243i
[2020-03-02T13:46:52.419980760Z INFO  solana_ledger::snapshot_utils] bank-serialize-ms took 7ms for slot 2197060 at "/tmp/.tmpGUlwxT/2197060/2197060"
[2020-03-02T13:46:52.420027780Z INFO  solana_metrics::metrics] datapoint: snapshot-bank-file slot=2197060i size=2794800i
[2020-03-02T13:46:52.420084890Z INFO  solana_ledger::snapshot_utils] Snapshot for bank: 2197060 has 1821 account storage entries
[2020-03-02T13:46:52.422244558Z INFO  solana_ledger::snapshot_utils] Generating snapshot archive for slot 2197060
[2020-03-02T13:46:52.426286281Z INFO  solana_metrics::metrics] datapoint: snapshot-status-cache-file slot=2197060i size=305760i


$ SOLANA_DISABLE_SNAPSHOT_COMPRESSION=1 LD_LIBRARY_PATH=. ../solana-ledger-tool --ledger newnew verify
4uhcVJyU9pJkvQyS88uRDiswHXSCkY3zQawwpjk2NsNY
[2020-03-02T15:33:25.021043796Z INFO  solana_ledger::blockstore] Maximum open file descriptors: 65000
[2020-03-02T15:33:25.171329871Z INFO  solana_ledger::blockstore] "/mnt/solana-data/tds-with-1.0/try5/newnew/rocksdb" open took 150ms
[2020-03-02T15:33:25.171369731Z INFO  solana_ledger::bank_forks_utils] Initializing snapshot path: "/mnt/solana-data/tds-with-1.0/try5/newnew/snapshot"
[2020-03-02T15:33:25.172079797Z INFO  solana_ledger::bank_forks_utils] Loading snapshot package: "/mnt/solana-data/tds-with-1.0/try5/newnew/snapshot-2197060-5qTAFmWaeyxmG6qXCeZS1ojH4rXmZVKA5VD5FGh6JpeF.tar.bz2"
[2020-03-02T15:35:52.747607712Z INFO  solana_ledger::snapshot_utils] snapshot untar took 147.6s
[2020-03-02T15:35:52.748096905Z INFO  solana_ledger::snapshot_utils] snapshot version: 1.0.0
[2020-03-02T15:35:52.748388028Z INFO  solana_ledger::snapshot_utils] Loading bank from "/mnt/solana-data/tds-with-1.0/try5/newnew/snapshot/.tmpLsCCXO/snapshots/2197060/2197060"
[2020-03-02T15:35:52.759477979Z INFO  solana_ledger::snapshot_utils] Rebuilding accounts...
[2020-03-02T15:36:21.534868800Z INFO  solana_ledger::snapshot_utils] Rebuilding status cache...
[2020-03-02T15:36:21.544955475Z INFO  solana_ledger::snapshot_utils] Loaded bank for slot: 2197060
[2020-03-02T15:36:21.545063955Z INFO  solana_runtime::accounts_db] total_stores: 8243, newest_slot: 2197060, oldest_slot: 0, max_slot: 2051388 (num=9), min_slot: 2171643 (num=1)
[2020-03-02T15:36:21.545102206Z INFO  solana_metrics::metrics] metrics disabled: SOLANA_METRICS_CONFIG: environment variable not found
[2020-03-02T15:36:21.545280127Z INFO  solana_metrics::metrics] datapoint: accounts_db-stores total_count=8243i
[2020-03-02T15:36:21.549511832Z WARN  solana_runtime::accounts_db] mismatched bank hash for slot 2197060: 9brAU4C5d5gq5QABjHmLex2L3nCgHyfN8yvjdwpfxhbs (calculated) != 5qTAFmWaeyxmG6qXCeZS1ojH4rXmZVKA5VD5FGh6JpeF (expected)
[2020-03-02T15:36:21.549549442Z WARN  solana_runtime::accounts] verify_bank_hash failed: MismatchedBankHash
thread 'main' panicked at 'Snapshot bank for slot 2197060 failed to verify', ledger/src/snapshot_utils.rs:459:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace

Good:




$ LD_LIBRARY_PATH=. ROOT_AT_ONCE=1 ../solana-ledger-tool --ledger ledger/ create-snapshot 2197060 newnew2
[2020-03-02T15:17:51.691866438Z INFO  solana_ledger::blockstore] Maximum open file descriptors: 65000
[2020-03-02T15:17:51.788710309Z INFO  solana_ledger::blockstore] "/mnt/solana-data/tds-with-1.0/try5/ledger/rocksdb" open took 96ms
[2020-03-02T15:17:51.788746949Z INFO  solana_ledger::bank_forks_utils] Initializing snapshot path: "/mnt/solana-data/tds-with-1.0/try5/ledger/snapshot"
[2020-03-02T15:17:51.789674498Z INFO  solana_ledger::bank_forks_utils] Loading snapshot package: "/mnt/solana-data/tds-with-1.0/try5/ledger/snapshot-2196960-BKVp1Jh8u23QnPoGPdYW82oXoirW65YWejSNYYwPEbWj.tar.bz2"
[2020-03-02T15:20:21.572587756Z INFO  solana_ledger::snapshot_utils] snapshot untar took 149.8s
[2020-03-02T15:20:21.572855609Z INFO  solana_ledger::snapshot_utils] snapshot version: 1.0.0
[2020-03-02T15:20:21.572904349Z INFO  solana_ledger::snapshot_utils] Loading bank from "/mnt/solana-data/tds-with-1.0/try5/ledger/snapshot/.tmp4PI0pz/snapshots/2196960/2196960"
[2020-03-02T15:20:21.577901778Z INFO  solana_ledger::snapshot_utils] Rebuilding accounts...
[2020-03-02T15:20:51.509661085Z INFO  solana_ledger::snapshot_utils] Rebuilding status cache...
[2020-03-02T15:20:51.517667783Z INFO  solana_ledger::snapshot_utils] Loaded bank for slot: 2196960
[2020-03-02T15:20:51.517752913Z INFO  solana_runtime::accounts_db] total_stores: 8358, newest_slot: 2196960, oldest_slot: 0, max_slot: 2146996 (num=9), min_slot: 1504465 (num=1)
[2020-03-02T15:20:51.517786124Z INFO  solana_metrics::metrics] metrics disabled: SOLANA_METRICS_CONFIG: environment variable not found
[2020-03-02T15:20:51.518149757Z INFO  solana_metrics::metrics] datapoint: accounts_db-stores total_count=8358i
[2020-03-02T15:20:51.522542270Z INFO  solana_runtime::accounts_db] scan took 84us merge took 340ns accumulate took 7us
[2020-03-02T15:20:51.522573720Z INFO  solana_runtime::bank] hard fork at bank 2196960
[2020-03-02T15:20:51.522586580Z INFO  solana_runtime::bank] bank frozen: 2196960 hash: EKu56JJW5cgiMufuZAorwTNW33YAeuX3suXsGKvQh3ja accounts_delta: 2M4rDVkGMcPsZspHfXxW1A26kU6d6bf3XRe1hqzBpvco signature_count: 4 last_blockhash: 9xGYFG7WfBK4S9BDtiJbA7UfQ4BVqiWK8nCu7P9hZgbx
[2020-03-02T15:20:51.522608560Z INFO  solana_runtime::bank] accounts hash slot: 2196960 stats: BankHashStats { num_removed_accounts: 2, num_added_accounts: 0, num_lamports_stored: 55429440, total_data_len: 7708, num_executable_accounts: 0 }
[2020-03-02T15:20:51.522617510Z INFO  solana_ledger::snapshot_utils] bank rebuild from snapshot took 29.9s
[2020-03-02T15:20:51.542563553Z INFO  solana_ledger::blockstore_processor] processing ledger from root slot 2196960...
[2020-03-02T15:20:51.619761079Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196961i block_height=1858552i
[2020-03-02T15:20:51.625131050Z INFO  solana_runtime::accounts_db] scan took 100us merge took 170ns accumulate took 5us
[2020-03-02T15:20:51.625159441Z INFO  solana_runtime::bank] bank frozen: 2196961 hash: A9HXzimMyoipVNHpAC4h9ibcGBEzAnEFHnCjVZb5e41X accounts_delta: DSrGH9Z8wx4XouK2y5CrabogdhmkxsVmxE2fbiuRAdor signature_count: 0 last_blockhash: BmDCRmcUHFt8DhN2nTDMNC9FXVYpHbWF3xs5ebbAMQLq
[2020-03-02T15:20:51.625190301Z INFO  solana_runtime::bank] accounts hash slot: 2196961 stats: BankHashStats { num_removed_accounts: 5, num_added_accounts: 0, num_lamports_stored: 5, total_data_len: 152698, num_executable_accounts: 0 }
[2020-03-02T15:20:51.627506803Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196962i block_height=1858553i
[2020-03-02T15:20:51.630936936Z INFO  solana_runtime::accounts_db] scan took 98us merge took 230ns accumulate took 4us
[2020-03-02T15:20:51.630961587Z INFO  solana_runtime::bank] bank frozen: 2196962 hash: 9hvAvjLjW1pmB7tqzoEsV44xS7d2Zvmdwjv7WGAYAEkB accounts_delta: CmH8NvizArg32GgNy67PNcHEyT8UHSM6MrscTn7FguGo signature_count: 0 last_blockhash: 5rYTG65GCWMS3s7pwFjy5o4WinqucQQ65DATjx4b2eV4
[2020-03-02T15:20:51.630986617Z INFO  solana_runtime::bank] accounts hash slot: 2196962 stats: BankHashStats { num_removed_accounts: 5, num_added_accounts: 0, num_lamports_stored: 5, total_data_len: 152698, num_executable_accounts: 0 }
[2020-03-02T15:20:51.632035817Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196963i block_height=1858554i
[2020-03-02T15:20:51.641396477Z INFO  solana_metrics::metrics] metrics disabled: SOLANA_METRICS_CONFIG: environment variable not found
[2020-03-02T15:20:51.654716436Z INFO  solana_runtime::accounts_db] scan took 143us merge took 26us accumulate took 162us
[2020-03-02T15:20:51.654751147Z INFO  solana_runtime::bank] bank frozen: 2196963 hash: CPT1oc6khN8ZP7bVG3DCU2TQYLcv5wzudJ7QYRnTqZVW accounts_delta: CVkTZJHyiWmHdeTDQsh6Z8zGsHqUb3nLPd8MPoMrRhWM signature_count: 98 last_blockhash: F9LamkFWQyaCZkgMbDRA3bRHbaVJTqwBNoHxp2kK5o8E
[2020-03-02T15:20:51.654783087Z INFO  solana_runtime::bank] accounts hash slot: 2196963 stats: BankHashStats { num_removed_accounts: 103, num_added_accounts: 0, num_lamports_stored: 5816864488496, total_data_len: 341544, num_executable_accounts: 0 }
[2020-03-02T15:20:51.655819267Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196964i block_height=1858555i
[2020-03-02T15:20:51.708130402Z INFO  solana_runtime::accounts_db] scan took 200us merge took 6us accumulate took 180us
[2020-03-02T15:20:51.708169103Z INFO  solana_runtime::bank] bank frozen: 2196964 hash: Cxxk485kPxTtjWZumBPoaDf7tsF5CoMJ4XMVCrrp5my4 accounts_delta: 45tMhGEGACG88jj8A78GF11xQVBUoEBYR4Af3hESWv1U signature_count: 176 last_blockhash: 4m1k6UAoFk67RNqMYMmByvsT9d7XryDc1w2qjDLNyXDZ
[2020-03-02T15:20:51.708201883Z INFO  solana_runtime::bank] accounts hash slot: 2196964 stats: BankHashStats { num_removed_accounts: 165, num_added_accounts: 0, num_lamports_stored: 8463120937743, total_data_len: 430186, num_executable_accounts: 0 }
[2020-03-02T15:20:51.709286154Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196965i block_height=1858556i
[2020-03-02T15:20:51.726225358Z INFO  solana_runtime::accounts_db] scan took 132us merge took 9us accumulate took 242us
[2020-03-02T15:20:51.726260248Z INFO  solana_runtime::bank] bank frozen: 2196965 hash: 5eNySJmPxpLf4rgyt3kHypxo43MnYiSKUU7qgWJdZ4VK accounts_delta: CPVZpUTzwmCE8uJzrBWHhdgeNKR21XhWmDpktQB9GRok signature_count: 74 last_blockhash: E7UVwcTWFxZiQ8S1m4XF5pu2QSARsUZ4fDyR4etSdtBk
[2020-03-02T15:20:51.726292349Z INFO  solana_runtime::bank] accounts hash slot: 2196965 stats: BankHashStats { num_removed_accounts: 75, num_added_accounts: 0, num_lamports_stored: 3636499297344, total_data_len: 279880, num_executable_accounts: 0 }
[2020-03-02T15:20:51.727357979Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196966i block_height=1858557i
[2020-03-02T15:20:51.743409964Z INFO  solana_runtime::accounts_db] scan took 136us merge took 3us accumulate took 323us
[2020-03-02T15:20:51.743447694Z INFO  solana_runtime::bank] bank frozen: 2196966 hash: 2rTdQCMXNHED4vDrqbkJXAwmdSuamDDv8uE4HDqr7Ymx accounts_delta: AgpYNvxMyERDHbRpPB5HWTDKxmgSfmFSKLUggtGnB4VR signature_count: 86 last_blockhash: A5Z32SHXruPh1zwAYvchBNXvyqJS7C9N866MQ1RFk1mz
[2020-03-02T15:20:51.743478674Z INFO  solana_runtime::bank] accounts hash slot: 2196966 stats: BankHashStats { num_removed_accounts: 90, num_added_accounts: 0, num_lamports_stored: 3371546611186, total_data_len: 314566, num_executable_accounts: 0 }
[2020-03-02T15:20:51.744588715Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196967i block_height=1858558i
[2020-03-02T15:20:51.762041134Z INFO  solana_runtime::accounts_db] scan took 184us merge took 27us accumulate took 177us
[2020-03-02T15:20:51.762079314Z INFO  solana_runtime::bank] bank frozen: 2196967 hash: 9pwdFEGUcyppPWvhbLiiVWH6M1tCedUnKE1d1Ncag2W8 accounts_delta: G2ZWgaZLtxdCpFsPXRjpFFa2yTvAq1KS6urBSGqbBb27 signature_count: 94 last_blockhash: FggJXeoXit5sU7VbFKur2yNjUjnNV8kZ9BSWAydo4xyN
[2020-03-02T15:20:51.762100824Z INFO  solana_runtime::bank] accounts hash slot: 2196967 stats: BankHashStats { num_removed_accounts: 97, num_added_accounts: 0, num_lamports_stored: 4165610156231, total_data_len: 326128, num_executable_accounts: 0 }
[2020-03-02T15:20:51.763227735Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196968i block_height=1858559i
[2020-03-02T15:20:51.799885799Z INFO  solana_runtime::accounts_db] scan took 137us merge took 30us accumulate took 198us
[2020-03-02T15:20:51.799953560Z INFO  solana_runtime::bank] bank frozen: 2196968 hash: Bqiz6gJFQ5mQiXd8mKkZwZSuy29Bh2trDjJnY9y5YiVk accounts_delta: GPPVyYfjvBXpaipRAZqBo9J2fb6Hhkd517ux4FJxZgaK signature_count: 270 last_blockhash: E2zDUHNCL7Wy4BKHhskrgYogYhBVmLtBf85JsXzxETNw
[2020-03-02T15:20:51.799986260Z INFO  solana_runtime::bank] accounts hash slot: 2196968 stats: BankHashStats { num_removed_accounts: 253, num_added_accounts: 0, num_lamports_stored: 13483662649264, total_data_len: 588200, num_executable_accounts: 0 }
[2020-03-02T15:20:51.801059020Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196969i block_height=1858560i
[2020-03-02T15:20:51.808817256Z INFO  solana_runtime::accounts_db] scan took 127us merge took 630ns accumulate took 177us
[2020-03-02T15:20:51.808857196Z INFO  solana_runtime::bank] bank frozen: 2196969 hash: 8gMM8riA6jcsRE8acM2PjcZimJgJ4XQmoc9Mb6XzRhZF accounts_delta: 5otiXDKyA3dADZixRRHUx5JET88jXVRRkwJJkew83A9 signature_count: 20 last_blockhash: HknEZ3h24MWk14fW1pyYaWdR4m5LpGZLtsuR2wsJVrcL
[2020-03-02T15:20:51.808887996Z INFO  solana_runtime::bank] accounts hash slot: 2196969 stats: BankHashStats { num_removed_accounts: 24, num_added_accounts: 0, num_lamports_stored: 543962485587, total_data_len: 187384, num_executable_accounts: 0 }
[2020-03-02T15:20:51.809970067Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196970i block_height=1858561i
[2020-03-02T15:20:51.822771740Z INFO  solana_runtime::accounts_db] scan took 141us merge took 26us accumulate took 232us
[2020-03-02T15:20:51.822795461Z INFO  solana_runtime::bank] bank frozen: 2196970 hash: DcPAeJRmfCVEr1WNkrs6zTXuyhxvwEo7gzaJ1hRSMR4X accounts_delta: 2LSwf8PHftx2bJMyUAMU8xXX39CZS1bqUb16ySthAf7s signature_count: 90 last_blockhash: F9JSksL45iJSefoVhSaGrj68q1D4RQ7cQzqEFvRda8pc
[2020-03-02T15:20:51.822825701Z INFO  solana_runtime::bank] accounts hash slot: 2196970 stats: BankHashStats { num_removed_accounts: 95, num_added_accounts: 0, num_lamports_stored: 4317846573484, total_data_len: 326128, num_executable_accounts: 0 }
[2020-03-02T15:20:51.823810310Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196971i block_height=1858562i
[2020-03-02T15:20:51.850622989Z INFO  solana_runtime::accounts_db] scan took 128us merge took 33us accumulate took 177us
[2020-03-02T15:20:51.850662029Z INFO  solana_runtime::bank] bank frozen: 2196971 hash: BUBQypHUBj3JdHTDBhr9vynSN44mrmKFC7CmDV82KJ6M accounts_delta: J4Yo9AR2cT6bf9BNzGgQZkQxKoWwJkY3P7NWa8LYFcsf signature_count: 152 last_blockhash: 811Bkq6NA3j7qmsxRHhic7Wbwv1eC78fvaXY31PPwrri
[2020-03-02T15:20:51.850702320Z INFO  solana_runtime::bank] accounts hash slot: 2196971 stats: BankHashStats { num_removed_accounts: 155, num_added_accounts: 0, num_lamports_stored: 6748570957219, total_data_len: 437894, num_executable_accounts: 0 }
[2020-03-02T15:20:51.851793511Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196972i block_height=1858563i
[2020-03-02T15:20:51.876262407Z INFO  solana_runtime::accounts_db] scan took 175us merge took 34us accumulate took 235us
[2020-03-02T15:20:51.876308427Z INFO  solana_runtime::bank] bank frozen: 2196972 hash: 2BnraaRi4FjzbymGxQ5Rm5P6FVhexq1RVokCENo2k4hV accounts_delta: 2vsAzDdPNeYa5z6riQUoL7XwUQg19ctSghuPnQzAsyHh signature_count: 158 last_blockhash: GDSZW88WppK5asiyspSWgKe14SCNNAei4WRS3EhA1KMU
[2020-03-02T15:20:51.876341238Z INFO  solana_runtime::bank] accounts hash slot: 2196972 stats: BankHashStats { num_removed_accounts: 158, num_added_accounts: 0, num_lamports_stored: 8616171467346, total_data_len: 437894, num_executable_accounts: 0 }
[2020-03-02T15:20:51.877408098Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196973i block_height=1858564i
[2020-03-02T15:20:51.881933722Z INFO  solana_runtime::accounts_db] scan took 92us merge took 300ns accumulate took 8us
[2020-03-02T15:20:51.881963052Z INFO  solana_runtime::bank] bank frozen: 2196973 hash: G9ojeqnCcMVGxg2oydyf3CTYvxZZ84gPriE6TQRjMroj accounts_delta: FidRqEYMSqNSx3A6iwdMc21Nyzakb2157t3sdUck47c3 signature_count: 6 last_blockhash: Bid3hgEHwZVRjyfpbp9JWMD83goAH58cgLENWsideEsE
[2020-03-02T15:20:51.881991873Z INFO  solana_runtime::bank] accounts hash slot: 2196973 stats: BankHashStats { num_removed_accounts: 11, num_added_accounts: 0, num_lamports_stored: 11995100165, total_data_len: 164260, num_executable_accounts: 0 }
[2020-03-02T15:20:51.882879581Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196974i block_height=1858565i
[2020-03-02T15:20:51.887565156Z INFO  solana_runtime::accounts_db] scan took 101us merge took 340ns accumulate took 11us
[2020-03-02T15:20:51.887594717Z INFO  solana_runtime::bank] bank frozen: 2196974 hash: 7yqxa92NPLgCmiDCwHKSVtu3vAvLi8VWwXTaRpyWYLii accounts_delta: EwnuSP7a8Gh8RJ1DmgY5ieXmmJDvxQaDuMG6PpwiVKY5 signature_count: 10 last_blockhash: 9QSGg4WgsAGKDtA2d5G1yRhSRDx2L7ziM7KPzhSUGFPs
[2020-03-02T15:20:51.887609257Z INFO  solana_runtime::bank] accounts hash slot: 2196974 stats: BankHashStats { num_removed_accounts: 14, num_added_accounts: 0, num_lamports_stored: 274962485599, total_data_len: 168114, num_executable_accounts: 0 }
[2020-03-02T15:20:51.888722748Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196975i block_height=1858566i
[2020-03-02T15:20:51.900822104Z INFO  solana_runtime::accounts_db] scan took 116us merge took 6us accumulate took 157us
[2020-03-02T15:20:51.900852334Z INFO  solana_runtime::bank] bank frozen: 2196975 hash: 38PkpL28feUgd3Cbdjk13X4pckyYuBHeRcmUx3mVtEyW accounts_delta: 5zfKcDaNfqEYo83gaLVAWz8F7zfapnrixK5AoMKjTyBe signature_count: 54 last_blockhash: 8kFSR3ydBwqDjaqVmQHYgdrUD7hqFtz5gbD1WzwHDxyp
[2020-03-02T15:20:51.900888245Z INFO  solana_runtime::bank] accounts hash slot: 2196975 stats: BankHashStats { num_removed_accounts: 54, num_added_accounts: 0, num_lamports_stored: 673812427777, total_data_len: 237486, num_executable_accounts: 0 }
[2020-03-02T15:20:51.901896124Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196976i block_height=1858567i
[2020-03-02T15:20:51.946302304Z INFO  solana_runtime::accounts_db] scan took 168us merge took 34us accumulate took 327us
[2020-03-02T15:20:51.946333754Z INFO  solana_runtime::bank] bank frozen: 2196976 hash: AUhoVhKgMcsqnHv5wcBukRLCWEdPYQfwNvZyD5q7TM3D accounts_delta: 3FAJ3gQwJFsUioLq4aZaA3z75k7en9jYKuS6uW19V1vx signature_count: 318 last_blockhash: 6gdp5sTreo73J11N4jZFQU7TxmmduH8Y32AWeC3mF1Xn
[2020-03-02T15:20:51.946372625Z INFO  solana_runtime::bank] accounts hash slot: 2196976 stats: BankHashStats { num_removed_accounts: 296, num_added_accounts: 0, num_lamports_stored: 17631241875235, total_data_len: 661426, num_executable_accounts: 0 }
[2020-03-02T15:20:51.947392444Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196977i block_height=1858568i
[2020-03-02T15:20:51.959232898Z INFO  solana_runtime::accounts_db] scan took 129us merge took 26us accumulate took 343us
[2020-03-02T15:20:51.959262429Z INFO  solana_runtime::bank] bank frozen: 2196977 hash: FhcBR1UEQRWNphmS8wso35qqPvddXS7LYcR3dC7j7TrY accounts_delta: G9GaEASZiYUm8Vpr42Er5pY1MPZevHXNu8RmxufaVdGM signature_count: 64 last_blockhash: 6soYEgCgrFgcVT5GZCnTVJNquvyEc2m13AJwC8r9ivdN
[2020-03-02T15:20:51.959287229Z INFO  solana_runtime::bank] accounts hash slot: 2196977 stats: BankHashStats { num_removed_accounts: 69, num_added_accounts: 0, num_lamports_stored: 2350787928790, total_data_len: 276026, num_executable_accounts: 0 }
[2020-03-02T15:20:51.960311519Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196978i block_height=1858569i
[2020-03-02T15:20:51.976061031Z INFO  solana_runtime::accounts_db] scan took 153us merge took 25us accumulate took 291us
[2020-03-02T15:20:51.976093412Z INFO  solana_runtime::bank] bank frozen: 2196978 hash: 9Ha6MXpv1784aoMqLFV1AXMNbTgNSrM3GFZtMWLar2kk accounts_delta: FgousJRUciSNk1nh8bPkT7RUkggK9avakogsyc2YqUis signature_count: 76 last_blockhash: 464UAoorQhUFKAsNgZDX3Gy5z5v6Wrcv1c5n48qqD8Zs
[2020-03-02T15:20:51.976126882Z INFO  solana_runtime::bank] accounts hash slot: 2196978 stats: BankHashStats { num_removed_accounts: 80, num_added_accounts: 0, num_lamports_stored: 4292644455071, total_data_len: 295296, num_executable_accounts: 0 }
[2020-03-02T15:20:51.976935720Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196979i block_height=1858570i
[2020-03-02T15:20:51.996203526Z INFO  solana_runtime::accounts_db] scan took 141us merge took 20us accumulate took 366us
[2020-03-02T15:20:51.996236335Z INFO  solana_runtime::bank] bank frozen: 2196979 hash: BxeXxRDomkrLvvp2sAMvozvjmqrUEyxCTeacvAGbAUF4 accounts_delta: 34ocJytyHtB5yssHBZvwqjZQPzvzUaQpfMwWfxuEwZkY signature_count: 128 last_blockhash: 4Qf1Ns1thymUz4qz1LthpMbTvV5ZUCZ8QYLdcB1SNB7H
[2020-03-02T15:20:51.996263476Z INFO  solana_runtime::bank] accounts hash slot: 2196979 stats: BankHashStats { num_removed_accounts: 133, num_added_accounts: 0, num_lamports_stored: 6278621487050, total_data_len: 399354, num_executable_accounts: 0 }
[2020-03-02T15:20:51.997089324Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196980i block_height=1858571i
[2020-03-02T15:20:52.012536543Z INFO  solana_metrics::counter] COUNTER:{"name": "vote-native", "counts": 1001, "samples": 1000,  "now": 1583162452012, "events": 1}
[2020-03-02T15:20:52.018824304Z INFO  solana_runtime::accounts_db] scan took 128us merge took 16us accumulate took 200us
[2020-03-02T15:20:52.018857924Z INFO  solana_runtime::bank] bank frozen: 2196980 hash: D3v4ptHjT5TZLsXVGV7mrcDUNT9MgEKhPckWUAvNtzDs accounts_delta: YqDD4Eh17FgEQkoYSpVN2zxG4PZByr5rbHdAq5d7i3M signature_count: 160 last_blockhash: Hpkw9QywjrxERRYECKiFGjdQb6cJNR3UffmVnNTT9dwY
[2020-03-02T15:20:52.018888994Z INFO  solana_runtime::bank] accounts hash slot: 2196980 stats: BankHashStats { num_removed_accounts: 163, num_added_accounts: 0, num_lamports_stored: 7452482913203, total_data_len: 453310, num_executable_accounts: 0 }
[2020-03-02T15:20:52.019994345Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196981i block_height=1858572i
[2020-03-02T15:20:52.028553448Z INFO  solana_runtime::accounts_db] scan took 132us merge took 17us accumulate took 168us
[2020-03-02T15:20:52.028591088Z INFO  solana_runtime::bank] bank frozen: 2196981 hash: 2ssbZmpHemAg6Kxak1ZtpNZREYQSGaTdx3gB6gatAmwx accounts_delta: 9dxTnsSmkgAotg6EYpwVuDPYbbmfG5gzReiNxytrXxBB signature_count: 40 last_blockhash: DnHA9PefYWejormELVubK4p9NGWqwQpMGtx38gZ86trX
[2020-03-02T15:20:52.028620338Z INFO  solana_runtime::bank] accounts hash slot: 2196981 stats: BankHashStats { num_removed_accounts: 45, num_added_accounts: 0, num_lamports_stored: 1490928186470, total_data_len: 229778, num_executable_accounts: 0 }
[2020-03-02T15:20:52.029758640Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196982i block_height=1858573i
[2020-03-02T15:20:52.044038637Z INFO  solana_runtime::accounts_db] scan took 126us merge took 25us accumulate took 275us
[2020-03-02T15:20:52.044065648Z INFO  solana_runtime::bank] bank frozen: 2196982 hash: 5XZbrGn2rWnMBbE1uqtBPBX9rG7R5ZR573edHmLoiwBR accounts_delta: Bm1aBRfzvZc4SXRFN7q9sLaqNHdaAAa9F6pBKVdRV6Ww signature_count: 72 last_blockhash: 6tKfQwT1NL29HSRAhZdEY81QGvE7tNYQvWv3w6d7qqRo
[2020-03-02T15:20:52.044100719Z INFO  solana_runtime::bank] accounts hash slot: 2196982 stats: BankHashStats { num_removed_accounts: 77, num_added_accounts: 0, num_lamports_stored: 4001755314207, total_data_len: 291442, num_executable_accounts: 0 }
[2020-03-02T15:20:52.045194109Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196983i block_height=1858574i
[2020-03-02T15:20:52.058003992Z INFO  solana_runtime::accounts_db] scan took 130us merge took 15us accumulate took 313us
[2020-03-02T15:20:52.058041043Z INFO  solana_runtime::bank] bank frozen: 2196983 hash: F1UqRxiPyTjKUWpkeufq6313ZcM2XpLafgFotEqvU57C accounts_delta: 3zsdNkirUPawGTKRndfsZ1bSbKjeYtf8kNRk15SNutVW signature_count: 90 last_blockhash: 3siecsDFZC39pR8ureqsjMgrRSypZJZH88pAfQJrmwBN
[2020-03-02T15:20:52.058059133Z INFO  solana_runtime::bank] accounts hash slot: 2196983 stats: BankHashStats { num_removed_accounts: 95, num_added_accounts: 0, num_lamports_stored: 4335846573486, total_data_len: 326128, num_executable_accounts: 0 }
[2020-03-02T15:20:52.059086553Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196984i block_height=1858575i
[2020-03-02T15:20:52.090627287Z INFO  solana_runtime::accounts_db] scan took 148us merge took 39us accumulate took 232us
[2020-03-02T15:20:52.090669158Z INFO  solana_runtime::bank] bank frozen: 2196984 hash: 8EcGriUcj3bXwbMWrjdw6GACCWXoYMibX9PQ9oot9xbK accounts_delta: GUpRCFCrMERTzgTUxUzKQWMvadzMXRCuMqnhPaTEyPW5 signature_count: 208 last_blockhash: dptWqU3NoGde51y18qRpMqsVKpRfLe3b9Q11W1TEZzV
[2020-03-02T15:20:52.090701108Z INFO  solana_runtime::bank] accounts hash slot: 2196984 stats: BankHashStats { num_removed_accounts: 193, num_added_accounts: 0, num_lamports_stored: 9907688832859, total_data_len: 476434, num_executable_accounts: 0 }
[2020-03-02T15:20:52.091774389Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196985i block_height=1858576i
[2020-03-02T15:20:52.096927938Z INFO  solana_runtime::accounts_db] scan took 100us merge took 380ns accumulate took 9us
[2020-03-02T15:20:52.096954439Z INFO  solana_runtime::bank] bank frozen: 2196985 hash: 9JuFfrkoo4gWMLcogrNP3QuQhy5ehbAijvMVjYb3q4Vx accounts_delta: 2uVtNh2vYVNhkvhjYUN8oKQCmgapvLfTJ3V46wQTAthJ signature_count: 6 last_blockhash: CrmMYW46g7d1YGA6JbiRabkp4svmx8hRh9E4LVHzGQ6B
[2020-03-02T15:20:52.096982979Z INFO  solana_runtime::bank] accounts hash slot: 2196985 stats: BankHashStats { num_removed_accounts: 10, num_added_accounts: 0, num_lamports_stored: 989967385431, total_data_len: 160406, num_executable_accounts: 0 }
[2020-03-02T15:20:52.098016539Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196986i block_height=1858577i
[2020-03-02T15:20:52.105466222Z INFO  solana_runtime::accounts_db] scan took 105us merge took 549ns accumulate took 161us
[2020-03-02T15:20:52.105501911Z INFO  solana_runtime::bank] bank frozen: 2196986 hash: 7sXDM9tBtqQhJregvYmcAeu2mDV1V25SkV9vJcuf8kYf accounts_delta: J5ipoZXwL6UCphNYNEEGoWAfnCNHirE4xZQTJU6BdJJZ signature_count: 28 last_blockhash: GUY48zmkBir6BkKMMYpyfVLh8kYzh42hB541ap5Gwr4Y
[2020-03-02T15:20:52.105536132Z INFO  solana_runtime::bank] accounts hash slot: 2196986 stats: BankHashStats { num_removed_accounts: 33, num_added_accounts: 0, num_lamports_stored: 1701970600940, total_data_len: 206654, num_executable_accounts: 0 }
[2020-03-02T15:20:52.106583502Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196987i block_height=1858578i
[2020-03-02T15:20:52.126376763Z INFO  solana_runtime::accounts_db] scan took 138us merge took 25us accumulate took 178us
[2020-03-02T15:20:52.126417214Z INFO  solana_runtime::bank] bank frozen: 2196987 hash: wn6At4Ev3dNHKvfpXe5zDkTfFnWEh9dwWvdwxdZstzm accounts_delta: C7G2eLKF6Ko3okp7C9xpA5G7nXPEB14kMfU1t869btQi signature_count: 118 last_blockhash: 78z9QYBYb1KLcW9p28Ub4twFQ4vsqAcPQ78vQQxnBQN1
[2020-03-02T15:20:52.126451354Z INFO  solana_runtime::bank] accounts hash slot: 2196987 stats: BankHashStats { num_removed_accounts: 121, num_added_accounts: 0, num_lamports_stored: 4681721014863, total_data_len: 372376, num_executable_accounts: 0 }
[2020-03-02T15:20:52.127587595Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196988i block_height=1858579i
[2020-03-02T15:20:52.163353170Z INFO  solana_runtime::accounts_db] scan took 148us merge took 18us accumulate took 233us
[2020-03-02T15:20:52.163391880Z INFO  solana_runtime::bank] bank frozen: 2196988 hash: FwuKtLyPu9D8A9bLF9moJ2Uajwd2r1YeEb3ExyDt91Y7 accounts_delta: 5E2FpKbTn1xcVA6MHdr9o4riXFDk7KRxAm8eYV1p8c4n signature_count: 238 last_blockhash: FV4uitNDGsHqWEFjLuf5i4UddMkGRqFFfbkGLX5WV99v
[2020-03-02T15:20:52.163415350Z INFO  solana_runtime::bank] accounts hash slot: 2196988 stats: BankHashStats { num_removed_accounts: 239, num_added_accounts: 0, num_lamports_stored: 12702073470258, total_data_len: 595908, num_executable_accounts: 0 }
[2020-03-02T15:20:52.164503742Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196989i block_height=1858580i
[2020-03-02T15:20:52.173195835Z INFO  solana_runtime::accounts_db] scan took 99us merge took 13us accumulate took 193us
[2020-03-02T15:20:52.173253326Z INFO  solana_runtime::bank] bank frozen: 2196989 hash: H9X4G7oSr2fUHN4BxUCvbDkdX2Q1UwrPsTV9Zpu83PGP accounts_delta: CgYquuHxTsYYd4hPgbrqhLPiHcmKqHcU9xw5xGaH32gT signature_count: 40 last_blockhash: XLVtG3KX4VExh7VjvhNNwLAjkr64J2PSEBMWTFrs23X
[2020-03-02T15:20:52.173287537Z INFO  solana_runtime::bank] accounts hash slot: 2196989 stats: BankHashStats { num_removed_accounts: 44, num_added_accounts: 0, num_lamports_stored: 275766798205, total_data_len: 225924, num_executable_accounts: 0 }
[2020-03-02T15:20:52.174385506Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196990i block_height=1858581i
[2020-03-02T15:20:52.193201799Z INFO  solana_runtime::accounts_db] scan took 170us merge took 28us accumulate took 336us
[2020-03-02T15:20:52.193239589Z INFO  solana_runtime::bank] bank frozen: 2196990 hash: G6BehQHWEi8AGSyAWe9MiRkF53WVVcj72qGkyfMbh4X6 accounts_delta: 7jC581DCqwnKRfeEqZA12rxdx1EM1BjoSQL1GQa87Mz5 signature_count: 116 last_blockhash: 3eAA8g3op5Prpu3fNKtu9vGchv5tZ6ZGo1P2zyoRcyUT
[2020-03-02T15:20:52.193280320Z INFO  solana_runtime::bank] accounts hash slot: 2196990 stats: BankHashStats { num_removed_accounts: 121, num_added_accounts: 0, num_lamports_stored: 6316663901307, total_data_len: 376230, num_executable_accounts: 0 }
[2020-03-02T15:20:52.194353320Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196991i block_height=1858582i
[2020-03-02T15:20:52.207078013Z INFO  solana_runtime::accounts_db] scan took 170us merge took 25us accumulate took 265us
[2020-03-02T15:20:52.207114903Z INFO  solana_runtime::bank] bank frozen: 2196991 hash: 4CYH4N5sCLWDxfVZVGjtFCYZyZGfP7U3C2417WKMejtx accounts_delta: 23pWhVjQFLP7yfaHPAebDiSu7B1yAUvn9QxAoWtE7eTW signature_count: 74 last_blockhash: 6LRRFqAERPGjYZ163cCkx5mDSxireaVEoTN9R6oW7Vcc
[2020-03-02T15:20:52.207146423Z INFO  solana_runtime::bank] accounts hash slot: 2196991 stats: BankHashStats { num_removed_accounts: 79, num_added_accounts: 0, num_lamports_stored: 3918760213797, total_data_len: 295296, num_executable_accounts: 0 }
[2020-03-02T15:20:52.208333665Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196992i block_height=1858583i
[2020-03-02T15:20:52.238481296Z INFO  solana_runtime::accounts_db] scan took 163us merge took 19us accumulate took 222us
[2020-03-02T15:20:52.238539656Z INFO  solana_runtime::bank] bank frozen: 2196992 hash: DGofYZgj5VXyvuct9vCk6UsNiT7CSgH2uXrUy5dQxaF8 accounts_delta: 5GKKZ7QVaG5PhmXBhKQM1UuRFw8bgsHGuaGHytPZKMSt signature_count: 192 last_blockhash: 9j5uNz6YXy9NMbUTm1DN1DNMBhDKBXfWkktHHgFD6YaM
[2020-03-02T15:20:52.238572567Z INFO  solana_runtime::bank] accounts hash slot: 2196992 stats: BankHashStats { num_removed_accounts: 181, num_added_accounts: 0, num_lamports_stored: 8842962764567, total_data_len: 461018, num_executable_accounts: 0 }
[2020-03-02T15:20:52.239701608Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196993i block_height=1858584i
[2020-03-02T15:20:52.245419083Z INFO  solana_runtime::accounts_db] scan took 104us merge took 390ns accumulate took 11us
[2020-03-02T15:20:52.245456424Z INFO  solana_runtime::bank] bank frozen: 2196993 hash: Etyz6VoEreB6oss4cbXpFwGQrABkKAC8gMVnJAPGdLzS accounts_delta: BAwBtYKTqPynbRfcRDoJjcS6ecgk45bn5MFCLV37WLFK signature_count: 10 last_blockhash: 287nSUckKgse4md57rWUpHShK2SDkhTKXR8P5V3dbzUe
[2020-03-02T15:20:52.245481893Z INFO  solana_runtime::bank] accounts hash slot: 2196993 stats: BankHashStats { num_removed_accounts: 15, num_added_accounts: 0, num_lamports_stored: 514990200319, total_data_len: 171968, num_executable_accounts: 0 }
[2020-03-02T15:20:52.246498064Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196994i block_height=1858585i
[2020-03-02T15:20:52.257104306Z INFO  solana_runtime::accounts_db] scan took 128us merge took 27us accumulate took 317us
[2020-03-02T15:20:52.257139257Z INFO  solana_runtime::bank] bank frozen: 2196994 hash: CQjBKcKRDci6iabNcrmbNWkTdqW7JNSXy7pRT4jfT7fb accounts_delta: E8jY9jDhkEviu4vBWSHsgCaNA3HhvdZ7KPWE73ozVeNR signature_count: 74 last_blockhash: 5aXBDEMCUwBw27w3wF5qtYyiee5tXLEsdTRqk5g6HSuZ
[2020-03-02T15:20:52.257155257Z INFO  solana_runtime::bank] accounts hash slot: 2196994 stats: BankHashStats { num_removed_accounts: 79, num_added_accounts: 0, num_lamports_stored: 3282866172854, total_data_len: 295296, num_executable_accounts: 0 }
[2020-03-02T15:20:52.258270467Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196995i block_height=1858586i
[2020-03-02T15:20:52.274926788Z INFO  solana_runtime::accounts_db] scan took 152us merge took 7us accumulate took 355us
[2020-03-02T15:20:52.274961509Z INFO  solana_runtime::bank] bank frozen: 2196995 hash: 8noFMDHr3HnutTMkTgu7fbsU4CeBN5TyYLkAhJSrc47B accounts_delta: CGBWxXV8Y98Zg4HStAnE4igpNLu6DAaUjQBwozQctFFg signature_count: 94 last_blockhash: EYD5E3GrjGSkaTdANwRWdXD1h2JGbTQUHAWsgj4rYXhT
[2020-03-02T15:20:52.274988668Z INFO  solana_runtime::bank] accounts hash slot: 2196995 stats: BankHashStats { num_removed_accounts: 99, num_added_accounts: 0, num_lamports_stored: 4431693300372, total_data_len: 333836, num_executable_accounts: 0 }
[2020-03-02T15:20:52.276061559Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196996i block_height=1858587i
[2020-03-02T15:20:52.310087828Z INFO  solana_runtime::accounts_db] scan took 177us merge took 37us accumulate took 194us
[2020-03-02T15:20:52.310221679Z INFO  solana_runtime::bank] bank frozen: 2196996 hash: NyPmifAkHbmPWUW6GhBfMqbagQdLST95ScR79ZNHT72 accounts_delta: 5P5ayaeKNmYzVaQzJ7GtumFNdwpdVCyCLK1oJrwozuC9 signature_count: 226 last_blockhash: FjV88M5MR1gV4XKQtQ5KVEDMuWtWNugSdts9AFvsqb5Z
[2020-03-02T15:20:52.310267700Z INFO  solana_runtime::bank] accounts hash slot: 2196996 stats: BankHashStats { num_removed_accounts: 216, num_added_accounts: 0, num_lamports_stored: 11984807807069, total_data_len: 530390, num_executable_accounts: 0 }
[2020-03-02T15:20:52.311274449Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196997i block_height=1858588i
[2020-03-02T15:20:52.321343536Z INFO  solana_runtime::accounts_db] scan took 111us merge took 15us accumulate took 268us
[2020-03-02T15:20:52.321380197Z INFO  solana_runtime::bank] bank frozen: 2196997 hash: AqdiBvLMGiqSdknh2vABgZtDM7n7cGfqtEUZn3hB3Xd8 accounts_delta: F6JTqfM16EFZbpjdg734ZLsUq9twrtUxjFbsScKrHLdH signature_count: 42 last_blockhash: Bvn1ZZRbp4MMeQrb5jWnZna4ceYwtN2sjmGvZi4nWTQJ
[2020-03-02T15:20:52.321410998Z INFO  solana_runtime::bank] accounts hash slot: 2196997 stats: BankHashStats { num_removed_accounts: 47, num_added_accounts: 0, num_lamports_stored: 1921960801266, total_data_len: 233632, num_executable_accounts: 0 }
[2020-03-02T15:20:52.322456747Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196998i block_height=1858589i
[2020-03-02T15:20:52.335428483Z INFO  solana_runtime::accounts_db] scan took 148us merge took 22us accumulate took 176us
[2020-03-02T15:20:52.335464893Z INFO  solana_runtime::bank] bank frozen: 2196998 hash: 2ePphHQmksn1y4wj2raaaqwvf3fi93Tejv43vcxNiAs7 accounts_delta: 8LW8DZY6UBRXYAfp3BcLD7CwjUKyGM6HgpTnXSjg6E2A signature_count: 76 last_blockhash: Dtsnepmr8iBUbznJ45C38LfbkQsTSGJkERKJiF6SbxQX
[2020-03-02T15:20:52.335501793Z INFO  solana_runtime::bank] accounts hash slot: 2196998 stats: BankHashStats { num_removed_accounts: 81, num_added_accounts: 0, num_lamports_stored: 2654717799567, total_data_len: 299150, num_executable_accounts: 0 }
[2020-03-02T15:20:52.336559713Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2196999i block_height=1858590i
[2020-03-02T15:20:52.351317837Z INFO  solana_runtime::accounts_db] scan took 161us merge took 10us accumulate took 162us
[2020-03-02T15:20:52.351356837Z INFO  solana_runtime::bank] bank frozen: 2196999 hash: 6KZzd4NjSXLFGLr7kqbbXhSb5kpvNMPuAva1WCWyssGg accounts_delta: B67MPF47TuSL4quL3hU5MgA8183JGsDhsRXm8wWsV4mM signature_count: 94 last_blockhash: BVw6juZ8X6yWtfnXTtQuBcsWhhFQkMtmo2TfyAVWkC1j
[2020-03-02T15:20:52.351400297Z INFO  solana_runtime::bank] accounts hash slot: 2196999 stats: BankHashStats { num_removed_accounts: 99, num_added_accounts: 0, num_lamports_stored: 4812637870904, total_data_len: 333836, num_executable_accounts: 0 }
[2020-03-02T15:20:52.352519417Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197000i block_height=1858591i
[2020-03-02T15:20:52.370490262Z INFO  solana_metrics::counter] COUNTER:{"name": "vote-native", "counts": 2001, "samples": 2000,  "now": 1583162452370, "events": 1}
[2020-03-02T15:20:52.380004473Z INFO  solana_runtime::accounts_db] scan took 145us merge took 41us accumulate took 421us
[2020-03-02T15:20:52.380038984Z INFO  solana_runtime::bank] bank frozen: 2197000 hash: 25nRZTNmKZ8ekezqwt1HExSUdhdHuya6puxoS3k5hATM accounts_delta: GoWEirbxgzoyaLdGejC8BJcHr4Nz7HukBk5AtkUeX914 signature_count: 178 last_blockhash: WLjoME218PzTKRCFtUZ3tVtQ1YtDYqXZjh3qnffLo78
[2020-03-02T15:20:52.380064414Z INFO  solana_runtime::bank] accounts hash slot: 2197000 stats: BankHashStats { num_removed_accounts: 181, num_added_accounts: 0, num_lamports_stored: 8727554573361, total_data_len: 487996, num_executable_accounts: 0 }
[2020-03-02T15:20:52.381150654Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197001i block_height=1858592i
[2020-03-02T15:20:52.390061831Z INFO  solana_runtime::accounts_db] scan took 116us merge took 10us accumulate took 200us
[2020-03-02T15:20:52.390098731Z INFO  solana_runtime::bank] bank frozen: 2197001 hash: HuefqXsHtbTGnu5nt3bsYoGZE9tGT5qSrrK4xCnWzNB2 accounts_delta: 52ZLR8RdMHYbVgbEptMUBTJMJ5tmHbSTesdMJCh7iFjT signature_count: 28 last_blockhash: Ca7F7nr3BZSFZPCGQRAHsEQkMRc5USRgTgFkVzDDFV9A
[2020-03-02T15:20:52.390133852Z INFO  solana_runtime::bank] accounts hash slot: 2197001 stats: BankHashStats { num_removed_accounts: 32, num_added_accounts: 0, num_lamports_stored: 1136809212463, total_data_len: 202800, num_executable_accounts: 0 }
[2020-03-02T15:20:52.391202871Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197002i block_height=1858593i
[2020-03-02T15:20:52.403499831Z INFO  solana_runtime::accounts_db] scan took 147us merge took 29us accumulate took 235us
[2020-03-02T15:20:52.403550391Z INFO  solana_runtime::bank] bank frozen: 2197002 hash: 4ZoXYqXHmgJcnpKPAqdLCkYRALbDZkgfmX8k1mRa8EBb accounts_delta: CBceq2tJjhwhHjEdudqdd9fuiLZzjRCbdjcB6rYKXMtr signature_count: 88 last_blockhash: 2P8wDEUgmfw8df4XxpJLTLHz53v3rEjJJz8nbnePKbu7
[2020-03-02T15:20:52.403569801Z INFO  solana_runtime::bank] accounts hash slot: 2197002 stats: BankHashStats { num_removed_accounts: 93, num_added_accounts: 0, num_lamports_stored: 4899707999906, total_data_len: 322274, num_executable_accounts: 0 }
[2020-03-02T15:20:52.404682332Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197003i block_height=1858594i
[2020-03-02T15:20:52.422047389Z INFO  solana_runtime::accounts_db] scan took 151us merge took 15us accumulate took 179us
[2020-03-02T15:20:52.422084590Z INFO  solana_runtime::bank] bank frozen: 2197003 hash: 49sbi6kdDWr324odp8YckhWNpMxkkoXd8nYa6TzDwADy accounts_delta: HZWsDxz3dMLYyrvbkXmVWLB746EWvJfAh9GneP7td7fa signature_count: 104 last_blockhash: CpxKVnQTJCAzDKb4A2f6eyBbr9PG9TmFuL6CT3Mbe58G
[2020-03-02T15:20:52.422110630Z INFO  solana_runtime::bank] accounts hash slot: 2197003 stats: BankHashStats { num_removed_accounts: 109, num_added_accounts: 0, num_lamports_stored: 4372836773800, total_data_len: 353106, num_executable_accounts: 0 }
[2020-03-02T15:20:52.422976649Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197004i block_height=1858595i
[2020-03-02T15:20:52.444652407Z INFO  solana_runtime::accounts_db] scan took 128us merge took 30us accumulate took 197us
[2020-03-02T15:20:52.444690138Z INFO  solana_runtime::bank] bank frozen: 2197004 hash: 5q5T51Rg6ECWy5MuBy93pvtyLaaHsyXzp2he5AUTSj5u accounts_delta: 4fFCVes4rPsa8nMyS6QGb5ZPM6dYrqKMrg5zhn6T9WpS signature_count: 112 last_blockhash: VWrbehV6b3YuXTuHCWvEbV5Pfk4vE9VCQK29jNqvP8Q
[2020-03-02T15:20:52.444724028Z INFO  solana_runtime::bank] accounts hash slot: 2197004 stats: BankHashStats { num_removed_accounts: 117, num_added_accounts: 0, num_lamports_stored: 6412540027217, total_data_len: 368522, num_executable_accounts: 0 }
[2020-03-02T15:20:52.445595807Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197005i block_height=1858596i
[2020-03-02T15:20:52.458827395Z INFO  solana_runtime::accounts_db] scan took 132us merge took 33us accumulate took 228us
[2020-03-02T15:20:52.458865325Z INFO  solana_runtime::bank] bank frozen: 2197005 hash: UeRvkLZkNa4jwmpm5oY815ypCVjWfQD4E1jGqPfxnaS accounts_delta: 4ZxjtPdvXmHLAaknjKLQA6XKUni5rLSh5z3UJvL3URME signature_count: 78 last_blockhash: EjW8ar1k1ZmFRm7qYTEo6rBmss7Me8ve8A8CDx21FT5k
[2020-03-02T15:20:52.458896995Z INFO  solana_runtime::bank] accounts hash slot: 2197005 stats: BankHashStats { num_removed_accounts: 83, num_added_accounts: 0, num_lamports_stored: 3307871072686, total_data_len: 303004, num_executable_accounts: 0 }
[2020-03-02T15:20:52.459901965Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197006i block_height=1858597i
[2020-03-02T15:20:52.467701170Z INFO  solana_runtime::accounts_db] scan took 114us merge took 8us accumulate took 147us
[2020-03-02T15:20:52.467739941Z INFO  solana_runtime::bank] bank frozen: 2197006 hash: AfSQ81sc7KnoQfS5yRJk3bxsZNY6mhCPSchJxZem8EzX accounts_delta: 2Uob4iMyzZqjbB2N2Voym6hWr6BoeHcmkHK1jYfADJ4Z signature_count: 20 last_blockhash: 8nzM2eoAH63DVessTKaAtPt9ywz2RRQozAyy6XpV9mHu
[2020-03-02T15:20:52.467788901Z INFO  solana_runtime::bank] accounts hash slot: 2197006 stats: BankHashStats { num_removed_accounts: 25, num_added_accounts: 0, num_lamports_stored: 1013827127504, total_data_len: 191238, num_executable_accounts: 0 }
[2020-03-02T15:20:52.468853601Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197007i block_height=1858598i
[2020-03-02T15:20:52.474123272Z INFO  solana_runtime::accounts_db] scan took 91us merge took 160ns accumulate took 8us
[2020-03-02T15:20:52.474153703Z INFO  solana_runtime::bank] bank frozen: 2197007 hash: 98uq9dDZn5p7zijBN7Aip68cpCpB71KQczrGjADFxPkM accounts_delta: 3wTheAvdZ3rDNG4u3PfamYjXM8duJEcANkkNaRxCh6Rk signature_count: 8 last_blockhash: 2tgV47zoBJ1F3Mg7c4TksXGVXggu6fwRX2fCj2vH8hqB
[2020-03-02T15:20:52.474192903Z INFO  solana_runtime::bank] accounts hash slot: 2197007 stats: BankHashStats { num_removed_accounts: 13, num_added_accounts: 0, num_lamports_stored: 910000000004, total_data_len: 168114, num_executable_accounts: 0 }
[2020-03-02T15:20:52.475235933Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197008i block_height=1858599i
[2020-03-02T15:20:52.513405322Z INFO  solana_runtime::accounts_db] scan took 140us merge took 23us accumulate took 240us
[2020-03-02T15:20:52.513447742Z INFO  solana_runtime::bank] bank frozen: 2197008 hash: EouDH23iU5jPLYK9Jy1mPNP5v4Hr3vVE6ToY5v1avrjy accounts_delta: CL3k7zK3C2dder11GsmXSUAaN3YG5yzmT5oeoP6LQpEB signature_count: 302 last_blockhash: BNCsd1c4mLbCfs8VCtEziaBhzf5FPMkaE1uPdPWmvUyo
[2020-03-02T15:20:52.513476572Z INFO  solana_runtime::bank] accounts hash slot: 2197008 stats: BankHashStats { num_removed_accounts: 278, num_added_accounts: 0, num_lamports_stored: 13430294088953, total_data_len: 622886, num_executable_accounts: 0 }
[2020-03-02T15:20:52.514616444Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197009i block_height=1858600i
[2020-03-02T15:20:52.521086036Z INFO  solana_runtime::accounts_db] scan took 114us merge took 7us accumulate took 144us
[2020-03-02T15:20:52.521123506Z INFO  solana_runtime::bank] bank frozen: 2197009 hash: 5SQ259RQeUxz5amo7dA2TNUFow6xiy9cyLfG6GYWuDe5 accounts_delta: HPo4VYBAVaMo5YrDu741DnewRnNB96wHvi22pLpgfYaq signature_count: 22 last_blockhash: vzg3zqwgrUsumBqeFwFSzSjZz2iiTbsM9R2RzxPLJdc
[2020-03-02T15:20:52.521157017Z INFO  solana_runtime::bank] accounts hash slot: 2197009 stats: BankHashStats { num_removed_accounts: 27, num_added_accounts: 0, num_lamports_stored: 1125846726871, total_data_len: 195092, num_executable_accounts: 0 }
[2020-03-02T15:20:52.522207027Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197010i block_height=1858601i
[2020-03-02T15:20:52.538593346Z INFO  solana_runtime::accounts_db] scan took 160us merge took 30us accumulate took 218us
[2020-03-02T15:20:52.538639696Z INFO  solana_runtime::bank] bank frozen: 2197010 hash: 4FgxWADe1zcdStfdP368M3f4E6kmmMH1hu5NWhhtUhB4 accounts_delta: BNgT63SqUMS8yaa1sJL8LTgHzCrFj2ftmVUep1EGeGUq signature_count: 78 last_blockhash: Z5RjyFfuFy1KTcDbWtGLszAGyiiuhS2RJbysdiQxxvB
[2020-03-02T15:20:52.538675816Z INFO  solana_runtime::bank] accounts hash slot: 2197010 stats: BankHashStats { num_removed_accounts: 82, num_added_accounts: 0, num_lamports_stored: 2326866173067, total_data_len: 299150, num_executable_accounts: 0 }
[2020-03-02T15:20:52.539768916Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197011i block_height=1858602i
[2020-03-02T15:20:52.556524058Z INFO  solana_runtime::accounts_db] scan took 147us merge took 26us accumulate took 225us
[2020-03-02T15:20:52.556560688Z INFO  solana_runtime::bank] bank frozen: 2197011 hash: 4L9D2YQd3ZiLdnbT6B4YiqHD3SzRWNJS7qhdGmAvzm5H accounts_delta: Gzn27LGqRAinfKb6ta6m6EU8ihe5ptDKjdDboxxc3nM4 signature_count: 86 last_blockhash: BpmTdVYSpizqW67rdJH16SV7JYVPaPPYiRqWZREDemZa
[2020-03-02T15:20:52.556593109Z INFO  solana_runtime::bank] accounts hash slot: 2197011 stats: BankHashStats { num_removed_accounts: 90, num_added_accounts: 0, num_lamports_stored: 5357796044138, total_data_len: 314566, num_executable_accounts: 0 }
[2020-03-02T15:20:52.557676860Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197016i block_height=1858603i
[2020-03-02T15:20:52.586451417Z INFO  solana_runtime::accounts_db] scan took 169us merge took 5us accumulate took 275us
[2020-03-02T15:20:52.586488858Z INFO  solana_runtime::bank] bank frozen: 2197016 hash: AxyCNnUAKBynEWNDKguy2VBRyXd2v8yDHVzF276mj4Qe accounts_delta: 4qCRYvMfW4bK3LsQt7eXzQthsTALHYViD93FWFNNKPzR signature_count: 202 last_blockhash: 59LFYroQapr4mFDsxPX5xUaFgVYFJ8RZNshPckQrC4W7
[2020-03-02T15:20:52.586520659Z INFO  solana_runtime::bank] accounts hash slot: 2197016 stats: BankHashStats { num_removed_accounts: 195, num_added_accounts: 0, num_lamports_stored: 9862054024059, total_data_len: 495704, num_executable_accounts: 0 }
[2020-03-02T15:20:52.587542188Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197017i block_height=1858604i
[2020-03-02T15:20:52.598752337Z INFO  solana_runtime::accounts_db] scan took 98us merge took 26us accumulate took 196us
[2020-03-02T15:20:52.598778246Z INFO  solana_runtime::bank] bank frozen: 2197017 hash: 4nJzFaGSPb53z4KYkTS9EXLmgMdQjxqL9tzpTp5Rwtmm accounts_delta: 42R8vUXdiMfR3QBWyxUqabjFyrGwWjpG2kipYaDq9uGJ signature_count: 54 last_blockhash: 6XjarN3jD2SCpbxJTF1LWHnTEgKZ8n4GUTwQiHfbx75A
[2020-03-02T15:20:52.598800826Z INFO  solana_runtime::bank] accounts hash slot: 2197017 stats: BankHashStats { num_removed_accounts: 59, num_added_accounts: 0, num_lamports_stored: 2527875972778, total_data_len: 256756, num_executable_accounts: 0 }
[2020-03-02T15:20:52.599940728Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197018i block_height=1858605i
[2020-03-02T15:20:52.612724451Z INFO  solana_runtime::accounts_db] scan took 149us merge took 26us accumulate took 241us
[2020-03-02T15:20:52.612757672Z INFO  solana_runtime::bank] bank frozen: 2197018 hash: DRL9Ms8eaQNGKWfma64FEWae9rZYFFGu5ZuVkrsTMgf2 accounts_delta: FWLVJCNUBt6DVLKAKrMnbAiCQm1SZuARqGAbi9UdYMDS signature_count: 76 last_blockhash: 4smSwAtRrnz3bV23kRwqCvKkAnSf4apgjvQmApf4Y6LX
[2020-03-02T15:20:52.612788442Z INFO  solana_runtime::bank] accounts hash slot: 2197018 stats: BankHashStats { num_removed_accounts: 81, num_added_accounts: 0, num_lamports_stored: 3619717799574, total_data_len: 299150, num_executable_accounts: 0 }
[2020-03-02T15:20:52.613897503Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197019i block_height=1858606i
[2020-03-02T15:20:52.631218460Z INFO  solana_runtime::accounts_db] scan took 185us merge took 7us accumulate took 219us
[2020-03-02T15:20:52.631258561Z INFO  solana_runtime::bank] bank frozen: 2197019 hash: GFGoywFayUc4kWaPndxaT8nSyE6z7Uxkb45VLcbawGXX accounts_delta: DGGbRhycwKmGQH1pb5R1RYGGAATUmQKAEfWsXxgbwZez signature_count: 94 last_blockhash: 5sq18reiMd8L6xeNMNi5GDahsrAraBoE3jKsc8B4u7z
[2020-03-02T15:20:52.631288971Z INFO  solana_runtime::bank] accounts hash slot: 2197019 stats: BankHashStats { num_removed_accounts: 99, num_added_accounts: 0, num_lamports_stored: 4423740614412, total_data_len: 333836, num_executable_accounts: 0 }
[2020-03-02T15:20:52.632396352Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197020i block_height=1858607i
[2020-03-02T15:20:52.660360551Z INFO  solana_runtime::accounts_db] scan took 164us merge took 64us accumulate took 399us
[2020-03-02T15:20:52.660393231Z INFO  solana_runtime::bank] bank frozen: 2197020 hash: YT3WjW88VMu6bjRoZ12nAAZXUXBB8mFu9xo2vsUJg4r accounts_delta: 4m4MH5syCwj6QVZAbHsJBvN2qXWvfZn8UAAUzsQUMLyu signature_count: 182 last_blockhash: C7vvAjnqvFNDMKcVTpShG5mAzfu1FT5eYBab48oQ8Nya
[2020-03-02T15:20:52.660422672Z INFO  solana_runtime::bank] accounts hash slot: 2197020 stats: BankHashStats { num_removed_accounts: 184, num_added_accounts: 0, num_lamports_stored: 9154355670484, total_data_len: 491850, num_executable_accounts: 0 }
[2020-03-02T15:20:52.661489782Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197021i block_height=1858608i
[2020-03-02T15:20:52.671233137Z INFO  solana_runtime::accounts_db] scan took 119us merge took 21us accumulate took 238us
[2020-03-02T15:20:52.671264647Z INFO  solana_runtime::bank] bank frozen: 2197021 hash: APrvak2DpS5JPKF3s71hXUcDxpCNS9X6pixbXfRyUf2Z accounts_delta: BLDV7ZKGh712psEZnFv8MtDEirddbxLnd5CGHZyfvdHu signature_count: 54 last_blockhash: CnPF38mNuxgHE2PTQBTtcmr6gePrf5ALhMnNkyiEKBfU
[2020-03-02T15:20:52.671295987Z INFO  solana_runtime::bank] accounts hash slot: 2197021 stats: BankHashStats { num_removed_accounts: 59, num_added_accounts: 0, num_lamports_stored: 2712890672307, total_data_len: 256756, num_executable_accounts: 0 }
[2020-03-02T15:20:52.672353047Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197022i block_height=1858609i
[2020-03-02T15:20:52.693571713Z INFO  solana_runtime::accounts_db] scan took 141us merge took 29us accumulate took 205us
[2020-03-02T15:20:52.693608273Z INFO  solana_runtime::bank] bank frozen: 2197022 hash: Atn9tYLkWTqjfnY2B8f4K1j3MfJPTnrJqtBXSQZZ2cqG accounts_delta: ENtcimhKUufUnXJXM23LcBn1hakG42yjYDss8pn9vB5L signature_count: 130 last_blockhash: F2czD5cPf3Y87bdJSUV9s5CDkKLsqMMeYVGBCyv5Z3Q2
[2020-03-02T15:20:52.693632003Z INFO  solana_runtime::bank] accounts hash slot: 2197022 stats: BankHashStats { num_removed_accounts: 135, num_added_accounts: 0, num_lamports_stored: 5366663901073, total_data_len: 403208, num_executable_accounts: 0 }
[2020-03-02T15:20:52.694543532Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197023i block_height=1858610i
[2020-03-02T15:20:52.709317534Z INFO  solana_runtime::accounts_db] scan took 138us merge took 16us accumulate took 345us
[2020-03-02T15:20:52.709345695Z INFO  solana_runtime::bank] bank frozen: 2197023 hash: 9tPHgnqLpx12aFGkqNhxBvJCSxVbVMpF3WYWzXj6BFeT accounts_delta: GkiCaBWh3y586dJWaN3a4SRPnBezocB87tjVn4354UWd signature_count: 92 last_blockhash: G3zSTyNbFodzUEw1iqYn2YpNXUqjr1hTREgAHLcHnnXt
[2020-03-02T15:20:52.709360855Z INFO  solana_runtime::bank] accounts hash slot: 2197023 stats: BankHashStats { num_removed_accounts: 97, num_added_accounts: 0, num_lamports_stored: 4190703100019, total_data_len: 329982, num_executable_accounts: 0 }
[2020-03-02T15:20:52.710391945Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197028i block_height=1858611i
[2020-03-02T15:20:52.736601898Z INFO  solana_runtime::accounts_db] scan took 159us merge took 25us accumulate took 323us
[2020-03-02T15:20:52.736638408Z INFO  solana_runtime::bank] bank frozen: 2197028 hash: HxDTsBNKXVkhvBZeVq4RQYS7Tbkr93NQ98GDdozMCveH accounts_delta: 58HRhrrnQV1u6YodRC2TpJQPbw74P15C6GGcEBHc2oUM signature_count: 134 last_blockhash: AXTYF5viUxwTCQHTmxz6uCzQTNL19CG5EgZu7bNkMw8y
[2020-03-02T15:20:52.736664639Z INFO  solana_runtime::bank] accounts hash slot: 2197028 stats: BankHashStats { num_removed_accounts: 134, num_added_accounts: 0, num_lamports_stored: 7457376954409, total_data_len: 391646, num_executable_accounts: 0 }
[2020-03-02T15:20:52.737673298Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197029i block_height=1858612i
[2020-03-02T15:20:52.742778438Z INFO  solana_runtime::accounts_db] scan took 132us merge took 230ns accumulate took 7us
[2020-03-02T15:20:52.742808058Z INFO  solana_runtime::bank] bank frozen: 2197029 hash: Aq7kbyCNZPT6kW6fjC95QuPsemfNDfuciqGRZPQqJSkH accounts_delta: 3aDH8hGZxHDteomd11jysVLRAcnRrCzAf43mKjjUjCu4 signature_count: 6 last_blockhash: 2cPnSSZBnMDfH4jhjZeTdj6fLmd61FQTVQYUWr8UreAb
[2020-03-02T15:20:52.742833319Z INFO  solana_runtime::bank] accounts hash slot: 2197029 stats: BankHashStats { num_removed_accounts: 11, num_added_accounts: 0, num_lamports_stored: 19934770883, total_data_len: 164260, num_executable_accounts: 0 }
[2020-03-02T15:20:52.743846718Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197030i block_height=1858613i
[2020-03-02T15:20:52.747831377Z INFO  solana_metrics::counter] COUNTER:{"name": "vote-native", "counts": 3001, "samples": 3000,  "now": 1583162452747, "events": 1}
[2020-03-02T15:20:52.758047785Z INFO  solana_runtime::accounts_db] scan took 157us merge took 18us accumulate took 177us
[2020-03-02T15:20:52.758100866Z INFO  solana_runtime::bank] bank frozen: 2197030 hash: 9X26ka3MpiF7LwAcyZFNAibjrWsAqWcxyRKAMcmW3Vae accounts_delta: 2pey3MNhwEw9EhLsYMv6FpnPDvEXDWgQKSkgaRydWk6Z signature_count: 86 last_blockhash: 3oX7F2ojf2DdMoeWLDodABPqRw6wJ5P3QDHB9e13xXGE
[2020-03-02T15:20:52.758132106Z INFO  solana_runtime::bank] accounts hash slot: 2197030 stats: BankHashStats { num_removed_accounts: 91, num_added_accounts: 0, num_lamports_stored: 4543712899726, total_data_len: 318420, num_executable_accounts: 0 }
[2020-03-02T15:20:52.759182836Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197031i block_height=1858614i
[2020-03-02T15:20:52.775196201Z INFO  solana_runtime::accounts_db] scan took 113us merge took 21us accumulate took 289us
[2020-03-02T15:20:52.775238882Z INFO  solana_runtime::bank] bank frozen: 2197031 hash: EA5dpjFxQveX4RMmFaJb3uEcfmfM8Eon4CQVvN3jqhMz accounts_delta: 2v3xpVsw6o5N1pk9VBL7Qmwi69ZqKwhVnK2mGx1yADbq signature_count: 112 last_blockhash: Dy5Yfx14ZcFtbp1vNTHuFzaxganYoGasMaP3EpZ151Su
[2020-03-02T15:20:52.775272081Z INFO  solana_runtime::bank] accounts hash slot: 2197031 stats: BankHashStats { num_removed_accounts: 114, num_added_accounts: 0, num_lamports_stored: 5531580757138, total_data_len: 356960, num_executable_accounts: 0 }
[2020-03-02T15:20:52.776199090Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197036i block_height=1858615i
[2020-03-02T15:20:52.807551524Z INFO  solana_runtime::accounts_db] scan took 134us merge took 28us accumulate took 322us
[2020-03-02T15:20:52.807591644Z INFO  solana_runtime::bank] bank frozen: 2197036 hash: 2wB1usUx9m8CKWtqUJSXBCxwpFNHt9kpNfeN9YMVrgfa accounts_delta: 3AZUW3rncsMZBTkx46dSCyLXhzty1WB5qHqZhfbJpNLx signature_count: 188 last_blockhash: 9e1cHj3Psw99cEAXTTcuAzVJoeeUTqbmYcBVDZcJMr1q
[2020-03-02T15:20:52.807665655Z INFO  solana_runtime::bank] accounts hash slot: 2197036 stats: BankHashStats { num_removed_accounts: 183, num_added_accounts: 0, num_lamports_stored: 9575184482301, total_data_len: 476434, num_executable_accounts: 0 }
[2020-03-02T15:20:52.808779645Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197037i block_height=1858616i
[2020-03-02T15:20:52.813566412Z INFO  solana_runtime::accounts_db] scan took 58us merge took 440ns accumulate took 6us
[2020-03-02T15:20:52.813591882Z INFO  solana_runtime::bank] bank frozen: 2197037 hash: 3vHGUW1hSNn3t1zTSbMFErNgzzN2yQXczKmqqCMj6Rm7 accounts_delta: 9AVQjTTCVt5iwuyUhx2fEf9KkHch8jaFJSP5f7nNy5j7 signature_count: 2 last_blockhash: qiuKu8iZuk9KYHkCeocCCz8ECDVGXHo7xqP2CrXZcMN
[2020-03-02T15:20:52.813653393Z INFO  solana_runtime::bank] accounts hash slot: 2197037 stats: BankHashStats { num_removed_accounts: 7, num_added_accounts: 0, num_lamports_stored: 9995100165, total_data_len: 156552, num_executable_accounts: 0 }
[2020-03-02T15:20:52.814704473Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197038i block_height=1858617i
[2020-03-02T15:20:52.819174086Z INFO  solana_runtime::accounts_db] scan took 105us merge took 230ns accumulate took 5us
[2020-03-02T15:20:52.819201366Z INFO  solana_runtime::bank] bank frozen: 2197038 hash: 2CiNQNuPvurfen9yHkmfsExsyfv8QGQt3DGt5c4xf5n6 accounts_delta: 9e6SRYa5EnZ8u2zDMM7RATMpSjwLLw6Gv1kE21GcUYJK signature_count: 2 last_blockhash: 9xTLMPFHWcH2rASoFHLcHX4fz9Lfm9ZB18YN82hR8pzN
[2020-03-02T15:20:52.819228826Z INFO  solana_runtime::bank] accounts hash slot: 2197038 stats: BankHashStats { num_removed_accounts: 7, num_added_accounts: 0, num_lamports_stored: 9995100165, total_data_len: 156552, num_executable_accounts: 0 }
[2020-03-02T15:20:52.820306287Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197039i block_height=1858618i
[2020-03-02T15:20:52.833277742Z INFO  solana_runtime::accounts_db] scan took 85us merge took 13us accumulate took 365us
[2020-03-02T15:20:52.833312933Z INFO  solana_runtime::bank] bank frozen: 2197039 hash: 32AhQKrsnF6C1getvZkrTpjXJ9nbbFkYAKqtXKkojJcH accounts_delta: 22bc3xd9vuS21aYJMAyLLtHeqYxYpR4G5W8Uyu9yY5AK signature_count: 58 last_blockhash: FqUmKy97oiijwaTEHyDz64W7n6M5Jfgqkt6e2gVKv5Zr
[2020-03-02T15:20:52.833328163Z INFO  solana_runtime::bank] accounts hash slot: 2197039 stats: BankHashStats { num_removed_accounts: 63, num_added_accounts: 0, num_lamports_stored: 3647871072718, total_data_len: 264464, num_executable_accounts: 0 }
[2020-03-02T15:20:52.834415253Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197040i block_height=1858619i
[2020-03-02T15:20:52.868337620Z INFO  solana_runtime::accounts_db] scan took 138us merge took 27us accumulate took 156us
[2020-03-02T15:20:52.868374931Z INFO  solana_runtime::bank] bank frozen: 2197040 hash: BVc2GjaJS9dmzs5kRbqupuxJnx66Pr4iMd7d8nQS3mFW accounts_delta: GDDc7HrgGeTnMHGww9R9XxgCsiuysZpMcAG61prEV2Ag signature_count: 214 last_blockhash: HD9yKoXdVhm6E7ni2NJ77RmM8BAz5ZcPS1fXB1KQQuLv
[2020-03-02T15:20:52.868406202Z INFO  solana_runtime::bank] accounts hash slot: 2197040 stats: BankHashStats { num_removed_accounts: 205, num_added_accounts: 0, num_lamports_stored: 9893003494459, total_data_len: 511120, num_executable_accounts: 0 }
[2020-03-02T15:20:52.869418351Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197041i block_height=1858620i
[2020-03-02T15:20:52.876586481Z INFO  solana_runtime::accounts_db] scan took 105us merge took 420ns accumulate took 8us
[2020-03-02T15:20:52.876615641Z INFO  solana_runtime::bank] bank frozen: 2197041 hash: 76sn4k9bKmv5nK8bBeMxHSK6Bjawj4kx8CwTmPi8FAwV accounts_delta: 7yJmcjLArm5h1h94Mn9YdPu6dELwhzeovGbC2CFuuUka signature_count: 10 last_blockhash: 9jypzrD9gVs63LZKQBFRo6yRPog5rtFiVzxDd3D19iGa
[2020-03-02T15:20:52.876644101Z INFO  solana_runtime::bank] accounts hash slot: 2197041 stats: BankHashStats { num_removed_accounts: 15, num_added_accounts: 0, num_lamports_stored: 27962485605, total_data_len: 171968, num_executable_accounts: 0 }
[2020-03-02T15:20:52.877662860Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197042i block_height=1858621i
[2020-03-02T15:20:52.891691306Z INFO  solana_runtime::accounts_db] scan took 144us merge took 26us accumulate took 277us
[2020-03-02T15:20:52.891721917Z INFO  solana_runtime::bank] bank frozen: 2197042 hash: D339oWRjwpyFaWWhsy7VLBeEnoLeNSe5hbeU1vWmdLh1 accounts_delta: 3BHCjzmbMgG6Tc91CBFCVh2GxYdr8HjNFNdoekde7wes signature_count: 108 last_blockhash: 3P1YNrXdbhsFRQfs756cKKpR7dgF9hVNzr1vyU7epVx9
[2020-03-02T15:20:52.891754127Z INFO  solana_runtime::bank] accounts hash slot: 2197042 stats: BankHashStats { num_removed_accounts: 113, num_added_accounts: 0, num_lamports_stored: 5691655785952, total_data_len: 360814, num_executable_accounts: 0 }
[2020-03-02T15:20:52.892704095Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197043i block_height=1858622i
[2020-03-02T15:20:52.910437377Z INFO  solana_runtime::accounts_db] scan took 189us merge took 28us accumulate took 157us
[2020-03-02T15:20:52.910474248Z INFO  solana_runtime::bank] bank frozen: 2197043 hash: 8CmmsBxqBGWwemAqJD7z6eP4hqB7vxWZZxLMAvzZy2eH accounts_delta: Hj2n2EhUv4zeqxXK9RjzfXWwLWLk1qFLPwWuzQGU5VEK signature_count: 128 last_blockhash: 2m7wgQwamJtiST5xFqHbAagdTxFqzy7Upwu9Ny8w3ykP
[2020-03-02T15:20:52.910507548Z INFO  solana_runtime::bank] accounts hash slot: 2197043 stats: BankHashStats { num_removed_accounts: 133, num_added_accounts: 0, num_lamports_stored: 5848729130425, total_data_len: 399354, num_executable_accounts: 0 }
[2020-03-02T15:20:52.911618769Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197052i block_height=1858623i
[2020-03-02T15:20:52.937306407Z INFO  solana_runtime::accounts_db] scan took 206us merge took 29us accumulate took 384us
[2020-03-02T15:20:52.937339787Z INFO  solana_runtime::bank] bank frozen: 2197052 hash: GXe5jJs5hnfhszUASei3fGe7mzLThgQAyf4RUJgq7fWT accounts_delta: GNQhQpUjPjtG1MUBw7YAJxkrTTBisECFatJezKhKP7tb signature_count: 160 last_blockhash: 7MQMpqJ3n5cwKENX8bG6AZZr4GNXgVKw9sCbTtN8GLc2
[2020-03-02T15:20:52.937374147Z INFO  solana_runtime::bank] accounts hash slot: 2197052 stats: BankHashStats { num_removed_accounts: 158, num_added_accounts: 0, num_lamports_stored: 8148231796386, total_data_len: 434040, num_executable_accounts: 0 }
[2020-03-02T15:20:52.938414238Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197053i block_height=1858624i
[2020-03-02T15:20:52.944147373Z INFO  solana_runtime::accounts_db] scan took 137us merge took 230ns accumulate took 6us
[2020-03-02T15:20:52.944177443Z INFO  solana_runtime::bank] bank frozen: 2197053 hash: Es59BSLAfvZQT97izcQhSBFGnyJHo5p2d4cLhhCUeiH5 accounts_delta: Ct2iaHupQ1wiN33Djr11MfpBA2sJ12recF9BtvxfAbW signature_count: 2 last_blockhash: H6umv1Rt7Snqq7RyiVT1y1fXTeUKKprH3GCRKBQLsTFW
[2020-03-02T15:20:52.944207173Z INFO  solana_runtime::bank] accounts hash slot: 2197053 stats: BankHashStats { num_removed_accounts: 7, num_added_accounts: 0, num_lamports_stored: 1000000005, total_data_len: 156552, num_executable_accounts: 0 }
[2020-03-02T15:20:52.945224553Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197054i block_height=1858625i
[2020-03-02T15:20:52.951358553Z INFO  solana_runtime::accounts_db] scan took 98us merge took 350ns accumulate took 331us
[2020-03-02T15:20:52.951389123Z INFO  solana_runtime::bank] bank frozen: 2197054 hash: 6fYgVRg9MxjF8rhnPUgcjsNSNJ4FVRsRdGMyBBBr6WUK accounts_delta: 74SJH5SQxoainFgydizMyHeK6chFvLG5KkGrviWsfDjF signature_count: 16 last_blockhash: cW9TFS4XaAh44hay14JudxSeYeQNRnZJ1p3jaMxWTob
[2020-03-02T15:20:52.951415673Z INFO  solana_runtime::bank] accounts hash slot: 2197054 stats: BankHashStats { num_removed_accounts: 21, num_added_accounts: 0, num_lamports_stored: 134942886014, total_data_len: 183530, num_executable_accounts: 0 }
[2020-03-02T15:20:52.952466593Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197055i block_height=1858626i
[2020-03-02T15:20:52.968659319Z INFO  solana_runtime::accounts_db] scan took 124us merge took 23us accumulate took 275us
[2020-03-02T15:20:52.968690740Z INFO  solana_runtime::bank] bank frozen: 2197055 hash: 5KxkPPpYruwRrPLz6mt7h7NmVfngEhZiCMEJ9ygjKTcJ accounts_delta: HEte6UJ1TmYFFA3VMVBFksnwm56zH5Piux149sUjArMg signature_count: 90 last_blockhash: G7Kre6dBrq7vVJwoXmXzXyQXfXaosdMr8hcjSvmcWvFT
[2020-03-02T15:20:52.968716570Z INFO  solana_runtime::bank] accounts hash slot: 2197055 stats: BankHashStats { num_removed_accounts: 92, num_added_accounts: 0, num_lamports_stored: 5901582441457, total_data_len: 314566, num_executable_accounts: 0 }
[2020-03-02T15:20:52.969745160Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197056i block_height=1858627i
[2020-03-02T15:20:53.002094733Z INFO  solana_runtime::accounts_db] scan took 128us merge took 39us accumulate took 178us
[2020-03-02T15:20:53.002137084Z INFO  solana_runtime::bank] bank frozen: 2197056 hash: DJbTdZs5JZNQsVQ9rNLvJfwXeRqyHsGMVSb4NoiUgqr9 accounts_delta: F2tPoUj67DU8nwAspP8AyVciT2W1HxG5RvT6jM28xqrw signature_count: 236 last_blockhash: 9BEebFoLdnzybsQWqhqqqe1zVQtGkB4xLRuZS6vGWVhF
[2020-03-02T15:20:53.002156413Z INFO  solana_runtime::bank] accounts hash slot: 2197056 stats: BankHashStats { num_removed_accounts: 224, num_added_accounts: 0, num_lamports_stored: 12079732778277, total_data_len: 541952, num_executable_accounts: 0 }
[2020-03-02T15:20:53.003212323Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197057i block_height=1858628i
[2020-03-02T15:20:53.012279081Z INFO  solana_runtime::accounts_db] scan took 91us merge took 7us accumulate took 170us
[2020-03-02T15:20:53.012318221Z INFO  solana_runtime::bank] bank frozen: 2197057 hash: 5op2NrrcGdGS13Eo63y7DkgpsqjAFd6fai21jGjbxXBr accounts_delta: SRUVeVWkPkX1HsmjZ1XVf54Bo533i3UXqb9fdAXrReh signature_count: 34 last_blockhash: 3iG47D2UEY9nF3qb4hj8LBSfWnVKk5KqJKvPL16wPKX4
[2020-03-02T15:20:53.012395322Z INFO  solana_runtime::bank] accounts hash slot: 2197057 stats: BankHashStats { num_removed_accounts: 39, num_added_accounts: 0, num_lamports_stored: 645905371601, total_data_len: 218216, num_executable_accounts: 0 }
[2020-03-02T15:20:53.013195020Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197058i block_height=1858629i
[2020-03-02T15:20:53.027007813Z INFO  solana_runtime::accounts_db] scan took 113us merge took 24us accumulate took 253us
[2020-03-02T15:20:53.027029494Z INFO  solana_runtime::bank] bank frozen: 2197058 hash: 2qnF9nc99CLsVQaKYL1WomKUVkZi1vVuzCj1JXXgRSZJ accounts_delta: 2RuK5HT5jYZrEWnHwy4JZHXrVmDpe8qkFNyZUSQnLdUw signature_count: 98 last_blockhash: 2mtVBYqE4jEVAQaCERk6ZSq4yUdbzAg8LFo5TU8SJ5nq
[2020-03-02T15:20:53.027051194Z INFO  solana_runtime::bank] accounts hash slot: 2197058 stats: BankHashStats { num_removed_accounts: 103, num_added_accounts: 0, num_lamports_stored: 4915758529472, total_data_len: 341544, num_executable_accounts: 0 }
[2020-03-02T15:20:53.028122184Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197059i block_height=1858630i
[2020-03-02T15:20:53.046115258Z INFO  solana_runtime::accounts_db] scan took 121us merge took 27us accumulate took 165us
[2020-03-02T15:20:53.046156648Z INFO  solana_runtime::bank] bank frozen: 2197059 hash: 3bmNTjYA6XsQ9zcJGNNNv4kKCy9vdUfUbNaRVu4kfRTo accounts_delta: FxJ8JDfvyyPomWQdcHFgBNPVsUGoqHqDUMqG2Ua6BvF5 signature_count: 98 last_blockhash: 3K8bFvFNHyGv1txWHfW8pUcpN1WVwMyc8t5V6KCfE8Vx
[2020-03-02T15:20:53.046176569Z INFO  solana_runtime::bank] accounts hash slot: 2197059 stats: BankHashStats { num_removed_accounts: 103, num_added_accounts: 0, num_lamports_stored: 4389703100034, total_data_len: 341544, num_executable_accounts: 0 }
[2020-03-02T15:20:53.046996196Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197060i block_height=1858631i
[2020-03-02T15:20:53.079271398Z INFO  solana_runtime::accounts_db] scan took 140us merge took 18us accumulate took 183us
[2020-03-02T15:20:53.079302589Z INFO  solana_runtime::bank] bank frozen: 2197060 hash: BDwmWnqkWeh4TcQwQUWmB9QZta9s73pQ9RgAkPbvAk6p accounts_delta: 8FLMLAJgMvpembgKg1PeGSJqK7rjUozxPLFyE5N5tkmC signature_count: 214 last_blockhash: 4xpY2b9aNytXX1o14drKg69bghYA3ivdGmdJd7JDviBE
[2020-03-02T15:20:53.079333229Z INFO  solana_runtime::bank] accounts hash slot: 2197060 stats: BankHashStats { num_removed_accounts: 214, num_added_accounts: 0, num_lamports_stored: 10398257827013, total_data_len: 545806, num_executable_accounts: 0 }
[2020-03-02T15:20:53.080437339Z INFO  solana_metrics::metrics] datapoint: bank-new_from_parent-heights slot_height=2197061i block_height=1858632i
[2020-03-02T15:20:53.082695411Z INFO  solana_ledger::blockstore_processor] ledger processed in 1539ms. 388 MB allocated. 1 fork at 2197060, with 1 frozen bank
Creating a snapshot of slot 2197060
[2020-03-02T15:20:53.086382277Z INFO  solana_runtime::accounts_db] total_stores: 8305, newest_slot: 2197060, oldest_slot: 0, max_slot: 2146996 (num=9), min_slot: 1504465 (num=1)
[2020-03-02T15:20:53.086477888Z INFO  solana_metrics::metrics] datapoint: accounts_db-stores total_count=8305i
[2020-03-02T15:20:53.087897322Z INFO  solana_ledger::snapshot_utils] Creating snapshot for slot 2197060, path: "/tmp/.tmpOggDLk/2197060/2197060"
[2020-03-02T15:20:53.095534175Z INFO  solana_metrics::metrics] datapoint: serialize_account_storage_ms duration=2i num_entries=8305i
[2020-03-02T15:20:53.095686347Z INFO  solana_metrics::metrics] datapoint: snapshot-bank-file slot=2197060i size=2797224i
[2020-03-02T15:20:53.095682937Z INFO  solana_ledger::snapshot_utils] bank-serialize-ms took 7ms for slot 2197060 at "/tmp/.tmpOggDLk/2197060/2197060"
[2020-03-02T15:20:53.095837529Z INFO  solana_ledger::snapshot_utils] Snapshot for bank: 2197060 has 1833 account storage entries
[2020-03-02T15:20:53.097917718Z INFO  solana_ledger::snapshot_utils] Generating snapshot archive for slot 2197060
[2020-03-02T15:20:53.101823967Z INFO  solana_metrics::metrics] datapoint: snapshot-status-cache-file slot=2197060i size=305760i



[2020-03-02T15:29:53.310801009Z INFO  solana_ledger::snapshot_utils] Successfully created "newnew2/snapshot-2197060-5qTAFmWaeyxmG6qXCeZS1ojH4rXmZVKA5VD5FGh6JpeF.tar.bz2". slot: 2197060, elapsed ms: 540208, size=88395687
[2020-03-02T15:29:53.310881670Z INFO  solana_metrics::metrics] datapoint: snapshot-package slot=2197060i duration_ms=540208i size=88395687i
Successfully created snapshot for slot 2197060: "newnew2/snapshot-2197060-5qTAFmWaeyxmG6qXCeZS1ojH4rXmZVKA5VD5FGh6JpeF.tar.bz2"
Shred version: 5437


$ SOLANA_DISABLE_SNAPSHOT_COMPRESSION=1 LD_LIBRARY_PATH=. ../solana-ledger-tool --ledger newne2w verify
Unable to access ledger path: Os { code: 2, kind: NotFound, message: "No such file or directory" }
ubuntu@ip-10-163-204-165:/mnt/solana-data/tds-with-1.0/try5$ SOLANA_DISABLE_SNAPSHOT_COMPRESSION=1 LD_LIBRARY_PATH=. ../solana-ledger-tool --ledger newnew2 verify
4uhcVJyU9pJkvQyS88uRDiswHXSCkY3zQawwpjk2NsNY
[2020-03-02T15:37:55.980479001Z INFO  solana_ledger::blockstore] Maximum open file descriptors: 65000
[2020-03-02T15:37:56.131472895Z INFO  solana_ledger::blockstore] "/mnt/solana-data/tds-with-1.0/try5/newnew2/rocksdb" open took 150ms
[2020-03-02T15:37:56.131511955Z INFO  solana_ledger::bank_forks_utils] Initializing snapshot path: "/mnt/solana-data/tds-with-1.0/try5/newnew2/snapshot"
[2020-03-02T15:37:56.132156101Z INFO  solana_ledger::bank_forks_utils] Loading snapshot package: "/mnt/solana-data/tds-with-1.0/try5/newnew2/snapshot-2197060-5qTAFmWaeyxmG6qXCeZS1ojH4rXmZVKA5VD5FGh6JpeF.tar.bz2"
[2020-03-02T15:40:24.581907876Z INFO  solana_ledger::snapshot_utils] snapshot untar took 148.4s
[2020-03-02T15:40:24.582213748Z INFO  solana_ledger::snapshot_utils] snapshot version: 1.0.0
[2020-03-02T15:40:24.582263369Z INFO  solana_ledger::snapshot_utils] Loading bank from "/mnt/solana-data/tds-with-1.0/try5/newnew2/snapshot/.tmplEzmay/snapshots/2197060/2197060"
[2020-03-02T15:40:24.588455472Z INFO  solana_ledger::snapshot_utils] Rebuilding accounts...
[2020-03-02T15:40:54.062114769Z INFO  solana_ledger::snapshot_utils] Rebuilding status cache...
[2020-03-02T15:40:54.072846722Z INFO  solana_ledger::snapshot_utils] Loaded bank for slot: 2197060
[2020-03-02T15:40:54.072954272Z INFO  solana_runtime::accounts_db] total_stores: 8305, newest_slot: 2197060, oldest_slot: 0, max_slot: 351916 (num=9), min_slot: 2176477 (num=1)
[2020-03-02T15:40:54.072991783Z INFO  solana_metrics::metrics] metrics disabled: SOLANA_METRICS_CONFIG: environment variable not found
[2020-03-02T15:40:54.073156774Z INFO  solana_metrics::metrics] datapoint: accounts_db-stores total_count=8305i
[2020-03-02T15:40:54.077952076Z INFO  solana_runtime::accounts_db] scan took 98us merge took 41us accumulate took 96us
[2020-03-02T15:40:54.077974226Z INFO  solana_runtime::bank] bank frozen: 2197060 hash: BDwmWnqkWeh4TcQwQUWmB9QZta9s73pQ9RgAkPbvAk6p accounts_delta: 8FLMLAJgMvpembgKg1PeGSJqK7rjUozxPLFyE5N5tkmC signature_count: 214 last_blockhash: 4xpY2b9aNytXX1o14drKg69bghYA3ivdGmdJd7JDviBE
[2020-03-02T15:40:54.078007006Z INFO  solana_runtime::bank] accounts hash slot: 2197060 stats: BankHashStats { num_removed_accounts: 214, num_added_accounts: 0, num_lamports_stored: 10398257827013, total_data_len: 545806, num_executable_accounts: 0 }
[2020-03-02T15:40:54.078039867Z INFO  solana_ledger::snapshot_utils] bank rebuild from snapshot took 29.5s
[2020-03-02T15:40:54.096521526Z INFO  solana_ledger::blockstore_processor] processing ledger from root slot 2197060...
[2020-03-02T15:40:54.161023055Z INFO  solana_ledger::blockstore_processor] ledger processed in 64ms. 83 MB allocated. 1 fork at 2197060, with 1 frozen bank
Ok

let (accounts, skip_account_assertion) = f(accounts, current_slot);

assert_eq!(4, accounts.accounts_index.read().unwrap().roots.len());
if skip_account_assertion {

This comment has been minimized.

Copy link
@ryoqun

ryoqun Mar 2, 2020

Author Member

The following assertion causes failure currently for test_accounts_purge_chained_purge_after_snapshot_restore.

This should be fixed by #8337

@mvines
mvines approved these changes Mar 2, 2020
Copy link
Member

mvines left a comment

Looks great to me as a workaround to get TdS unstuck. It would be nice to find a cleaner way, if possible, as a follow-up.

Thank you for the quick turnaround @ryoqun, you rock!

@mvines

This comment has been minimized.

Copy link
Member

mvines commented Mar 2, 2020

Bench failure is unrelated, :shipit:

@mvines mvines merged commit 5d9130a into solana-labs:master Mar 2, 2020
11 of 13 checks passed
11 of 13 checks passed
buildkite/solana Build #19917 failed (58 minutes, 46 seconds)
Details
buildkite/solana/bench Failed (exit status 101)
Details
Rule: v1.0 backport (backport) Backports have been created
Details
Summary 3 rules match and 2 potential rules
Details
buildkite/solana/checks Passed (4 minutes, 42 seconds)
Details
buildkite/solana/coverage Passed (16 minutes, 36 seconds)
Details
buildkite/solana/local-cluster Passed (22 minutes, 27 seconds)
Details
buildkite/solana/move Passed (4 minutes, 17 seconds)
Details
buildkite/solana/pipeline-upload Passed (2 seconds)
Details
buildkite/solana/shellcheck Passed (33 seconds)
Details
buildkite/solana/stable Passed (48 minutes, 7 seconds)
Details
buildkite/solana/stable-perf Passed (12 minutes, 15 seconds)
Details
ci-gate Pull Request accepted for CI pipeline
@ryoqun ryoqun added the v0.23 label Mar 3, 2020
@ryoqun

This comment has been minimized.

Copy link
Member Author

ryoqun commented Mar 3, 2020

This got needed in the v0.23 branch for #8436

@ryoqun

This comment has been minimized.

Copy link
Member Author

ryoqun commented Mar 3, 2020

@Mergifyio refresh

solana-grimes pushed a commit that referenced this pull request Mar 3, 2020
automerge
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked issues

Successfully merging this pull request may close these issues.

3 participants
You can’t perform that action at this time.