TdS snapshots fail to verify

[mvines]

After TdS restarted, nodes started generating bad snapshots.

STR:

1. Run solana-validator -l ledger -n 35.203.170.30:8001 --expected-shred-version 5437 --trusted-validator 5D1fNXzvv5NjV1ysLjirC4WY92RNsVH18vjmcszZd8o

Example failure output:

[2020-03-02T05:11:24.363024211Z INFO  solana_validator]   Downloaded http://175.113.78.40:8899/snapshot-2216292-CcAbAUEz9B37pANfVDfXXx84XKwa8BZz7z6enUnD1i3R.tar.bz2 (90315267 bytes) in 5.223901195s
[2020-03-02T05:11:24.404712301Z WARN  solana_core::validator] identity pubkey: 63MGLbDbPnFrBAmhb7HstjPiF6gnvhhj8qeLB3gCfErM
[2020-03-02T05:11:24.404730801Z WARN  solana_core::validator] vote pubkey: 9QLgBAgjzDkf3Y7rP15zywMBgxVJc2HC87yG4ubRq7SL
[2020-03-02T05:11:24.404734801Z WARN  solana_core::validator] CUDA is disabled
[2020-03-02T05:11:24.404736601Z INFO  solana_core::validator] AVX detected
[2020-03-02T05:11:24.404737901Z INFO  solana_core::validator] entrypoint: Some(ContactInfo { id: 11111111111111111111111111111111, gossip: V4(35.203.170.30:8001), tvu: V4(0.0.0.0:0), tvu_forwards: V4(0.0.0.0:0), repair: V4(0.0.0.0:0), tpu: V4(0.0.0.0:0), tpu_forwards: V4(0.0.0.0:0), storage_addr: V4(0.0.0.0:0), rpc: V4(0.0.0.0:0), rpc_pubsub: V4(0.0.0.0:0), serve_repair: V4(0.0.0.0:0), wallclock: 1583125871245, shred_version: 0 })
[2020-03-02T05:11:24.404748901Z INFO  solana_core::validator] Initializing sigverify, this could take a while...
[2020-03-02T05:11:24.404750401Z INFO  solana_core::validator] Done.
[2020-03-02T05:11:24.404751601Z INFO  solana_core::validator] creating bank...
[2020-03-02T05:11:24.406324700Z INFO  solana_core::validator] genesis hash: 4uhcVJyU9pJkvQyS88uRDiswHXSCkY3zQawwpjk2NsNY
[2020-03-02T05:11:24.406352700Z INFO  solana_ledger::blockstore] Maximum open file descriptors: 65000
[2020-03-02T05:11:24.640746842Z INFO  solana_ledger::blockstore] "/ws/mvines/ledger/rocksdb" open took 234ms
[2020-03-02T05:11:24.641732041Z INFO  solana_ledger::bank_forks_utils] Initializing snapshot path: "/ws/mvines/ledger/snapshot"
[2020-03-02T05:11:24.642197241Z INFO  solana_ledger::bank_forks_utils] Loading snapshot package: "/ws/mvines/ledger/snapshot-2216292-CcAbAUEz9B37pANfVDfXXx84XKwa8BZz7z6enUnD1i3R.tar.bz2"
[2020-03-02T05:19:50.800598475Z INFO  solana_ledger::snapshot_utils] snapshot untar took 506.2s
[2020-03-02T05:19:50.800633275Z INFO  solana_ledger::snapshot_utils] snapshot version: 1.0.0
[2020-03-02T05:19:50.800659075Z INFO  solana_ledger::snapshot_utils] Loading bank from "/ws/mvines/ledger/snapshot/.tmpbYWB3X/snapshots/2216292/2216292"
[2020-03-02T05:19:50.804684486Z INFO  solana_ledger::snapshot_utils] Rebuilding accounts...
[2020-03-02T05:29:14.829520614Z INFO  solana_ledger::snapshot_utils] Rebuilding status cache...
[2020-03-02T05:29:14.838537823Z INFO  solana_ledger::snapshot_utils] Loaded bank for slot: 2216292
[2020-03-02T05:29:14.838634424Z INFO  solana_runtime::accounts_db] total_stores: 8386, newest_slot: 2216292, oldest_slot: 0, max_slot: 2146936 (num=9), min_slot: 2171751 (num=1)
[2020-03-02T05:29:14.838693924Z INFO  solana_metrics::metrics] datapoint: accounts_db-stores total_count=8386i
[2020-03-02T05:29:14.860167947Z WARN  solana_runtime::accounts_db] mismatched bank hash for slot 2216292: HnoDAJopjHyKQsp2rohY4SYRhkZeVdAw7wxcHkyHqCUC (calculated) != CcAbAUEz9B37pANfVDfXXx84XKwa8BZz7z6enUnD1i3R (expected)
[2020-03-02T05:29:14.860186447Z WARN  solana_runtime::accounts] verify_bank_hash failed: MismatchedBankHash
thread 'main' panicked at 'Snapshot bank for slot 2216292 failed to verify', ledger/src/snapshot_utils.rs:458:9
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace.
[2020-03-02T05:29:14.860235447Z ERROR solana_metrics::metrics] datapoint: panic program="validator" thread="main" one=1i message="panicked at 'Snapshot bank for slot 2216292 failed to verify', ledger/src/snapshot_utils.rs:458:9" location="ledger/src/snapshot_utils.rs:458:9"