Validator cannot catchup in TdS with v0.23.6

[kwunyeung]

Problem

Many validators cannot catchup in TdS after the restart with v0.23.6. The log keeps repeating this.

Feb 23 10:02:58 solana-validator-eu solana-validator[5112]: [2020-02-23T10:02:58.180738984Z INFO  solana_metrics::metrics] datapoint: serve_repair-repair_orphan repair-orphan=1533611i
Feb 23 10:02:58 solana-validator-eu solana-validator[5112]: [2020-02-23T10:02:58.180743288Z INFO  solana_metrics::metrics] datapoint: serve_repair-repair_orphan repair-orphan=1534123i
Feb 23 10:02:58 solana-validator-eu solana-validator[5112]: [2020-02-23T10:02:58.180807848Z INFO  solana_metrics::metrics] datapoint: serve_repair-repair_orphan repair-orphan=1535467i
Feb 23 10:02:58 solana-validator-eu solana-validator[5112]: [2020-02-23T10:02:58.180813023Z INFO  solana_metrics::metrics] datapoint: serve_repair-repair_orphan repair-orphan=1536147i
Feb 23 10:02:58 solana-validator-eu solana-validator[5112]: [2020-02-23T10:02:58.180885828Z INFO  solana_metrics::metrics] datapoint: serve_repair-repair_orphan repair-orphan=1536567i
Feb 23 10:02:58 solana-validator-eu solana-validator[5112]: [2020-02-23T10:02:58.282431063Z INFO  solana_metrics::metrics] datapoint: serve_repair-repair_highest repair-highest-slot=1532964i repair-highest-ix=0i

Slots is always falling behind.

⠄ Validator is 140572 slots away (us:1575619 them:1716191) and falling behind at -3.2 slots/second

I have tried removing the ledger and restart the node. The snapshot is always starting at height 1532964.

Proposed Solution

[mvines]

A full log of the solana-validator process output would be very helpful to debug this

[mvines]

Ah, I'm able to reproduce this now. First my validator says:

solana_validator] Highest available snapshot slot is 1725568

and then it downloads a snapshot for a really old slot:

solana_ledger::snapshot_utils] Loaded bank for slot: 1532964

[mvines]

Some nodes, such as 206.189.221.221, seem to not be generating actual snapshot.tar.bz2 files even though they're publishing newer snapshot slots in gossip. We need to tighten down the way that a snapshot is fetched such that the requesting node can specify a specific snapshot slot/hash and if it doesn't get exactly that back then it aborts and retries (perhaps blacklisting that node).

[mvines]

The temporary workaround for Tds stage 1 is to manually download a snapshot from the bootstrap leader:

$ rm -rf ledger/
$ mkdir ledger/
$ curl http://216.24.140.155:8899/snapshot.tar.bz2 --output ledger/snapshot.tar.bz2 

and then quickly start your validator with the --no-snapshot-fetch option to ensure that it uses the snapshot.tar.bz2 that was just downloaded instead of trying to re-fetch a new one.

[mvines]

Fixed by #8482