Add local cluster tests that broadcast duplicate slots

[jstarry]

Problem

Lacking local cluster tests for faulty nodes which sends duplicate shreds for complete blocks

Summary of Changes

• Added new "duplicate node" local cluster test which runs two nodes with the same keys (one gets setup after the other)
• Added a new broadcast stage for sending duplicate shreds:
  • Partitions the cluster based on stake
  • Does not send invalid entries (which would just cause the recipient to mark the slot as dead)
  • Wait to broadcast alternative version of a slot until all shreds can be sent at once (to avoid nodes getting duplicate shreds from repair)
  • Broadcasts duplicate shreds after a configurable slot delay

[jstarry]

Test commands with log level

RUST_LOG=warn,solana_local_cluster=info RUST_BACKTRACE=1 cargo test --package solana-local-cluster --test local_cluster -- test_duplicate_node --exact --nocapture --ignored
RUST_LOG=warn,solana_local_cluster=info RUST_BACKTRACE=1 cargo test --package solana-local-cluster --test local_cluster -- test_fake_shreds_broadcast_leader --exact --nocapture --ignored

[jstarry]

cc @carllin @sakridge

[carllin]

@jstarry thanks!

[stale]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[jstarry]

@carllin we still want this to land after your changes right? If so, I'll keep it from getting going stale

[carllin]

@jstarry yeah, there's a few more changes before this one can go in, but that's the goal!

[stale]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[stale]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[stale]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[stale]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[stale]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[carllin]

Found one more edge case, fix here: carllin@e5cacb1

Broadcasts need to handle interrupted slots, otherwise they'll start from the shred index of the previous interrupted block

[stale]

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[stale]

This stale pull request has been automatically closed. Thank you for your contributions.

[codecov]

Codecov Report

Merging #13995 (e7338b1) into master (8e1e57a) will decrease coverage by 0.1%.
The diff coverage is 0.5%.

@@            Coverage Diff            @@
##           master   #13995     +/-   ##
=========================================
- Coverage    82.7%    82.6%   -0.2%     
=========================================
  Files         430      431      +1     
  Lines      120480   120674    +194     
=========================================
- Hits        99734    99728      -6     
- Misses      20746    20946    +200     

[carllin]

Tagging @AshwinSekar, these are some tests, specifically test_duplicate_shreds_broadcast_leader that tests a leader broadcasting two versions of the same block. Might be useful to integrate this into a test as well across multiple nodes (can spin up a single bad leader with the custom broadcast stage BroadcastStageType::BroadcastDuplicates I think).

[AshwinSekar]

For my understanding, why do we send out the duplicate shreds first before the real recipients?

[jstarry]

It's arbitrary. They should be received around the same time. The key part here is just that two sets of nodes get different versions of the same block and they are completely unaware that another version exists. Later, we send out a shred that belongs to the duplicate version so that nodes learn of the existence of a duplicate and start acting to fix the forking.