Later updates to EpochSlots can overwrite earlier updates

[carllin]

Problem

We currently track 255 EpochSlots bitvectors that are updated via push_epoch_slots(slots) where we:

1. Read out all the individually indexed EpochSlots here:

   solana/gossip/src/cluster_info.rs

   Line 907 in 0b0b9d9

   let epoch_slots = gossip.crds.get(&label)?.value.epoch_slots()?;

2. Update the appropriate EpochSlots, then push them to the pending push queue

   solana/gossip/src/cluster_info.rs

   Line 948 in 0b0b9d9

   self.local_message_pending_push_queue

An issue arises when there are two updates to the same indexed EpochSlots structure (say index 0), before the asynchronous flush of the pending flush queue happens in the gossip thread, so the updated value is not yet observable in the Crds table.

In this case, the second update will read out a stale EpochSlots from the Crds table in step 1) above that doesn't include the first update, updates that value with the second update, then pushes that item to the pending flush queue. Then when the push happens later, the later update doesn't have the first update, thereby overwriting the first update when both values are pushed to the network.

We could probably simulate the problem in this test if we got rid of the calls to flush_push_queue() in between the updates

solana/gossip/src/cluster_info.rs

Lines 4136 to 4137 in 0b0b9d9

	cluster_info.push_epoch_slots(&range[..16000]);
	cluster_info.flush_push_queue();

.

Proposed Solution

Should we also check the push queue for more recent versions of the same message label in crds.get()?

[github-actions]

This issue has been automatically locked since there has not been any activity in past 7 days after it was closed. Please open a new issue for related bugs.