implements copy-on-write for staked-nodes

[behzadnouri]

Problem

Bank::staked_nodes and Bank::epoch_staked_nodes redundantly clone
staked-nodes HashMap even though an immutable reference will suffice:
https://github.com/solana-labs/solana/blob/a9014cece/runtime/src/vote_account.rs#L77

Summary of Changes

This commit implements copy-on-write semantics for staked-nodes by
wrapping the underlying HashMap in Arc<...>.

[codecov]

Codecov Report

Merging #19090 (6e3799e) into master (8878f52) will increase coverage by 0.0%.
The diff coverage is 100.0%.

@@           Coverage Diff           @@
##           master   #19090   +/-   ##
=======================================
  Coverage    82.8%    82.8%           
=======================================
  Files         449      449           
  Lines      128281   128312   +31     
=======================================
+ Hits       106280   106319   +39     
+ Misses      22001    21993    -8     

[carllin]

hmmmm where the old code persisted the new pubkey in self.staked_nodes, does this clone self.staked_nodes and only add the new entry to the local, cloned copy of staked_nodes? This won't persist the insert into the self.staked_nodes for other callers to observe right?

From what I can tell this function is only called from Bank::store_account() which is used in ledger-tool and expects the store to be persisted.

[behzadnouri]

It clones the inner pointer, so the changes are persisted.
Please see the added test_staked_nodes_cow or the example in Arc::make_mut:
https://doc.rust-lang.org/std/sync/struct.Arc.html#method.make_mut

[carllin]

I see, I mixed up what was being cloned, thanks for the clarification.

And the cost of cloning is acceptable because Bank::store_account() is only used in ledger-tool where there are no other outstanding references that would trigger this clone?

To avoid silent perf issues in the future if somebody calls store_account more intensively, should we be more explicit about this and use get_mut() instead of make_mut() and return a boolean from this function to indicate success/failure,

[behzadnouri]

This code only makes a clone if:

• .staked_nodes was invoked since last mutation (i.e. since last make_mut).
• That returned Arc<...> is still alive when you want to mutate the hash-map again (i.e. when you call make_mut).

Current master code is making a clone every time .staked_ndoes is invoked:
https://github.com/solana-labs/solana/blob/fd937548a/runtime/src/vote_account.rs#L77

So this code always does fewer clones than current master code, and make_mut does not necessarily make a clone.

[carllin]

That returned Arc<...> is still alive when you want to mutate the hash-map again (i.e. when you call make_mut).

Right, I'm worried about these calls

solana/ledger-tool/src/main.rs

Lines 2064 to 2067 in fd93754

	bank.store_account(
	&faucet_pubkey,
	&AccountSharedData::new(faucet_lamports, 0, &system_program::id()),
	);

silently incurring a clone in the future if some other thread is holding another reference to this Arc. I feel like being explicit with a result from something like a store_without_clone that we can assert on makes this more transparent, and avoids silently incurring this cost.

[behzadnouri]

Can you please describe the scenario that previously it was not making a clone and now it does?

1. Previously every time one calls .staked_nodes() it had to make a clone. No exceptions there.
2. Now it clones only if all these happen:
   • you call .staked_nodes() (since the very last time make_mut was invoked).
   • and hold on to the returned Arc<...>
   • and then also try to mutate stake-nodes while holding to that Arc.
     • After which there is no more clones if you mutate staked-nodes again.

(2) is a subset of (1), so for every time that this code ends up making a clone, it was making a clone previously anyways; and so this code is removing clones, not adding clones.

[carllin]

Can you please describe the scenario that previously it was not making a clone and now it does?

I don't think there is one currently, was referring to future-proofing calls like

solana/ledger-tool/src/main.rs

Lines 2064 to 2067 in fd93754

	bank.store_account(
	&faucet_pubkey,
	&AccountSharedData::new(faucet_lamports, 0, &system_program::id()),
	);

in case anybody does add a case where they hold a returned Arc. This way anybody calling these functions will be cognizant of when a clone is incurred.

[behzadnouri]

Still like current call-sites, this code cannot introduce any new clones (where with the old code there was no clone).
I would be happy to discuss this more over a call or something next time we chat. For now, any objections to merge this?

[carllin]

I would be happy to discuss this more over a call or something next time we chat

Sounds good, will probably be clearer with a discussion. No other objections!

[behzadnouri]

thank you 🙏

[sakridge]

lgtm