(WIP) Prototype of the tiered accounts storage #30626
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This was referenced Mar 7, 2023
yhchiang-sol
force-pushed
the
tiered-storage
branch
5 times, most recently
from
886f917
to
6716225
Compare
yhchiang-sol
force-pushed
the
tiered-storage
branch
3 times, most recently
from
37ce4ba
to
fbada9a
Compare
yhchiang-sol
force-pushed
the
tiered-storage
branch
11 times, most recently
from
529a7d0
to
5efd2db
Compare
yhchiang-sol
force-pushed
the
tiered-storage
branch
from
5efd2db
to
81cbc5b
Compare
|
In your proposal's doc, please correct the front matter to allow the docs to generate in the future: |
yhchiang-sol
force-pushed
the
tiered-storage
branch
6 times, most recently
from
c91d56a
to
9a3257d
Compare
yhchiang-sol
force-pushed
the
tiered-storage
branch
5 times, most recently
from
d8b3716
to
c0987fa
Compare
yhchiang-sol
force-pushed
the
tiered-storage
branch
11 times, most recently
from
83a275b
to
aec582f
Compare
yhchiang-sol
force-pushed
the
tiered-storage
branch
from
aec582f
to
8fe151d
Compare
yhchiang-sol
force-pushed
the
tiered-storage
branch
from
8fe151d
to
ed65842
Compare
yhchiang-sol
force-pushed
the
tiered-storage
branch
from
ed65842
to
f0ef926
Compare
yhchiang-sol
commented
Jul 26, 2023
Comment on lines
+88
to
+228
| account.data(), | ||
| account.owner(), | ||
| account.executable(), | ||
| pubkey, | ||
| hash, | ||
| write_version, | ||
| &mut cursor, | ||
| &mut footer, | ||
| &mut account_metas, | ||
| &mut account_pubkeys, | ||
| &mut owners_table, | ||
| data_block_writer, | ||
| &mut buffered_account_metas, | ||
| &mut buffered_account_pubkeys, | ||
| &mut dummy_hash, | ||
| &mut account_index_entries, | ||
| ) | ||
| .unwrap(); | ||
| } else { | ||
| data_block_writer = self | ||
| .write_single_account( | ||
| 0, // lamports | ||
| u64::MAX, // rent_epoch, | ||
| &default_data, | ||
| &default_pubkey, | ||
| false, // executable | ||
| pubkey, | ||
| hash, | ||
| write_version, | ||
| &mut cursor, | ||
| &mut footer, | ||
| &mut account_metas, | ||
| &mut account_pubkeys, | ||
| &mut owners_table, | ||
| data_block_writer, | ||
| &mut buffered_account_metas, | ||
| &mut buffered_account_pubkeys, | ||
| &mut dummy_hash, | ||
| &mut account_index_entries, | ||
| ) | ||
| .unwrap(); | ||
| } | ||
| } | ||
|
|
||
| // Persist the last block if any | ||
| if buffered_account_metas.len() > 0 { | ||
| self.flush_account_block( | ||
| &mut cursor, | ||
| &mut footer, | ||
| &mut account_metas, | ||
| &mut account_pubkeys, | ||
| &mut buffered_account_metas, | ||
| &mut buffered_account_pubkeys, | ||
| data_block_writer, | ||
| ) | ||
| .ok()?; | ||
| } | ||
|
|
||
| assert_eq!(buffered_account_metas.len(), 0); | ||
| assert_eq!(buffered_account_pubkeys.len(), 0); | ||
| assert_eq!(footer.account_entry_count, account_metas.len() as u32); | ||
|
|
||
| self.write_account_pubkeys_block(&mut cursor, &mut footer, &account_index_entries) | ||
| .ok()?; | ||
|
|
||
| self.write_owners_block(&mut cursor, &mut footer, &owners_table.owners_vec) | ||
| .ok()?; | ||
|
|
||
| footer.write_footer_block(&self.storage).ok()?; | ||
|
|
||
| assert_eq!(account_metas.len(), account_pubkeys.len()); | ||
| assert_eq!(account_metas.len(), len - skip); | ||
|
|
||
| let mut stored_accounts_info: Vec<StoredAccountInfo> = Vec::with_capacity(len); | ||
| for _ in skip..len { | ||
| stored_accounts_info.push(StoredAccountInfo { offset: 0, size: 0 }); | ||
| } | ||
| for i in 0..account_metas.len() { | ||
| let index = input_pubkey_map.get(&account_pubkeys[i]).unwrap(); | ||
|
|
||
| // of ALIGN_BOUNDARY_OFFSET, while cold storage actually talks about index | ||
| // instead of offset. | ||
| stored_accounts_info[*index].offset = i * ALIGN_BOUNDARY_OFFSET; | ||
| stored_accounts_info[*index].size = W::stored_size(&footer, &account_metas, i); | ||
| } | ||
| match footer.account_meta_format { | ||
| AccountMetaFormat::Hot => info!( | ||
| "[Hot] append_accounts successfully completed. Footer: {:?}", | ||
| footer | ||
| ), | ||
| /* | ||
| AccountMetaFormat::Cold => info!( | ||
| "[Cold] append_accounts successfully completed. Footer: {:?}", | ||
| footer | ||
| ), | ||
| */ | ||
| } | ||
|
|
||
| Some(stored_accounts_info) | ||
| } |
There was a problem hiding this comment.
@brooksprumo: FYI
github-actions
bot
added
the
stale
yhchiang-sol
added
work in progress
wen-coding
pushed a commit
to wen-coding/solana
that referenced
this pull request
Aug 15, 2023
#### Summary of Changes This PR introduces HotAccountMeta, the storage and in-memory struct of the metadata struct for a hot account. #### Test Plan Unit tests are included in this PR. Tested in mnb w/ the prototype implementation of the tiered account storage (solana-labs#30626)
wen-coding
pushed a commit
to wen-coding/solana
that referenced
this pull request
Aug 15, 2023
#### Summary of Changes This PR introduces HotAccountMeta, the storage and in-memory struct of the metadata struct for a hot account. #### Test Plan Unit tests are included in this PR. Tested in mnb w/ the prototype implementation of the tiered account storage (solana-labs#30626)
github-actions
bot
added
the
stale
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is the WIP prototype of tiered accounts storage.