Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Storage] Introduce support for namespace in state store #324

Merged
merged 1 commit into from Mar 29, 2022

Conversation

sitalkedia
Copy link
Contributor

@sitalkedia sitalkedia commented Mar 26, 2022

Motivation

First PR for introducing fine-grained state storage support. This diff introduces a concept of namespace in the state store - with only account blob based namespace support to start with. The key to the state store for a namespace object can be namespace_+ rawkey. This way we can support different namespace objects within the same state store with the guarantee that there will not be any key collision. In order to support this, all account related concept has been moved out of the storage layer and instead the storage layer majorly understands the concept of StateStoreKey and StateStoreValue, which are arbitrary byte arrays. Please note that this is a major format change on the storage side and is not backward compatible - we need to wipe out the previous data from dev-net before rolling this out.

The diff touches many files because of the extensive API change, however, important changes to be reviewed at

types/src/state_store/state_store_key.rs
types/src/state_store/state_store_value.rs
storage/storage-interface/src/state_view.rs
storage/aptosdb/src/state_store/mod.rs
execution/executor/src/components/apply_chunk_output.rs

Test Plan

Modified existing UTs for new API interface.

@aptos-bot aptos-bot added this to In Review in bors Mar 26, 2022
@netlify
Copy link

netlify bot commented Mar 26, 2022

Deploy Preview for aptos-developer-docs canceled.

Name Link
🔨 Latest commit 90646f5
🔍 Latest deploy log https://app.netlify.com/sites/aptos-developer-docs/deploys/6242971c2a3c4500095f1b35

@sitalkedia
Copy link
Contributor Author

/canary

@aptos-bot aptos-bot moved this from In Review to Canary in bors Mar 26, 2022
@aptos-bot
Copy link
Contributor

💔 Test Failed - build-test-deploy

@aptos-bot aptos-bot moved this from Canary to In Review in bors Mar 26, 2022
@sitalkedia
Copy link
Contributor Author

/canary

aptos-bot pushed a commit that referenced this pull request Mar 26, 2022
Closes: #324
@aptos-bot aptos-bot moved this from In Review to Canary in bors Mar 26, 2022
@sitalkedia
Copy link
Contributor Author

/canary

@aptos-bot
Copy link
Contributor

This PR is already being canaried

@aptos-bot
Copy link
Contributor

💔 Test Failed - build-test-deploy

@aptos-bot aptos-bot moved this from Canary to In Review in bors Mar 26, 2022
@sitalkedia
Copy link
Contributor Author

/canary

@aptos-bot aptos-bot moved this from In Review to Canary in bors Mar 26, 2022
@aptos-bot
Copy link
Contributor

Forge run: https://circleci.com/gh/aptos-labs/aptos-core/5977
Forge Test Result: all up : 1966 TPS, 2236 ms latency, 3400 ms p99 latency,no expired txns

@aptos-bot
Copy link
Contributor

💔 Test Failed - build-test-deploy

@aptos-bot aptos-bot moved this from Canary to In Review in bors Mar 26, 2022
@sitalkedia
Copy link
Contributor Author

/canary

@aptos-bot aptos-bot moved this from In Review to Canary in bors Mar 26, 2022
@aptos-bot
Copy link
Contributor

Forge run: https://circleci.com/gh/aptos-labs/aptos-core/5983
Forge Test Result: all up : 1858 TPS, 2353 ms latency, 3300 ms p99 latency,no expired txns

@aptos-bot
Copy link
Contributor

💔 Test Failed - build-test-deploy

@aptos-bot aptos-bot moved this from Canary to In Review in bors Mar 26, 2022
@sitalkedia
Copy link
Contributor Author

/canary

@aptos-bot aptos-bot moved this from In Review to Canary in bors Mar 26, 2022
aptos-bot pushed a commit that referenced this pull request Mar 28, 2022
@aptos-bot aptos-bot moved this from In Review to Canary in bors Mar 28, 2022
@aptos-bot aptos-bot moved this from Canary to In Review in bors Mar 28, 2022
@aptos-bot
Copy link
Contributor

Forge run: https://circleci.com/gh/aptos-labs/aptos-core/6238
Forge Test Result: all up : 1801 TPS, 2430 ms latency, 5100 ms p99 latency,no expired txns

@sitalkedia
Copy link
Contributor Author

@msmouse - I checked the failed tests and it seems like these are due to the new hash value of the keys creating different number of intermediate nodes as compared to the previous hashing scheme, I have updated the test. Please let me know your thoughts - cacb3f2

@sitalkedia
Copy link
Contributor Author

/canary

aptos-bot pushed a commit that referenced this pull request Mar 29, 2022
Closes: #324
@aptos-bot aptos-bot moved this from In Review to Canary in bors Mar 29, 2022
@msmouse
Copy link
Contributor

msmouse commented Mar 29, 2022

@sitalkedia Makes sense! (And it's a tricky one!)

@msmouse
Copy link
Contributor

msmouse commented Mar 29, 2022

I think we can squash the two commits before landing this one.

@aptos-bot
Copy link
Contributor

Forge run: https://circleci.com/gh/aptos-labs/aptos-core/6268
Forge Test Result: all up : 1936 TPS, 2269 ms latency, 3150 ms p99 latency,no expired txns

@aptos-bot
Copy link
Contributor

☀️ Canary successful

@aptos-bot aptos-bot moved this from Canary to In Review in bors Mar 29, 2022
@sitalkedia
Copy link
Contributor Author

/land

@aptos-bot aptos-bot moved this from In Review to Queued in bors Mar 29, 2022
@aptos-bot
Copy link
Contributor

🔒 Merge Conflict

@aptos-bot aptos-bot moved this from Queued to In Review in bors Mar 29, 2022
@sitalkedia
Copy link
Contributor Author

/land

@aptos-bot aptos-bot moved this from In Review to Queued in bors Mar 29, 2022
@aptos-bot aptos-bot moved this from Queued to Testing in bors Mar 29, 2022
@aptos-bot
Copy link
Contributor

Forge run: https://circleci.com/gh/aptos-labs/aptos-core/6300
Forge Test Result: Forge test runner is terminated

@aptos-bot aptos-bot removed this from Testing in bors Mar 29, 2022
@aptos-bot aptos-bot merged commit 90646f5 into aptos-labs:main Mar 29, 2022
@aptos-bot
Copy link
Contributor

Forge run: https://circleci.com/gh/aptos-labs/aptos-core/6306
Forge Test Result: ``

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[storage][statedb] M0: Make it possible to store different types of objects, in addition to account blob
3 participants