Releases: node-real/bsc
State Expiry v0.1.0 Release
Introduction
Yes, we know, running a full node is painful. From NodeReal’s BSC Annual Report 2023, we observed that running a full node requires minimal storage of 1.6TB.
As of writing this blog, it already exceeded 2TB. However, only a minority of key-value pairs have been accessed in the recent period. With this in mind, BSC developers have come up with a non-consensus state expiry scheme to minimize the state data that full nodes have to store.
Unveiling the smallest full node ever
Say goodbye to full node pain: Introducing the smallest full node ever.
| Setup | Chaindata (GB) | Account Trie (GB) | Contract Trie (GB) | Account Snap (GB) | Contract Snap (GB) |
|---|---|---|---|---|---|
| PBSS | 788.93 | 37.67 | 456.98 | 12.45 | 238.08 |
| PBSS + State Expiry (not pruned) | 797.46 | 37.68 | 459.89 | 12.45 | 238.61 |
| PBSS + State Expiry (pruned trie only) | 388.07 | 37.68 | 51.84 | 12.45 | 238.61 |
| PBSS + State Expiry (pruned trie and snap) | 162.5 | 37.68 | 51.84 | 12.45 | 15.48 |
Note: Chain data here refers to all state data excluding the ancient store (older blocks, header, tx, receipt, etc) and transaction index.
Based on the result above, we can observe a 79.4% chain data size reduction by pruning away expired trie and snap key-value pairs. Another significant advantage is that even with the inclusion of additional state epoch metadata, it only contributes to an increase of 1.02% of total chain data size. With an epoch period of 1 month, each transaction will access the expired state approximately 6% of the time.
How it works
State Epoch Metadata
In this design, we introduced a new state metadata called state epoch. A state epoch is a unit measurement to determine if a state is expired or not. A state epoch period is measured using a fixed number of blocks (e.g. 1 epoch for every 100000 blocks). In our state expiry rule, once a state has been left behind the latest epoch for at least 2 epochs, then it’s considered expired and can be pruned away.
State Expiry
The state epoch metadata is stored in the branch nodes of the contract trie, as shown in the following figure:
The length of the epoch map corresponds to the length of the number of child nodes (i.e. 16), where each epoch points to the direct corresponding child node.
Every state access operation (i.e. SLOAD and SSTORE) requires the traversal from the root of the contract trie to the value in the leaf node. During the traversal process, it is possible to pass by the branch nodes and have their state epoch checked against the state expiry rule. If it’s expired, then an error is returned back and the parent process will perform a state revive operation.
During the offline expired pruning process, each contract trie is scanned and the portion of the expired subtries is evaluated. Expired subtries are pruned, deleting the trie nodes from the database and shrinking the trie. The following figure shows an example:
Revive using a RemoteDB
If an expired state has been pruned away and some state operation needs to access the expired state, then state revival needs to be performed. State revive is done by requesting an MPT proof of the expired state from an entity called remoteDB. A remoteDB is simply a regular full node that contains the entire state or an archive node. The following figure illustrates the interaction between the different types of nodes:
How to use it
Refer to this guide to experience this feature.
Future roadmap
We have the smallest full node, but the next milestone is to build the smallest and most performant full node. In the future, the development team aims to increase the performance of the full node which includes optimizing remoteDB query, pruning, revive, and more!
We built it for you
We understand that it has not been easy to run a BSC full node, and that’s why we built this feature for you. Try it out and let us know your thoughts! If you face any issues, raise a GitHub issue on the BSC repository and the team will be happy to assist you.
Conclusion
State growth is a problem, and we found a way to mitigate the issue. By labeling a state with an additional state epoch metadata, an expired state can be pruned away and recovered later using proof.
Assets
| Assets | Sha256 Checksum |
|---|---|
| mainnet.zip | aeeae3392bd51af57fb1e2eddeefff7e84d58a37f87513495348d82deee5824f |
| testnet.zip | bcb79b8731c50ddaa3a5a2e3e92afa4f04b0c04c72b6b37b8a69ecd776304f15 |
| geth_linux | 0c925b830224b0342a13d969c831ccdf7b6abefdcf42dfab347b0cf895306e8c |
| geth_mac | 7898971eca4c5453d7f696b7ed2b745759172b6297bfae8c8352b063dd9ca33d |
| geth_windows | 4e04ad84921c810834a17b930ce0ff098f3b2524a13aaff2e44c2d37dc11398e |
| geth_linux_arm64 | 3a2dbc313e76771352e11c025818b9168262a41095e80548a22fec28be69b428 |
v1.1.18_hf+Parallel-2.0_hf
Changelog
Assets
| Assets | Sha256 Checksum |
|---|---|
| mainnet.zip | 6dd6976b9c8d407e95ed99cd46f7badfa410f3f374ea3e360defab0f63fa3ed2 |
| testnet.zip | bea96c7577f7ff8fe25384a3975059350fe2955e5b953180b26bc8944b6b9542 |
| geth_linux | d13fd2f086888ee97fb9959ff6a0edadf757176e62fa61812e7dce5612092b4c |
| geth_mac | 7ef8d35302f112ef1367922fa7ea4725dece10651363e871c31cec1eebb45e41 |
| geth_windows | 242c127df98fe7a80d07ae4be052ff00344e948f5ad56a57a60ffe0f4b82e0fd |
| geth_linux_arm-5 | fb9cbd7032b02b71730f8d7ee147d28f0e510b1dd3c8685b6f7edc4430602f7c |
| geth_linux_arm-6 | 4a4c2554e1daa50dfa891e2af4be82496b8d0741dca2c87267a2ee926eea5841 |
| geth_linux_arm-7 | 01c2ea7ba5e006a3f5ed63df491eed2cb5fd2ad36963a5fb48ce8da5a4d7b2c8 |
| geth_linux_arm64 | f0d231951c5ca925dc8dce88804cb6ab4d5bc6fb479c94548f18a0ba7fffe154 |
v1.1.18_hf+Parallel-2.0
Changelog
Assets
| Assets | Sha256 Checksum |
|---|---|
| mainnet.zip | 6dd6976b9c8d407e95ed99cd46f7badfa410f3f374ea3e360defab0f63fa3ed2 |
| testnet.zip | c9c20ceb98911cc3fa7ceda3e5efbf17a3791fdc46f2f6ab13af7ac77f1a65eb |
| geth_linux | 5bfd63b8ef44fc2831837d7f8235131b1948693abd2e657ffaa1225a82cddfe0 |
| geth_mac | 0cb158dfa475ef3607a667a236f2d410e63892db714584274b3ff0974607857c |
| geth_windows | 387208d9d8222a2a1ad23224f69b2d56fae4d48925943c0e51f5f5d6d19b007d |
| geth_linux_arm-5 | a5f6ca37a1e18cf242344412bb459908dcbd4f293d6d434da82fb47a424b2678 |
| geth_linux_arm-6 | b21e6e5ff8edcd29e2e1c71c3664b32f1338332f0e12a351e6494caa8057b9cb |
| geth_linux_arm-7 | 46c196700df30ace06d4b64ae9be186f2e87a01c8067419781d059ac0fd48546 |
| geth_linux_arm64 | bae302c03d94ebd93ec17340ee527e46ee3f52e87521aa6a9fd66ca8c4471dab |
v1.1.16+Parallel-2.0
About
This is a dedicated release for the feature of Parallel EVM, which was proposed in BEP130;
Since the feature is a bit complicated, we targeted to implement it with 3 phases:
Parallel 1.0: the main components and procedure will be implemented in phase 1.0.
Parallel 2.0: a performance enhancement phase, to redesign the pipeline, solve the conflict challenges, shadowed slot...
Parallel 3.0: to support validator mode, parallel mining would be a bit different from the importing procedure, it will be started later.
We release Parallel 2.0 right now, in case people of the community may interest and wanna to have a try of it.
For the architecture designed of Parallel 2.0, you may refer #39
Right now, it support BSC mainnet & testnet only; Ethereum can be supported too, but some porting efforts may be needed.
We have done lots of stable tests, we covered almost all the blocks of BSC mainnet & testnet and lots of corner cases have been fixed. But we still can not guarantee that this release is bug free, if you encounter any problem, pls let us know by raising issues in node-real bsc repo.
Parallel transaction running is a hot topic, it would be great if more people can join it and push it forward.
Usage
It is quite easy to enable the parallel evm mode: geth --syncmode full --parallel
or to set the parallel number manually: geth --syncmode full --parallel --parallel.num 10
--parallel Enable the experimental parallel transaction execution mode, only valid in full sync mode (default = false)
--parallel.num value Number of slot for transaction execution, only valid in parallel mode (runtime calculated, no fixed default value)
v1.1.10+Parallel-1.0
Changelog
Assets
| Assets | Sha256 Checksum |
|---|---|
| mainnet.zip | 6dd6976b9c8d407e95ed99cd46f7badfa410f3f374ea3e360defab0f63fa3ed2 |
| testnet.zip | c9c20ceb98911cc3fa7ceda3e5efbf17a3791fdc46f2f6ab13af7ac77f1a65eb |
| geth_linux | 579ad0cbdbc87abcc6d87a64b1420624a1d51defb567c340407bb215a6fc92b4 |
| geth_mac | 58dc67c93931639e88971921bedbf12a0835af15b3101a35a7c12fe3cdfc4b7a |
| geth_windows | 1657f82c2c508630b5e5a88690d31bf81f3d1248638bc9ea2854b3dc992eeb1f |
| geth_linux_arm5 | feb99f01d2ddfe563ee1ddd2389378cbbf0bd4b03abaa864a5e3de0fc9d2424b |
| geth_linux_arm6 | 0e512a2d138f38f8c3df1d8a8aac8eadf28dd515c5ba66face654f1c8b30ece0 |
| geth_linux_arm7 | 8ff6e97463357b64bdbe326c98c146ba56761df6848cbc6976cb36690730c48e |
| geth_linux_arm64 | 8bc70dc8c08a169c29393047d3aa3ec4c1693267b8626c0309c0be31449e07b0 |
v1.1.10
Changelog
FEATURE
- #885 add Euler Hardfork, including BEP-127 and BEP-131
BUGFIX
- #856 fix logic issue: handlers.removePeer() is called twice
- #860 fix:defer bloomprocessor close
- #877 fix validator pipecommit issue
- #870 fix:Shift panic for zero length of heads
Assets
| Assets | Sha256 Checksum |
|---|---|
| mainnet.zip | 6dd6976b9c8d407e95ed99cd46f7badfa410f3f374ea3e360defab0f63fa3ed2 |
| testnet.zip | c9c20ceb98911cc3fa7ceda3e5efbf17a3791fdc46f2f6ab13af7ac77f1a65eb |
| geth_linux | ba89651ddadc243162b8d8cca263e87cd3663f15217d78784a0e286b7e8fddca |
| geth_mac | e900220ec49c6981831a01fe1d962f8a9901f5eb181130e41d121de75a7bf7fc |
| geth_windows | 285bb9f0aa5cb10c13a47facb5c47fbfc281b5df8a513100d3950517c835e9c2 |
| geth_linux_arm5 | 4cfe6c9d3024faee6e59c1a17f24f9b8cf2e7b7fa9f7f685ddf408ac9f5fa80d |
| geth_linux_arm6 | d00c2e2231a9c7db1cb9774318dd628435ce686afbadd5f883d509bc72395c1b |
| geth_linux_arm7 | 8758652e2eb3186f4cd5ceaf786a3698608c5cd4d8fbdd0f5160de6159f731bf |
| geth_linux_arm64 | 50b06b292be758630beaf17decbaa5386e87ad2afe5749591ed653ec8ada38e3 |
v1.1.9
Changelog
IMPROVEMENT
- #792 add shared storage for prefetching state data
- #795 implement state verification pipeline in pipecommit
- #803 prefetch state data during the mining process
- #812 skip verification on account storage root to tolerate with fastnode when doing diffsync
- #818 add shared storage to the prefetcher of miner
- #820 disable diffsync when pipecommit is enabled
- #830 change the number of prefetch threads
BUGFIX
- #797 fix race condition on preimage in pipecommit
- #808 fix code of difflayer not assign when new smart contract created
- #817 fix bugs of prune block tool
- #834 fix deadlock when failed to verify state root in pipecommit
- #835 fix deadlock on miner module when failed to commit trie
- #842 fix invalid nil check of statedb in diffsync
Assets
| Assets | Sha256 Checksum |
|---|---|
| mainnet.zip | 6dd6976b9c8d407e95ed99cd46f7badfa410f3f374ea3e360defab0f63fa3ed2 |
| testnet.zip | c9c20ceb98911cc3fa7ceda3e5efbf17a3791fdc46f2f6ab13af7ac77f1a65eb |
| geth_linux | c69ade6eb11399aa2296933279a2760f947092d77ddc4b4c1a8bda95707db107 |
| geth_mac | 5648f57f988752a2e0cd8fc5a6cfccdc905d01e2604788766e59ca66e9316066 |
| geth_windows | 03aa459e2b49c8a095d388369f9650e52df540e759497a0da203f1c7d5cd4d21 |
| geth_linux_arm5 | da6494d7cc0c7d7553c97cfcfd48fb55dd106b9d00c39c0f5bbb49bf39b18d63 |
| geth_linux_arm6 | ea3be85fd42f701c45104bef2bfbae7209833e9c7f96677bf2a8545e3370d5cb |
| geth_linux_arm7 | caba40242d5d5ed0615ec197ef3349b6dd7372c4fa92ad556ce16963ce43a9e2 |
| geth_linux_arm64 | d5d2c75ae6b8a2857435ea1fe823284d695aceb863e0d4ed9a0dbf983bec851b |
v1.1.8+Parallel-1.0
Changelog
Assets
| Assets | Sha256 Checksum |
|---|---|
| mainnet.zip | 4a2ad47362afa6c387ed9acd7f8f050b356e61ac77b8094b6116aa17ae90972b |
| testnet.zip | c9c20ceb98911cc3fa7ceda3e5efbf17a3791fdc46f2f6ab13af7ac77f1a65eb |
| geth_linux | de1529750abb0d8f3d8901e58da2232110d2f60c1e9db3eccddd1bf6241b8001 |
| geth_mac | e3ae4866b684e736f06f6fc0972e5273d8521c27189778a743b860f071ebebd3 |
| geth_windows | f685d04db77125d60434db1b80a9e8b264753a87efbdf1caeaf81dad2d7a3a26 |
| geth_linux_arm5 | 7b2505f052bdb52e10e67ada4bb3354c0c35212fb9ccbfc414ad415bb50efb12 |
| geth_linux_arm6 | cd1c98a24f4e254bb6f2f5b84b51a54438a47289f5425718e8a669a7303c6f40 |
| geth_linux_arm7 | d4256cc659f381df0f1a45d940681a20e5628796a6eab6f1df52dda278348e9b |
| geth_linux_arm64 | 5cbff129ffe4dfa8f19a411ea7153f38f36a9034e66291a69543a7a834f4e576 |