Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DupSort of hash state #896

Merged
merged 12 commits into from
Aug 12, 2020
Merged

DupSort of hash state #896

merged 12 commits into from
Aug 12, 2020

Conversation

AskAlexSharov
Copy link
Collaborator

@AskAlexSharov AskAlexSharov commented Aug 10, 2020

Main idea:

CurrentStateBucket layout which utilises DupSort feature of LMDB (store multiple values inside 1 key).
LMDB stores multiple values of 1 key in B+ tree (as keys which have no values).
It allows do Seek by values and other features of LMDB.
You can think about it as Sub-Database which doesn't store any values.

-------------------------------------------------------------
       key              |            value
-------------------------------------------------------------
[acc_hash]              | [acc_value]
[acc_hash]+[inc]        | [storage1_hash]+[storage1_value]
                        | [storage2_hash]+[storage2_value] // this value has no own key. it's 2nd value of [acc_hash]+[inc] key.
                        | [storage3_hash]+[storage3_value]
                        | ...
[acc_hash]+[old_inc]    | [storage1_hash]+[storage1_value]
                        | ...
[acc2_hash]             | [acc2_value]
                        ...

On 10.5M block CurrentState bucket of this layout using 8Gb (40Gb in current layout)

Notice:

  • This PR touching only hashed state, not plain state - to minimize risk. But env variable "DUPSORT_PLAIN" still works.
  • Implementation is fully done inside kv_lmdb.go but in future I plan change KV interface closer to LMDB interface and move logic to Trie Loader. (feel free to push back and force me move logic to loader now)

@AskAlexSharov AskAlexSharov force-pushed the dupsort_hash_state branch 2 times, most recently from 8398018 to 027e201 Compare August 11, 2020 11:28
@AskAlexSharov AskAlexSharov merged commit 760fe0f into master Aug 12, 2020
@AlexeyAkhunov AlexeyAkhunov deleted the dupsort_hash_state branch February 3, 2021 19:06
battlmonstr pushed a commit that referenced this pull request Sep 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants