Add XOR Hashes to future versions + Fix RawIterate by Kbhat1 · Pull Request #85 · sei-protocol/sei-db

Kbhat1 · 2025-02-05T17:33:55Z

Describe your changes and provide context

Process: every HashRange blocks, kick off goroutine that will go back to the last time a hash was saved, compute hashes for all the key/vals in HashRange chunks, save it to pebble
Update RawIterate to return the parsed key (vs the whole key with s/k:)
Adds xor hasher to sei-db (better at this level then at sei-chain)
Adds a computeHashes function that will store the pebbledb xor hashes for future versions not just scanner

Testing performed to validate your change

ss/pebbledb/db.go

+		go func(mod string) {
+			defer close(dataCh)
+
+			_, err := db.RawIterate(mod, func(key, value []byte, ver int64) bool {
+				// Only feed data whose version is in [beginBlock..endBlock]
+				if ver >= beginBlock && ver <= endBlock {
+					dataCh <- types.RawSnapshotNode{
+						StoreKey: mod,
+						Key:      key,
+						Value:    value,
+						Version:  ver,
+					}
+				}
+				// false => keep iterating
+				return false
+			})
+			if err != nil {
+				// In your environment, you might want to handle errors differently:
+				panic(fmt.Errorf("error scanning module %s: %w", mod, err))
+			}
+		}(moduleName)


ss/util/xor_hasher.go

blindchaser · 2025-02-26T07:00:12Z

ss/util/xor_hasher.go

+	}
+	// Push all the data to its corresponding channel based on version
+	for data := range x.DataCh {
+		index := data.Version / x.NumBlocksPerWorker


may panic when index exceeds size of allChannels?

Fyi @blindchaser this was taken from the sei-chain implementation. I'm going to leave it as is for now. Can optimize this after but I wanted to keep it as the original implementation: https://github.com/sei-protocol/sei-chain/blob/main/tools/hash_verification/hasher/xor_hasher.go

blindchaser · 2025-02-26T07:18:37Z

ss/util/xor_hasher.go

+}
+
+// HashSingle computes the hash of a single data element.
+func (x XorHashCalculator) HashSingle(data []byte) []byte {


should we use pointer receiver func (x *XorHashCalculator) to match var _ HashCalculator = (*XorHashCalculator)(nil)? also it saves space

Fyi @blindchaser this was taken from the sei-chain implementation. I'm going to leave it as is for now. Can optimize this after but I wanted to keep it as the original implementation: https://github.com/sei-protocol/sei-chain/blob/main/tools/hash_verification/hasher/xor_hasher.go

ss/pebbledb/db.go

udpatil · 2025-02-26T22:05:42Z

ss/util/modules.go

+	"acc",
+	"bank",
+	"capability",
+	"distribution",


do we need to include dex for historical purposes? since the module used to exist, doing a backfill for the XOR hash should include that, right?

Good point, I’ll add it in

Fyi I'm not going to add this in @udpatil , even for the separate tool we used for backfilling historical hashes while the node is offline https://github.com/sei-protocol/sei-chain/blob/main/tools/hash_verification/pebbledb/scanner.go doesn't include it, we only really need to verify inconsistencies in the existing modules

udpatil · 2025-02-27T01:36:55Z

ss/pebbledb/db.go

+				return false
+			})
+			if err != nil {
+				panic(fmt.Errorf("error scanning module %s: %w", mod, err))


@Kbhat1 if we include dex for historical data, do we need to handle this more gracefully to be able to also process ranges that no longer have the dex store? I don't recall what happens to the underlying module store, whether it will load and have no keys to rawIterate or whether it will just fail?

I'll confirm this in a test, I think we should be okay it shouldn't throw an error but will confirm. Good call out

@udpatil check comment above

dex shouldn't matter since util.Modules will exclude dex anyway, so we won't compute hash for dex module

Kbhat1 · 2025-02-28T15:13:54Z

@udpatil @blindchaser could you both take a look again? Made some fixes, addressed comments, passing and verified on node hashes being written. Let me know if any questions.

ss/pebbledb/db.go

yzang2019 · 2025-03-03T15:14:18Z

ss/pebbledb/db.go

+				return false
+			})
+			if err != nil {
+				panic(fmt.Errorf("error scanning module %s: %w", mod, err))


dex shouldn't matter since util.Modules will exclude dex anyway, so we won't compute hash for dex module

ss/pebbledb/db.go

Kbhat1 · 2025-03-05T17:58:20Z

@yzang2019 could you take a look again when possible? All comments addressed, will merge soon

yzang2019 · 2025-03-06T03:25:40Z

ss/pebbledb/db.go

+
+	if db.config.HashRange > 0 {
+		go func(ver int64) {
+			if err := db.computeMissingRanges(ver); err != nil {


Is it possible to only run this when previous one finishes?

Fyi mutex added to ensure this

Kbhat1 · 2025-03-07T18:26:19Z

@yzang2019 lmk if you can take a look again

ss/pebbledb/db.go

+		go func(ver int64) {
+			// Try to acquire lock, return immediately if already locked
+			if !db.hashComputationMu.TryLock() {
+				return
+			}
+			defer db.hashComputationMu.Unlock()
+
+			if err := db.computeMissingRanges(ver); err != nil {
+				fmt.Printf("maybeComputeMissingRanges error: %v\n", err)
+			}
+		}(version)


…i-db into UpdateRawIterate

ss/util/xor_hasher.go

+		go func(index int, data chan types.RawSnapshotNode) {
+			defer wg.Done()
+			var hashResult []byte
+			for item := range data {
+				entryHash := x.HashSingle(Serialize(item))
+				if hashResult == nil {
+					hashResult = entryHash
+				} else {
+					hashResult = x.HashTwo(hashResult, entryHash)
+				}
+			}
+			allHashes[index] = hashResult
+		}(i, subsetChan)


Kbhat1 added 4 commits February 5, 2025 12:29

RawIterate Parsed Key

43ece0f

Add Delete Version

b62a7d9

Add future hashes

dcdcbe4

compute hashes

56d9ecc

Kbhat1 changed the title ~~RawIterate Parsed Key~~ Add XOR Hashes to future versions + Fix RawIterate Feb 21, 2025

github-advanced-security bot found potential problems Feb 21, 2025

View reviewed changes

Remove comment

a3748fe

Kbhat1 requested review from blindchaser and yzang2019 February 22, 2025 03:29

Default hash range

802cdeb

blindchaser reviewed Feb 26, 2025

View reviewed changes

hash Test

fc57c28

udpatil reviewed Feb 26, 2025

View reviewed changes

udpatil reviewed Feb 27, 2025

View reviewed changes

Kbhat1 added 2 commits February 27, 2025 12:29

Update numWorkers

37a08ea

Normalize by begin block range

f74e65e

Kbhat1 requested review from blindchaser, philipsu522 and udpatil February 28, 2025 15:14

yzang2019 reviewed Mar 3, 2025

View reviewed changes

ss/pebbledb/db.go Show resolved Hide resolved

yzang2019 reviewed Mar 4, 2025

View reviewed changes

ss/pebbledb/db.go Outdated Show resolved Hide resolved

ss/pebbledb/db.go Outdated Show resolved Hide resolved

Kbhat1 added 2 commits March 4, 2025 10:05

Cache last ranged hash

10d3883

Remove fmt print

e40d44f

Kbhat1 requested a review from yzang2019 March 5, 2025 17:58

yzang2019 reviewed Mar 6, 2025

View reviewed changes

Only one hash computation at a time

02f3bb3

Kbhat1 requested a review from yzang2019 March 7, 2025 18:26

github-advanced-security bot found potential problems Mar 7, 2025

View reviewed changes

yzang2019 approved these changes Mar 10, 2025

View reviewed changes

Kbhat1 added 4 commits March 17, 2025 10:52

Merge branch 'main' into UpdateRawIterate

7ed0e6a

Toml

c3e348d

Merge branch 'UpdateRawIterate' of https://github.com/sei-protocol/se…

dab8a8d

…i-db into UpdateRawIterate

Lint

6d1632e

github-advanced-security bot found potential problems Mar 18, 2025

View reviewed changes

Kbhat1 merged commit 40e9138 into main Mar 18, 2025
4 of 5 checks passed

Conversation

Kbhat1 commented Feb 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Describe your changes and provide context

Testing performed to validate your change

Uh oh!

Uh oh!

Check notice

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Kbhat1 Feb 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Kbhat1 commented Feb 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Kbhat1 commented Mar 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Kbhat1 commented Mar 7, 2025

Uh oh!

Check notice

Check notice

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Kbhat1 commented Feb 5, 2025 •

edited

Loading

Kbhat1 Feb 27, 2025 •

edited

Loading

Kbhat1 commented Feb 28, 2025 •

edited

Loading

Kbhat1 commented Mar 5, 2025 •

edited

Loading