refactor(storage): remove vnode pruning from state store and keyspace

[xx01cyx]

What's changed and what's your intention?

Changes

According to our offline discussion, we'll encode vnode into storage key, instead of passing it directly into keyspace and state store. Therefore, relevant interfaces should be modified. This PR do the following refactors:

• Remove vnode from keyspace
• Remove vnode from state store interfaces (get, scan, iter, etc.)
• Discard read pruning by vnode in Hummock
• Discard struct VNodeBitmap

Limitations

VNodeBitmap in SST meta still exists. This should be discarded as well to make our compaction no longer consistent-hash-aware.

Checklist

• All checks passed in ./risedev check (or alias, ./risedev c)

[codecov]

Codecov Report

Merging #3208 (84d7575) into main (6e7ca58) will increase coverage by 0.12%.
The diff coverage is 73.45%.

❗ Current head 84d7575 differs from pull request most recent head 669ecd7. Consider uploading reports for the commit 669ecd7 to get more accurate results

@@            Coverage Diff             @@
##             main    #3208      +/-   ##
==========================================
+ Coverage   73.50%   73.62%   +0.12%     
==========================================
  Files         746      744       -2     
  Lines      101925   101858      -67     
==========================================
+ Hits        74916    74996      +80     
+ Misses      27009    26862     -147     

Flag	Coverage Δ
rust	73.62% <73.45%> (+0.12%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
src/bench/ss_bench/operations/get.rs	0.00% <0.00%> (ø)
...rc/bench/ss_bench/operations/prefix_scan_random.rs	0.00% <ø> (ø)
src/common/src/hash/key.rs	85.44% <ø> (ø)
src/common/src/lib.rs	100.00% <ø> (ø)
src/ctl/src/cmd_impl/hummock/list_kv.rs	0.00% <0.00%> (ø)
src/meta/src/manager/hash_mapping.rs	97.39% <ø> (ø)
src/meta/src/stream/meta.rs	48.96% <ø> (ø)
src/meta/src/stream/scheduler.rs	88.53% <ø> (ø)
src/meta/src/stream/stream_manager.rs	70.74% <ø> (+0.02%)	⬆️
src/storage/src/hummock/snapshot_tests.rs	94.68% <ø> (ø)
... and 62 more

📣 Codecov can now indicate which changes are the most critical in Pull Requests. Learn more

[soundOfDestiny]

as PR title

[fuyufjh]

LGTM

[skyzh]

I'd like to hold this PR for a little bit. Looks like a big decision indeed 🤣

[xx01cyx]

I'd like to hold this PR for a little bit.

I'd like to know the concern 🤔

[fuyufjh]

I'd like to hold this PR for a little bit. Looks like a big decision indeed 🤣

The changes look huge but it's primarily removing an argument from functions. No real logic touched. <- That's why I clicked approve button 😁

[skyzh]

Thought for a moment, I have no clear idea on this change. Also thought about some edge use cases (like simple agg, MV and index), and all of them can still work without passing vnode to the storage layer to do any filtering. I think executors can still fit in the current design, with only vnode in compute layer and without vnode in storage layer.

[xxchan]

Then how does batch RowSeqScan partition? 😇

[xxchan]

BTW, does #2887 also become unnecessary (executors doesn't need vnode_bitmap)?

[BugenZhao]

Then how does batch RowSeqScan partition? 😇

With the vnode prepending to the key, it seems partition scan can be simply implemented with range scan with the vnode prefix?

[xxchan]

Then how does batch RowSeqScan partition? 😇

With the vnode prepending to the key, it seems partition scan can be simply implemented with range scan with the vnode prefix?

So each scan executor's vnodes will be a consecutive range (instead of consistent hash)?

[fuyufjh]

Then how does batch RowSeqScan partition? 😇

With the vnode prepending to the key, it seems partition scan can be simply implemented with range scan with the vnode prefix?

So each scan executor's vnodes will be a consecutive range (instead of consistent hash)?

As discussed in Patrick's proposal, the new row format will be like:

table_id, vnode_id, pk_columns...

Thus, partition by consistent-hashing can be represented as a set of prefix range of keys.

[xx01cyx]

BTW, does #2887 also become unnecessary (executors doesn't need vnode_bitmap)?

According to the new design, we won't rely on vnode bitmap to do read pruning or compaction, so exactly, they are not required by executors.