Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(pageserver): add metadata key range and aux key encoding #7401

Merged
merged 7 commits into from
Apr 23, 2024

Conversation

skyzh
Copy link
Member

@skyzh skyzh commented Apr 17, 2024

Problem

Extracted from #7375. We assume everything >= 0x80 are metadata keys. AUX file keys are part of the metadata keys, and we use 0x90 as the prefix for AUX file keys.

The AUX file encoding is described in the code comment. We use xxhash128 as the hash algorithm. It seems to be portable according to the introduction,

xxHash is an Extremely fast Hash algorithm, processing at RAM speed limits. Code is highly portable, and produces hashes identical across all platforms (little / big endian).

...though whether the Rust version follows the same convention is unknown and might need manual review of the library. Anyways, we can always change the hash algorithm before rolling it out in staging/end-user, and I made a quick decision to use xxhash here because it generates 128b hash + portable. We can save the discussion of which hash algorithm to use later.

Summary of changes

Checklist before requesting a review

  • I have performed a self-review of my code.
  • If it is a core feature, I have added thorough tests.
  • Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
  • If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

  • Do not forget to reformat commit message to not include the above checklist

Signed-off-by: Alex Chi Z <chi@neon.tech>
@skyzh skyzh requested a review from a team as a code owner April 17, 2024 07:10
@skyzh skyzh requested review from problame and jcsp April 17, 2024 07:10
@skyzh
Copy link
Member Author

skyzh commented Apr 17, 2024

I'm thinking about adding some test cases for key ordering, maybe in later pull requests. Ordering of keys are not important as we are using hashing for AUX keys. It might be important if at some point we transform to variable-sized keys or have some metadata keys requiring ordering when scanning.

Signed-off-by: Alex Chi Z <chi@neon.tech>
Copy link

github-actions bot commented Apr 17, 2024

2766 tests run: 2646 passed, 0 failed, 120 skipped (full report)


Flaky tests (1)

Postgres 16

  • test_vm_bit_clear_on_heap_lock: debug

Code coverage* (full report)

  • functions: 28.1% (6482 of 23065 functions)
  • lines: 46.9% (45932 of 97884 lines)

* collected from Rust tests only


The comment gets automatically updated with the latest test results
e48f8d9 at 2024-04-23T15:23:06.848Z :recycle:

@skyzh
Copy link
Member Author

skyzh commented Apr 17, 2024

checked by default non-rel-block key is on shard 0 as in key_is_shard0

pageserver/src/aux_file.rs Outdated Show resolved Hide resolved
pageserver/src/aux_file.rs Show resolved Hide resolved
pageserver/src/aux_file.rs Show resolved Hide resolved
pageserver/src/aux_file.rs Outdated Show resolved Hide resolved
pageserver/src/aux_file.rs Show resolved Hide resolved
pageserver/src/aux_file.rs Outdated Show resolved Hide resolved
libs/pageserver_api/src/key.rs Outdated Show resolved Hide resolved
Signed-off-by: Alex Chi Z <chi@neon.tech>
Signed-off-by: Alex Chi Z <chi@neon.tech>
pageserver/src/aux_file.rs Outdated Show resolved Hide resolved
pageserver/src/aux_file.rs Outdated Show resolved Hide resolved
pageserver/src/aux_file.rs Show resolved Hide resolved
@skyzh skyzh mentioned this pull request Apr 22, 2024
24 tasks
Signed-off-by: Alex Chi Z <chi@neon.tech>
@skyzh skyzh requested review from arpad-m and jcsp April 22, 2024 15:13
@skyzh
Copy link
Member Author

skyzh commented Apr 23, 2024

reserving merge conflict with #7404 and not adding aux key range into is_inherited_keys. need to fix it in the future.

Signed-off-by: Alex Chi Z <chi@neon.tech>
@skyzh skyzh enabled auto-merge (squash) April 23, 2024 15:03
@skyzh skyzh merged commit 89f023e into main Apr 23, 2024
46 of 47 checks passed
@skyzh skyzh deleted the skyzh/metadata-key-encoding branch April 23, 2024 15:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants