Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

pageserver: shard-aware WAL ingest #6025

Closed
Tracked by #5507
jcsp opened this issue Dec 4, 2023 · 0 comments · Fixed by #6024
Closed
Tracked by #5507

pageserver: shard-aware WAL ingest #6025

jcsp opened this issue Dec 4, 2023 · 0 comments · Fixed by #6024
Assignees
@jcsp
Labels
c/storage/pageserver Component: storage: pageserver t/feature Issue type: feature, for new features or requests

Comments

@jcsp
Copy link
Collaborator

jcsp commented Dec 4, 2023

No description provided.

@jcsp jcsp added t/feature Issue type: feature, for new features or requests c/storage/pageserver Component: storage: pageserver labels Dec 4, 2023
@jcsp jcsp mentioned this issue Dec 4, 2023
Closed
@jcsp jcsp self-assigned this Dec 4, 2023
@jcsp jcsp mentioned this issue Dec 4, 2023
Merged
@jcsp jcsp closed this as completed in #6024 Dec 8, 2023
jcsp added a commit that referenced this issue Dec 8, 2023
@jcsp @arpad-m
pageserver: filtered WAL ingest for sharding (#6024)
2c54434 
## Problem

Currently, if one creates many shards they will all ingest all the data:
not much use! We want them to ingest a proportional share of the data
each.

Closes: #6025

## Summary of changes

- WalIngest object gets a copy of the ShardIdentity for the Tenant it
was created by.
- While iterating the `blocks` part of a decoded record, blocks that do
not match the current shard are ignored, apart from on shard zero where
they are used to update relation sizes in `observe_decoded_block` (but
not stored).
- Before committing a `DataDirModificiation` from a WAL record, we check
if it's empty, and drop the record if so. This check is necessary
(rather than just looking at the `blocks` part) because certain record
types may modify blocks in non-obvious ways (e.g.
`ingest_heapam_record`).
- Add WAL ingest metrics to record the total received, total committed,
and total filtered out
- Behaviour for unsharded tenants is unchanged: they will continue to
ingest all blocks, and will take the fast path through `is_key_local`
that doesn't bother calculating any hashes.

After this change, shards store a subset of the tenant's total data, and
accurate relation sizes are only maintained on shard zero.

---------

Co-authored-by: Arpad Müller <arpad-m@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jcsp jcsp

Labels
c/storage/pageserver Component: storage: pageserver t/feature Issue type: feature, for new features or requests
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

pageserver: filtered WAL ingest for sharding neondatabase/neon
1 participant
@jcsp