Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

kvserver: sep-raft-log: integrate snapshot application #93251

Open
tbg opened this issue Dec 8, 2022 · 2 comments
Open

kvserver: sep-raft-log: integrate snapshot application #93251

tbg opened this issue Dec 8, 2022 · 2 comments
Labels
C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-kv-replication KV Replication Team

Comments

@tbg
Copy link
Member

tbg commented Dec 8, 2022 •
edited by cockroach-jira-scripts

Snapshot application is a complex operation. A set of SSTables is constructed and atomically ingested. These SSTs both truncate the existing raft log (i.e. touch raft state) and replace the state machine state. Additionally, they might also atomically replicaGC additional initialized replicas (touching yet more raft and state machine state).

We lose this atomicity with the separate raft log.

https://github.com/tbg/cockroach/blob/e292a73f2af0e55aa760cce168622732c0716ff6/pkg/kv/kvserver/replicas_storage.go has some discussion of how this is handled, but we need to flesh it out and execute it.

Epic: CRDB-220

Jira issue: CRDB-22244
@tbg tbg added C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-kv-replication KV Replication Team labels Dec 8, 2022
@blathers-crl
Copy link

blathers-crl bot commented Dec 8, 2022

cc @cockroachdb/replication

@tbg tbg mentioned this issue Dec 19, 2022
Closed
tbg added a commit to tbg/cockroach that referenced this issue Dec 19, 2022
@tbg
logstore: introduce logstore.Engine
4240b28 
This introduces the concept of an Engine specific for use in a log
store. In other words, it's the beginnings of a logical separation of
the state machine and log engines, i.e. cockroachdb#16624.

For now, both continue to be backed by the same storage.Engine, and
LogEngine is not correctly used in all places. For example, snapshot
application hasn't yet been updated to account for the possibility
of two separate engines and writes a set of SSTS that is atomically
ingested into the single engine currently present but which logically
spans both engines (cockroachdb#93251).

Epic: CRDB-220
Release note: None
@tbg
Copy link
Member Author

tbg commented Dec 19, 2022

https://github.com/cockroachdb/cockroach/pull/91579/files#diff-57be82ecc91b85772e9d59c726c4cb5e044fd31931b94cc2c0e38f702b0006c3R564 has some useful code

@tbg tbg mentioned this issue Jan 6, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
C-enhancement Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception) T-kv-replication KV Replication Team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@tbg