engineccl: use incremental, records-based format for data key rotations #70140
Labels
A-storage
Relating to our storage engine (Pebble) on-disk storage.
C-enhancement
Solution expected to add code/behavior + preserve backward-compat (pg compat issues are exception)
T-storage
Storage Team
Projects
The CockroachDB encryption-at-rest implementation uses a file called the 'data keys registry' for storing the encryption keys used to encrypt a store's files. When the active data key is rotated, a new file is added to a protocol buffer and the active file is replaced with a new one containing the updated protocol buffer. As a store accumulates data keys, the cost of rotation grows. Rotation requires a O(# data keys) write to disk. Currently, the encryption-at-rest implementation never removes data keys from the data keys registry, which exacerbates the impact of this rewrite (see #70138).
We should modify the data keys registry to use Pebble's record package to write incremental updates to the data keys registry during rotations. Since 21.2 this is the approach that the file registry uses to record which files are encrypted with which data keys.
Jira issue: CRDB-9958
Epic CRDB-16419
The text was updated successfully, but these errors were encountered: