storage: add log_reader option to translate offsets #17384
Conversation
|
new failures in https://buildkite.com/redpanda/redpanda/builds/46771#018e79be-3214-40fa-b530-9da74449b1e6: new failures in https://buildkite.com/redpanda/redpanda/builds/46771#018e79be-3211-4e52-ad63-917d85052941: new failures in https://buildkite.com/redpanda/redpanda/builds/46771#018e79be-320e-4313-8d42-adfeacb48e14: new failures in https://buildkite.com/redpanda/redpanda/builds/46771#018e79be-3217-41d6-ac3b-2a4429ae34c5: new failures in https://buildkite.com/redpanda/redpanda/builds/46771#018e79d0-002c-4cec-be97-2d1befcdef74: new failures in https://buildkite.com/redpanda/redpanda/builds/46771#018e79d0-0024-4d45-8afa-7ac0caa312ae: new failures in https://buildkite.com/redpanda/redpanda/builds/46771#018e79d0-0027-4768-9931-04d2a3feb466: new failures in https://buildkite.com/redpanda/redpanda/builds/46771#018e79d0-0029-4b1f-8b55-7018f42f1f43: |
|
ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/46771#018e79be-320e-4313-8d42-adfeacb48e14 ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/46771#018e79be-3211-4e52-ad63-917d85052941 ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/46771#018e79d0-0024-4d45-8afa-7ac0caa312ae ducktape was retried in https://buildkite.com/redpanda/redpanda/builds/46955#018e834a-d06c-4a6e-a0f1-c7e1c1746300 |
andrwng
force-pushed
the
storage-log-reader-translation
branch
from
1e1fa1f
to
30b77e9
Compare
andrwng
requested review from
dotnwat,
mmaslankaprv and
ztlpn
and removed request for
dotnwat
Previously offset translation on the read path was done in the Kafka layer, with the storage layer providing primitives to help translate. As we clarify the requirements of the storage log implementation, offset translation has made its way into the purview of storage. To allow the storage layer to own offset translation end-to-end on the read path, and allow for future log implementations that translate offsets differently (e.g. by storing translated offsets), this adds an option to the log reader to translate. The corresponding code is removed from the Kafka layer.
andrwng
force-pushed
the
storage-log-reader-translation
branch
from
30b77e9
to
b4f54b1
Compare
| if (_config.translate_offsets) { | ||
| vassert( | ||
| _translator, "Expected offset translactor to be initialized"); | ||
| for (auto& b : batches) { | ||
| b.header().base_offset = _translator->from_log_offset( | ||
| b.base_offset()); | ||
| } | ||
| } |
There was a problem hiding this comment.
nice. i wonder if at some point this could become cheap enough that'd we could always translate the offset into a new b.translated_offset()
There was a problem hiding this comment.
Yeah in the future once getting the translated offset is a matter of reading deserializing the metadata from an entry, it seems plausible that that offset gets put into the ctx as well, along with the term.
Previously offset translation on the read path was done in the Kafka layer, with the storage layer providing primitives to help translate. As we clarify the requirements of the storage log implementation, offset translation has made its way into the purview of storage.
To allow the storage layer to own offset translation end-to-end on the read path, and allow for future log implementations that translate offsets differently (e.g. by storing translated offsets), this adds an option to the log reader to translate. The corresponding code is removed from the Kafka layer.
Backports Required
Release Notes