KAFKA-8717: Reuse cached offset metadata when reading from log

[hachikuji]

Although we currently cache offset metadata for the high watermark and last stable offset, we don't use it when reading from the log. Instead we always look it up from the index. This patch pushes fetch isolation into Log.read so that we are able to reuse the cached offset metadata.

Committer Checklist (excluded from commit message)

• Verify design and implementation
• Verify test coverage and CI build status
• Verify documentation (including upgrade notes)

[ijuma]

Thanks for the PR. Concretely, does this mean we save one index look up in the usual consumer reading to the end of the log (hw or lso) case? Even though one would expect the index to be in the page cache, always good to do less work when possible.

[hachikuji]

@ijuma Yes, that's right. This came up in the context of KIP-392 because we realized that, unlike the leader, the follower would always need one lookup to fetch the high watermark or LSO. However, we saw that the leader was already doing an extra lookup whenever it read from the log, so we thought we could eliminate that lookup and bring the follower fetch at least up to parity with the leader's current behavior.

To be perfectly honest, I am skeptical of the benefit of this offset metadata caching. I did some brief consumer performance testing with this patch and saw basically no difference. But it's a bit harder to say in a more general context with more partitions and more consumers. In any case, I thought this presented a nice opportunity to simplify some of the internal fetch APIs a little bit.

[hachikuji]

retest this please

[hachikuji]

retest this please

[junrao]

@hachikuji : Thanks for the PR. Great cleanup patch. A few comments below. Also, since this PR is kind of tricky, it's probably useful to have a jira to track it.

[hachikuji]

retest this please

[junrao]

@hachikuji : Thanks for the PR. LGTM. Just a minor comment below.

[junrao]

less than => less than or equal to ?