pageserver: fix vectored read aux key handling

[VladLazar]

Problem

Vectored get would descend into ancestor timelines for aux files.
This is not the behaviour of the legacy read path and blocks cutting
over to the vectored read path.

Fixes #7379

Summary of Changes

Treat non inherited keys specially in vectored get. At the point when
we want to descend into the ancestor mark all pending non inherited keys
as errored out at the key level. Note that this diverges from the
standard vectored get behaviour for missing keys which is a top level
error. This divergence is required to avoid blocking compaction in case
such an error is encountered when compaction aux files keys. I'm pretty
sure the bug I just described predates the vectored get implementation,
but it's still worth fixing.

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

2772 tests run: 2654 passed, 0 failed, 118 skipped (full report)

---

Code coverage* (full report)

• functions: 28.1% (6467 of 23043 functions)
• lines: 46.9% (45795 of 97715 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
8835086 at 2024-04-23T11:19:05.334Z :recycle:}

[skyzh]

my main concern of this pull request is that we also have a is_inherited_key function, which serves almost the same purpose of NON_INHERITED_RANGE. I'm wondering if there could be some hints like "if you change is_inherited_key, also change NON_INHERITED_RANGE", or have a single place to define both things, so as to avoid problems in the future.

[VladLazar]

my main concern of this pull request is that we also have a is_inherited_key function, which serves almost the same purpose of NON_INHERITED_RANGE. I'm wondering if there could be some hints like "if you change is_inherited_key, also change NON_INHERITED_RANGE", or have a single place to define both things, so as to avoid problems in the future.

Yeah that's true. Let's define is_inherited_key in terms of NON_INHERITED_RANGE: !NON_INHERITED_RANGE.contains(&key).

[skyzh]

Rest LGTM

[arpad-m]

We can merge this in my opinion, but I don't think it being a range helps us a lot: we'll need support for a list of ranges in the future instead of just one. That being said, refactoring from this state should not be harder than refactoring from a simpler version of this PR that bakes in the assumption that NON_INHERITED_RANGE is just one element large.