feat(pageserver): generate basebackup from aux file v2 storage

[skyzh]

Problem

This pull request adds the new basebackup read path + aux file write path. In the regression test, all logical replication tests are run with matrix aux_file_v2=false/true.

Summary of changes

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

3042 tests run: 2915 passed, 0 failed, 127 skipped (full report)

---

Flaky tests (4)

Postgres 16

• test_vm_bit_clear_on_heap_lock: debug

Postgres 15

• test_partial_evict_tenant[relative_spare]: release

Postgres 14

• test_partial_evict_tenant[relative_equal]: release
• test_partial_evict_tenant[relative_spare]: release

Code coverage* (full report)

• functions: 31.2% (6252 of 20016 functions)
• lines: 46.7% (46832 of 100259 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
801550e at 2024-05-07T16:38:45.689Z :recycle:}

[skyzh]

All failed tests pass locally so I assume it's something wrong with vectored get env variable in CI. Fix in #7569.

[skyzh]

rebased to the new scan implementation pull request and added aux_file_v2 test matrix for regress tests.

[VladLazar]

The looks good to me, but I have questions about the testing.
Assuming a bug with aux files v2, I don't think there is a guarantee that postgress will generate an error. Have you considered validating the two aux files implementations against each other (at least in the tests)?

[skyzh]

Added a new cross-validation mode to aux file policy flag, waiting for CI runs.

[skyzh]

added 26efd47 to fix a bug on the code path of using vectored get for sequential get. It should throw a missing key error, but it doesn't.

Think about the case if a key does not exist. In the code path, we will create a ValueReconstructState of empty image and empty WAL, which then gets passed to reconstruct_value, and throwing "base image for {key} at {request_lsn} not found" PageReconstructError::Other error. This case is actually a missing key error.

Therefore, if the cached image is not found, we do not add it to the value reconstruct state to avoid calling into reconstruct_value.

It's unknown why this case only happens when v1+v2 are both enabled. Probably due to the difference of layer map in two cases, where in one case, the vectored get code path could catch the missing key error earlier than the point that reconstruct_value gets called. We only run cross validation instead of only running v2.

[VladLazar]

Looking good. Just a few nits.