tests: wait for uploads in test_secondary_downloads

[jcsp]

Problem

• test_secondary_downloads is flaky #6966

This test occasionally failed with some layers unexpectedly not present on the secondary pageserver. The issue in that failure is the attached pageserver uploading heatmaps that refer to not-yet-uploaded layers.

Summary of changes

After uploading heatmap, drain upload queue on attached pageserver, to guarantee that all the layers referenced in the haetmap are uploaded.

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

2718 tests run: 2580 passed, 0 failed, 138 skipped (full report)

---

Code coverage* (full report)

• functions: 28.1% (6280 of 22338 functions)
• lines: 46.9% (44179 of 94105 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
10c6e31 at 2024-03-25T09:59:24.316Z :recycle:}

[VladLazar]

The test change itself makes sense, but isn't this a property that we should enforce in the pageserver? From a quick look at the code it looks like it may happen during normal operation (i.e. not forcing the heatmap upload).

[jcsp]

The test change itself makes sense, but isn't this a property that we should enforce in the pageserver? From a quick look at the code it looks like it may happen during normal operation (i.e. not forcing the heatmap upload).

The download path is intentionally sloppy: it permits the heatmap to contain a layer that 404s on read (this is normal in cases like GC/compaction). If we have a layer enqueued for upload, and we must choose between omitting it (secondary gets a less complete set of layers) and including it (secondary will get a 404 and skip it), including it is generally nicer behavior.