dube: timeout individual layer evictions, log progress and record metrics

[koivunej]

Because of bugs evictions could hang and pause disk usage eviction task. One such bug is known and fixed #6928. Guard each layer eviction with a modest timeout deeming timeouted evictions as failures, to be conservative.

In addition, add logging and metrics recording on each eviction iteration:

• log collection completed with duration and amount of layers
  • per tenant collection time is observed in a new histogram
  • per tenant layer count is observed in a new histogram
• record metric for collected, selected and evicted layer counts
• log if eviction takes more than 10s
• log eviction completion with eviction duration

Additionally remove dead code for which no dead code warnings appeared in earlier PR.

Follow-up to: #6060.

[github-actions]

2448 tests run: 2326 passed, 0 failed, 122 skipped (full report)

---

Flaky tests (1)

Postgres 14

• test_pageserver_chaos[None]: release

Code coverage* (full report)

• functions: 28.8% (6933 of 24085 functions)
• lines: 47.4% (42570 of 89824 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
d97f1e1 at 2024-02-29T20:29:58.767Z :recycle:}

[koivunej]

Should do the same for eviction_task.

[problame]

While convincing myself of the cancel-safety of evict_layers, I noticed that it leaks the JoinSet js on cancellation.
Well, it does join_handle.abort() on each task.

So, I think secondary_tenant.evict_layer() has probably not been audited for cancellation safety.
I don't know the invariants, but, it doesn't look cancel safe (could be cancelled by the runtime when returning Poll::Ready from remove_file). It's a pre-existing issue though, and secondaries aren't prod-enabled yet, so, let's punt that to a new issue. At a mininum, the the SecondaryTenant::evict_layer function needs a comment stating it promises to be cancel-safe.

[koivunej]

So, I think secondary_tenant.evict_layer() has probably not been audited for cancellation safety.

Excellent point. This PR was done before secondary tenants or at least their eviction, and I did not revisit it while fixing conflicts.

[problame]

Love the smoke test, great work!

I like the added metrics, not just their names.

Not sure why you went for a timeout parameter instead of making callers who care use tokio::time::timeout, but 🤷

[koivunej]

Not sure why you went for a timeout parameter instead of making callers who care use tokio::time::timeout, but 🤷

I was thinking it's the mandatory nature makes sense here. Waiting is a nice to have, it's not required except for our tests. Timeout does not cancel it.

[koivunej]

Rebase needed for build-image tools.

[koivunej]

Rebase needed (?) for test_encode #6963.