feat(pageserver): add metrics for aux file size

[skyzh]

Problem

ref #7443

Summary of changes

This pull request adds a size estimator for aux files. Each timeline stores a cached isize for the estimated total size of aux files. It gets reset on basebackup, and gets updated for each aux file modification. TODO: print a warning when it exceeds the size.

The size metrics is not accurate. Race between on_basebackup and other functions could create a negative basebackup size, but the chance is rare. Anyways, this does not impose any extra I/Os to the storage as everything is computed in-memory.

The aux files are only stored on shard 0. As basebackups are only generated on shard 0, only shard 0 will report this metrics.

Checklist before requesting a review

• I have performed a self-review of my code.
• If it is a core feature, I have added thorough tests.
• Do we need to implement analytics? if so did you add the relevant metrics to the dashboard?
• If this PR requires public announcement, mark it with /release-notes label and add several sentences in this section.

Checklist before merging

• Do not forget to reformat commit message to not include the above checklist

[github-actions]

3048 tests run: 2915 passed, 0 failed, 133 skipped (full report)

---

Flaky tests (3)

Postgres 16

• test_lock_time_tracing: release

Postgres 15

• test_partial_evict_tenant[relative_equal]: release
• test_download_remote_layers_api: release

Code coverage* (full report)

• functions: 31.4% (6321 of 20137 functions)
• lines: 47.3% (47634 of 100778 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
24ae9b2 at 2024-05-13T15:41:49.881Z :recycle:}

[arpad-m]

What happens on a pageserver restart? I don't see any persistence of this value, so it's lost whenever we restart the pageserver, until the next basebackup happens.

We should add persistence somehow.

We already have a non-persisted value via #6736 but it's only of limited use as we regularly restart pageservers.

[skyzh]

I don’t plan to persist it because it’s an estimator. If we configure an alert for aux file size and some tenant exceeds the limit, it will fire before the page server restart so it does not matter whether it’s persisted or not?

[skyzh]

... or we can have logical size calculation to recompute aux file size in later patches

[arpad-m]

The huge aux file counts we have seen in the past were not always from a few days but often the results of buildups over weeks, or maybe even months. So accurate values are kinda important.

Again, as I've said, we already have a metric that only runs during the lifetime of the process. It's better than nothing but still not what we want.

If we computed it during logical size calculation, it would be equivalent to preparing the basebackup, so kinda expensive. Ideally, reading the prior size would be an O(1) operation. In other words, it should be materialized somewhere.

[skyzh]

Then I'd like to wait on #7663 and persist it in timeline metadata / index_part.json

[skyzh]

Per discussion, we can merge this pull request first and we will have at least some metrics, and add this to initial logical size calculation in the future.