refactor(LayerManager): resident layers query

[koivunej]

Refactor out layer accesses so that we can have easy access to resident layers, which are needed for number of cases instead of layers for eviction. Simplifies the heatmap building by only using Layers, not RemoteTimelineClient.

Cc: #5331

[github-actions]

2430 tests run: 2319 passed, 0 failed, 111 skipped (full report)

---

Flaky tests (3)

Postgres 16

• test_crafted_wal_end[last_wal_record_crossing_segment]: release
• test_vm_bit_clear_on_heap_lock: debug

Postgres 15

• test_pg_regress[4]: release

Code coverage (full report)

• functions: 54.9% (11866 of 21609 functions)
• lines: 82.1% (66232 of 80663 lines)

---

_{The comment gets automatically updated with the latest test results
c66553e at 2024-02-12T14:28:11.599Z :recycle:}

[koivunej]

I had not expected heatmap be using the same API we explicitly named *_for_disk_usage_eviction. Will fix a bit later.

[arpad-m]

If we contend for the lock, then won't we be missing out on evictions in a situation when there is high contention? In the worst case we'd not evict at all, right?

[koivunej]

If we contend for the lock, then won't we be missing out on evictions in a situation when there is high contention? In the worst case we'd not evict at all, right?

Correct for both. This is however for a single timeline. It is thought to be unlikely that we could have more than 8 timelines in critical sections, or at least I thought so before starting to write this, earlier had noted the following:

This solution is based on a hope that all timelines will not be busy at the same with layer flushing, compaction, applying garbage collection.

There is of course one more additional source for why the try_read would fail: ingestion. We would have to be hitting the lock very precisely when walreceiver is committing, page requests would similarly wait then.

This was not my idea, nor am I sure if this works. Seems however the unluckyness cases would have to be really unlucky. In the meantime, we should still evict from other tenants.

[arpad-m]

I'm still not really convinced that the new behaviour on contention --- us skipping eviction entirely --- is the right call. If there is no contention, there is no real difference either as in that instance the blocking wait just immediately yields the result.
But @koivunej has told me on slack that there is future work building on this, so I'm approving.

[koivunej]

Yes, it is unclear so I'll just drop the option returning for now.