Epic: improve unit test coverage in pageserver

[LizardWizzard]

Motivation

Mainly we have integration tests written in python. These tests are suitable for complex scenarios that involve many components, but fall short when you need to test on a module basis. Currently we dont have much unit tests. Adding new ones is quite hard for scenarios that involve playing with actual data (for example actual storage implementation).

Unit tests have a shorter feedback loop and can be run significantly faster than integration tests that involve many processes. They also facilitate better onboarding experience for new engineers since its way easier to understand whats going on in unit test compared to integration tests where a failure can be caused by any component and we have a lot of moving parts there.

There are some smoke tests for storage in https://github.com/neondatabase/neon/blob/main/pageserver/src/tenant.rs#L2913 but this is not enough. This doesnt excersize gc and compaction paths.

Other aspects like tenant relocation are only covered in integration tests. Tests for eviction and ondemand download require

DoD

All major components have their test suite that is found sufficient. Is it ok to measure by code coverage? Its not ideal, but can be helpful. See list of components and modules in task list.

Implementation ideas

Take areas one by one, analyze ideas for testing, apply needed refactorings for ease of testing, write tests.

For some cases tools like proptest can be useful.

Tasks

Beta Give feedback

1. Start splitting tenant.rs into manageable modules. Cover each one with tests. (I'll create a task for each module)
2. The same for timeline.rs
3. Revisit helpers to populate storage with some test data. It should be easy to just throw some data into layers.
4. pageserver: add more GC tests #3415
   a/tech_debt c/storage c/storage/pageserver +3
5. Cover compaction
6. Regresion test for Compaction causes full reimaging #2948
7. Cover pageserver startup activities (reconciling with remote storage, special file handling)
8. Cover management api
9. Cover pg backend common library
10. Branch downloads ancestors remote layer
11. unit-test disk-usage based eviction policy #3909
    c/storage c/storage/pageserver +2
12. Add a test for retryable delete operation, failpoint may not be the right way, it should also test pageserver management API doesn't poll request handlers to completion if client hangs up [P:1] [S:1] #3478, maybe use failpoint with sleep and really short http client timeout to trigger the disconnect

List to be continued, feel free to add areas that you feel are not tested enough

Other related tasks and Epics