bench: run branch_creation_many at 500, seeded

[koivunej]

We have a benchmark for creating a lot of branches, but it does random things, and the branch count is not what we is the largest maximum we aim to support. If this PR would stabilize the benchmark total duration it means that there are some structures which are very much slower than others. Then we should add a seed-outputting variant to help find and reproduce such cases.

Additionally, record for the benchmark:

• shutdown duration
• startup metrics once done (on restart)
• duration of first compaction completion via debug logging

[VladLazar]

Looks good. I'd suggest adding a temporary pytest.mark.repeat() on the test to check for flakiness before merging.

[github-actions]

2570 tests run: 2437 passed, 0 failed, 133 skipped (full report)

---

Flaky tests (2)

Postgres 16

• test_compute_pageserver_connection_stress: release
• test_vm_bit_clear_on_heap_lock: debug

Code coverage* (full report)

• functions: 28.8% (6992 of 24266 functions)
• lines: 47.4% (43008 of 90731 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
8e2913d at 2024-03-07T13:40:10.407Z :recycle:}

[koivunej]

Looks good. I'd suggest adding a temporary pytest.mark.repeat() on the test to check for flakiness before merging.

You mean the seeding could make it flaky or what? Note we don't even run these without run-benchmarks label, which I was not thinking this would require.

[koivunej]

Drafted to avoid the e2e while waiting the new token.

[koivunej]

Well, this PR took long enough to get going, might as well continue with the added metrics.

[koivunej]

Example run output for me:

test_branch_creation_many[release-pg15-one_ancestor-500].branch_creation_duration_max: 0.042 s
test_branch_creation_many[release-pg15-one_ancestor-500].branch_creation_duration_avg: 0.030 s
test_branch_creation_many[release-pg15-one_ancestor-500].branch_creation_duration_stdev: 0.002 s
test_branch_creation_many[release-pg15-one_ancestor-500].shutdown: 0.713 s
test_branch_creation_many[release-pg15-one_ancestor-500].restart_after.background_jobs_can_start: 1.025 s
test_branch_creation_many[release-pg15-one_ancestor-500].restart_after.complete: 1.025 s
test_branch_creation_many[release-pg15-one_ancestor-500].restart_after.initial: 0.011 s
test_branch_creation_many[release-pg15-one_ancestor-500].restart_after.initial_tenant_load: 1.025 s
test_branch_creation_many[release-pg15-one_ancestor-500].restart_after.initial_tenant_load_remote: 1.025 s
test_branch_creation_many[release-pg15-one_ancestor-500].compaction: 5.689 s