Suppress wal lag timeout warnings right after tenant attachment

[arpad-m]

As seen in https://github.com/neondatabase/cloud/issues/17335, during releases we can have ingest lags that are above the limits for warnings. However, such lags are part of normal pageserver startup.

Therefore, calculate a certain cooldown timestamp until which we accept lags. The heuristic is chosen to grow the later we get to fully load the tenant, and we also add 60 seconds as a grace period after that term.

[github-actions]

5058 tests run: 4872 passed, 0 failed, 186 skipped (full report)

---

Flaky tests (5)

Postgres 17

• test_pg_regress[None]: debug-x86-64
• test_subscriber_restart: debug-x86-64

Postgres 15

• test_lr_with_slow_safekeeper: release-arm64
• test_restart_endpoint_after_switch_wal: release-arm64

Postgres 14

• test_storage_controller_onboarding[False]: release-x86-64

Code coverage* (full report)

• functions: 31.3% (7487 of 23883 functions)
• lines: 49.5% (60117 of 121349 lines)

* collected from Rust tests only

---

_{The comment gets automatically updated with the latest test results
1038b60 at 2024-10-03T01:19:13.068Z :recycle:}

[skyzh]

LGTM, but this patch seems to break backward compatibility?

[arpad-m]

this patch seems to break backward compatibility?

yeah the test expects the warning to be present but it is still within the 60 seconds grace period. Actually it makes sense for the warning to fire in that instance.

I think the heuristic needs to be adjusted. Instead of blanket-allowing all lag warnings until a certain timestamp, it might make more sense to, until a certain timestamp, allow lag warnings that are below a certain duration. I will adjust the PR according to that.