Backlog Metric Attribution: part 2

[Shivs11]

What changed?

• This PR is the part 1 of this effort.
• PR Backlog metrics (count, age) to now have versioning based attribution #9300 added a dynamic config with the hope that simply making it a 0 would revert the backlog count (and age) metric to be emitted from the old location. However, I quickly realized that it was not doing that. This PR fixes that by first checking if that dynamic config is enabled or not. If it is enabled, we emit the newly created physical level backlog metrics. If it is not emitted, we emit the backlog metrics the way they used to before Backlog metrics (count, age) to now have versioning based attribution #9300 went in.
• Additionally, cursor bot had this comment pointing out that our use of Describe task queue to get the backlog metrics to emit would cause task queues to remain "alive" indefinitely and never be unloaded, potentially causing strain on Matching service. This PR fixes that by skipping the "mark alive" step when Describe is called internally for metrics use, and only marking it alive when it's called externally.
• See cdf/revert-9300...cdf/pick-9300-9316 to picture part 1 and part 2 as one diff, if that's helpful

Why?

• Safety and correctness.

How did you test it?

• built
• run locally and tested manually
• covered by existing tests
• added new unit test(s)
• added new functional test(s)

Potential risks

• these are risky, but i have a dynamic config flag now to quickly revert things in prod if things go wrong.

---

Note

Medium Risk
Changes affect Matching backlog metrics and task-queue liveness behavior; incorrect gating could break dashboards/alerts or impact queue unloading behavior under load.

Overview
Adjusts backlog gauge emission to conditionally switch between the new physical backlog metrics and the legacy approximate_backlog_* metrics based on BacklogMetricsEmitInterval: when attribution is enabled, only the unversioned queue emits physical_approximate_backlog_*; when disabled, all applicable queues fall back to emitting the original approximate_backlog_* series.

Refactors Describe into an internal describe(..., skipMarkAlive) variant and uses it for periodic logical backlog metric emission so internal metric collection no longer calls MarkAlive, avoiding unintended prevention of idle task queue unloading.

^{Written by Cursor Bugbot for commit e79f933. This will update automatically on new commits. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

[cursor]

Metric family switch leaves stale gauges

Medium Severity

emitZeroPhysicalBacklogGauges now clears only one gauge family based on BacklogMetricsEmitInterval(). If the config flips after metrics were emitted, the previously used family is never reset, so old backlog values remain published for unloaded queues. This leaves stale values in either physical_* or approximate_* metrics.

Additional Locations (1)

• service/matching/db.go#L781-L787

 

[carlydf]

I think this might be ok, because, when attribution is enabled, emitZeroPhysicalBacklogGauges zeroes the physical_* metric, and emitZeroLogicalBacklogForQueue zeroes the approximate_* metric.

When attribution is disabled, emitZeroPhysicalBacklogGauges zeroes the approximate_* metric, and emitZeroLogicalBacklogForQueue also zeroes the `approximate_* metric. That seems weird, but I think double-zeroing might be ok? It's only happening on unload...

anyway, ideally we aren't flip flopping between enable and disable much anyways..

[carlydf]

Most important is that if we need to revert the new change, we can, and after we revert, things get added to and zeroed out correctly. I think it's ok if the newly-added physical_* metrics become stale as a result, in the worst case scenario