fix(libsy): in-flight algo count metric

[messiaen]

What

Adds switchyard_algorithms_in_flight{algorithm} gauge.

Why

Helps monitor load and stuck algorithms.

Closes

SWITCH-1238

Summary by CodeRabbit

• Observability

  • Added tracking for algorithms currently in flight, including runs suspended during unanswered routing calls.
  • In-flight counts now return to zero when runs resolve or are abandoned, including failure and early-exit scenarios.

• Documentation

  • Documented the new in-flight algorithms metric and its cardinality.
  • Added troubleshooting guidance for runs that remain in flight.

[github-actions]

PR Preview Action v1.8.1
🚀 View preview at https://NVIDIA-NeMo.github.io/Switchyard/pr-preview/pr-576/
Built to branch gh-pages at 2026-08-31 17:31 UTC. Preview will be ready when the GitHub Pages deployment is complete.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 9067eca8-3153-4fbd-9e38-ae5a0b1d5fe5

📥 Commits

Reviewing files that changed from the base of the PR and between 4fb3251 and 8a1ffff.

📒 Files selected for processing (3)

• crates/libsy-llm-client/tests/observability.rs
• crates/libsy/src/observability.rs
• docs/internal/metrics_reference.md

Included review availability: Your plan provides up to 12 included reviews per hour; 11 remain after this review.

---

Walkthrough

The change adds an RAII guard for algorithm in-flight metrics. Tests cover unresolved routing calls, completed runs, and abandoned runs. The metrics reference documents the new gauge and triage guidance.

Changes

Algorithm observability

Layer / File(s)	Summary
Track active algorithm runs crates/libsy/src/observability.rs	observe_run creates an InFlightRun guard. The guard increments switchyard.algorithms_in_flight on creation and decrements it on drop.
Validate and document the metric crates/libsy-llm-client/tests/observability.rs, docs/internal/metrics_reference.md	Tests verify metric values for unresolved, completed, and abandoned runs. Documentation describes the gauge, its label, and related triage guidance.

Estimated code review effort: 3 (Moderate) | ~20 minutes

Merge Risk: ⚪ Minimal · up to 8a1ff

This PR adds an in-flight algorithm gauge and documentation/tests without any identified merge-blocking risk; it is merge-ready after normal checks and review.

Poem

A rabbit counts each running flight
One hop up, then back to right
Unanswered paths remain in view
Resolved runs return to zero too
Dropped streams clear the trail
Metrics tell the whole tale

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	Docstring coverage is 81.82% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 11 functions across 2 files. (1 skipped: 1 …
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly identifies the primary change: fixing the in-flight algorithm count metric. It is concise and related to the pull request changes.

Full details: Docstring Coverage

Explanation

Docstring coverage is 81.82% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 11 functions across 2 files. (1 skipped: 1 unsupported.)

---

_{Comment @coderabbitai help to get the list of available commands.}