Harden chat fallback observability and acceptance coverage (Vibe Kanban)

[jbax1899]

Summary

This PR hardens the VoltAgent-backed primary text path by improving per-request fallback observability and adding acceptance coverage for normal and degraded orchestration flows.

What Changed

• Extended chat.orchestration.timing logging in packages/backend/src/services/chatOrchestrator.ts with explicit operator-facing fields:
  • Tool context: toolName, toolReasonCode, toolRequestReasonCode
  • Fallback context: fallbackReasons
  • User-facing outcome context: responseId, responseAction, responseModality, responseProvenance, responseRiskTier, responseModelVersion, responseCitationCount, responseMessageLength
• Added acceptance tests in packages/backend/test/chatOrchestrator.test.ts to lock behavior:
  • Normal message flow emits response summary fields in orchestration timing logs.
  • Degraded search-drop flow emits stable fallback reason codes and fallback reasons for triage.

Why These Changes Were Made

The pilot goal is to validate readiness of the existing VoltAgent primary text path without rebuilding infrastructure. The key remaining hardening gap was fast operator triage: being able to answer, from one place, what happened, why fallback occurred, and what the user received.

These updates make fail-open decisions explicit and reviewable at request granularity, while preserving backend ownership of provenance/trace semantics and runtime accountability.

Important Implementation Details

• No runtime seam redesign was introduced; this is a delta hardening pass on the existing backend orchestration path.
• Fallback/reroute reason-code semantics are preserved and surfaced directly in orchestration timing payloads.
• Added tests validate both steady-state and degraded behavior so the pilot can use stable reason-code visibility as an acceptance signal.
• Scope remains intentionally narrow: backend orchestration observability and acceptance coverage only.

This PR was written using Vibe Kanban

Summary by CodeRabbit

• Chores

  • Expanded telemetry to capture richer tool execution details (name, status, reason codes, eligibility, request reason), fallback indicators and reasons, and extended response metadata (id, action, modality, provenance, risk tier, model version, citation count, message length) for improved monitoring and diagnostics.

• Tests

  • Added tests validating telemetry payloads for normal message flows and scenarios where search is skipped, including fallback behavior and detailed response fields.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: deb3ea3d-77f3-4ca8-80a1-621a397e8045

📥 Commits

Reviewing files that changed from the base of the PR and between 0c5422c and 4301b98.

📒 Files selected for processing (3)

• packages/backend/src/services/chatOrchestrator.ts
• packages/backend/src/services/chatService.ts
• packages/backend/test/chatOrchestrator.test.ts

✅ Files skipped from review due to trivial changes (1)

• packages/backend/test/chatOrchestrator.test.ts

---

📝 Walkthrough

Walkthrough

Adds passing of toolRequest into chat execution, surfaces final tool execution telemetry from runChatMessages, and expands orchestration telemetry with tool and response metadata; test coverage added to assert the new telemetry payloads.

Changes

Cohort / File(s)	Summary
Orchestration telemetry packages/backend/src/services/chatOrchestrator.ts	Expanded telemetry payload: replaced prior tool fields with toolName, toolStatus, toolReasonCode, toolRequestReasonCode, toolEligible, added fallbackReasons, fallbackApplied logic update, and multiple response metadata fields (responseId, responseAction, responseModality, responseProvenance, responseRiskTier, responseModelVersion, responseCitationCount, responseMessageLength). Minor formatting adjustments; control flow for non-message actions unchanged.
Chat service API / types packages/backend/src/services/chatService.ts	Added toolRequest?: ToolInvocationRequest to RunChatMessagesInput, introduced FinalToolExecutionTelemetry type, computed and returned finalToolExecutionTelemetry when available (augmenting from effective tool execution context and toolRequest reason/eligibility).
Tests packages/backend/test/chatOrchestrator.test.ts	Added two tests stubbing logger.info to assert chat.orchestration.timing payload: normal planner→message path checks response metadata and no fallback; a search-skipped scenario flips modelProfiles to disallow search and asserts tool skipped reason codes and fallbackReasons. Restores globals in finally blocks.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related PRs

• feat: planner-aware profile routing, execution timeline metadata, and Discord /chat model switching (with Ollama support) #200 — Modifies backend orchestration and runtime telemetry surfaces (closely related changes to chatOrchestrator and chatService telemetry).
• Deterministic policy-ranked tool fallback routing with auditable reason codes (Vibe Kanban) #216 — Touches tool-execution handling and telemetry (reason codes and fallback behavior) that intersect this PR’s telemetry fields.
• Standardize tool invocation contracts and fail-open tooling metadata (Vibe Kanban) #215 — Adds/updates ToolInvocationRequest and tool execution telemetry surfaces used when passing toolRequest into chat execution.

Poem

🐰
Hopping through logs with a curious nose,
I stash tool names where the telemetry grows.
Citations counted, reasons neatly penned,
Fallbacks and lengths in every log I tend.
Tests clap their paws — this hop’s a win, my friend!

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'Harden chat fallback observability and acceptance coverage' accurately summarizes the main changes: expanded observability fields in orchestration logs and new acceptance tests for fallback scenarios.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch vk/55db-voltagent-pilot

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}