Runtime Registry, Reasoning Support & Arena Stability

Runtime Registry, Reasoning Support & Arena Stability

Highlights

• Runtime-Scoped Registry Modernization: Introduced runtime-scoped registry preparation and discovery planning so datasets, metrics, prompts, arena assets, and backend dependencies resolve more predictably from the active pipeline context.
• Execution Pipeline Hardening: Reworked pipeline building, sample ingress, execution control, runtime metadata, and support artifact handling to make large evaluation runs more diagnosable and resilient.
• Backend Reasoning & Compatibility Upgrades: Added thinking-mode handling and Qwen3 validation, improved vLLM multimodal compatibility, and tightened OpenAI HTTP, LiteLLM, and generic backend error contracts.
• Arena Reliability Improvements: Hardened websocketRGB shutdown, replay paging, scheduler diagnostics, human-input routing, and renderer resolution across interactive arena flows.
• Broader Validation Coverage: Consolidated the test suite layout and added substantial regression coverage across runtime, registry, observability, backends, arena behavior, and sandbox lifecycle handling.

Runtime & Registry

• Added runtime-scoped registry lifecycle, manifest-backed discovery planning, and overlay preparation so pipeline assets resolve from the active runtime view instead of global import assumptions.
• Introduced staged pipeline building, execution-controller primitives, sample ingress validation, support-artifact normalization, runtime metadata, and step-factory wiring for clearer pipeline contracts.
• Preserved sample transport fields and inference-only artifacts more consistently during execution, improving downstream reporting and step-to-step data integrity.
• Reorganized many custom configs into topic-oriented directories and refreshed framework documentation to point at the new entrypoints and examples.
• Removed the legacy Support CLI surface from docs and source in favor of a cleaner config-first workflow for dataset onboarding and validation.

Backend Compatibility & Reasoning

• Added thinking-mode support for model responses, including Qwen3-focused validation paths and utilities for extracting or stripping reasoning content.
• Tightened vLLM behavior with prompt-token normalization, multimodal renderer compatibility patches, explicit multimodal limits, and safer shutdown ordering.
• Improved backend robustness by restoring HTTP concurrency, deduplicating retry behavior, aligning OpenAI-style response contracts, and normalizing backend and judge error payloads.
• Hardened LiteLLM request isolation and log redaction, and added a new DeepSeek-based HLE judge metric with tighter provider routing behavior.
• Kept single-worker sample execution on the calling thread where required, reducing threading-related runtime surprises.

Arena Stability & Replay

• Improved replay handling with streamed and paged replay events for more reliable inspection of longer or noisier arena sessions.
• Isolated human input by sample route and added shared runtime service management so concurrent arena sessions bind action servers, visualizers, and websocketRGB hubs more safely.
• Hardened scheduler error handling, boolean parsing, gradio diagnostics, backend probe behavior, and websocketRGB shutdown paths.
• Expanded arena-side prompt and runtime support for VizDoom, including S4 image-history prompting and safer runtime renderer resolution.
• Continued cleanup across Mahjong, PettingZoo-related execution paths, and arena adapter/runtime coverage to reduce interactive flow regressions.

Observability & Validation

• Strengthened observability with healthier log-sink routing, recorder semantics, trace resilience, compact run identity handling, and better failure summaries.
• Improved metric reliability by deferring optional aggregator imports, honoring registry defaults, restoring choice-pattern overrides, and guarding parser edge cases such as MathVista code-mode handling.
• Added persistent sandbox lease tracking, stale-sandbox startup cleanup, safer pool behavior, and session-routing support for managed runtimes.
• Consolidated and migrated the test suite into clearer unit, integration, and e2e layouts, with broader regression coverage for registry runtime scope, role pools, backends, arena services, observability, and sandbox cleanup.

Notes

This release is a broad platform update focused on runtime architecture hardening, registry modernization, backend compatibility, arena stability, and regression-proofing rather than a single feature headline. The biggest user-visible changes are cleaner config organization, more reliable backend and arena execution, and stronger observability around complex runs.
This is a substantially broader release centered on runtime-scoped asset resolution, execution-path cleanup, thinking-mode and vLLM compatibility work, arena replay and human-input hardening, and a major expansion of regression coverage.