Expose operational telemetry and SLO gates for agent harness (unified harness hot-swapping between openai)

[haasonsaas]

Summary

Define the observable contract for latency, cost, correctness, and degraded-mode behavior.

This issue was generated from an org-wide EvalOps mining pass on 2026-05-10 07:57 UTC. It combines live GitHub repo signals with a per-repo arXiv search. Treat the research links as grounding for a concrete implementation, not as a request for a literature review.

Repo Evidence

• Repository description: Unified harness for hot-swapping between OpenAI Agents SDK and Anthropic Claude Agent SDK with shared tool registry
• Tree signals: 0 docs files, 1 workflows, 0 proto files, 5 test-like files.
• README.md:13 includes latent-spec language: ✅ Hot-Swapping - Switch providers at runtime without code changes ✅ Lazy Loading - SDKs imported only when needed ✅ Streaming Support - Real-time response streaming with consistent deltas
• README.md:158 includes latent-spec language: If your agents need live web results, register the built-in You.com adapter as a tool:
• tests/test_config.py:63 includes latent-spec language: # Zero retries should be valid config = HarnessConfig(retry_attempts=0)
• tests/test_registry.py:153 includes latent-spec language: # Should have 50 tools (5 threads * 10 tools each) assert len(registry.get_all()) == 50

Research Grounding

Repo axes: memory, evaluation, tooling, infra

Search keywords: str, openai, tool, harness, claude, def, providers, harnessconfig, provider, tools, async, config

• arXiv:2509.19209v1 A Knowledge Graph and a Tripartite Evaluation Framework Make Retrieval-Augmented Generation Scalable and Transparent (Olalekan K. Akindele, Bhupesh Kumar Mishra, Kenneth Y. Wertheim), 2025.
• arXiv:2603.20309v1 BubbleRAG: Evidence-Driven Retrieval-Augmented Generation for Black-Box Knowledge Graphs (Duyi Pan, Tianao Lou, Xin Li, Haoze Song, Yiwen Wu, Mengyi Deng), 2026.
• arXiv:2504.05163v2 Evaluating Knowledge Graph Based Retrieval Augmented Generation Methods under Knowledge Incompleteness (Dongzhuoran Zhou, Yuqicheng Zhu, Xiaxia Wang, Yuan He, Jiaoyan Chen, Steffen Staab), 2025.
• arXiv:2504.08893v1 Knowledge Graph-extended Retrieval Augmented Generation for Question Answering (Jasper Linders, Jakub M. Tomczak), 2025.
• arXiv:2510.14271v1 Less is More: Denoising Knowledge Graphs For Retrieval Augmented Generation (Yilun Zheng, Dan Yang, Jie Li, Lin Shang, Lihui Chen, Jiahao Xu), 2025.
• arXiv:2508.09460v1 Towards Self-cognitive Exploration: Metacognitive Knowledge Graph Retrieval Augmented Generation (Xujie Yuan, Shimin Di, Jielong Tang, Libin Zheng, Jian Yin), 2025.
• arXiv:2512.20626v2 MegaRAG: Multimodal Knowledge Graph-Based Retrieval Augmented Generation (Chi-Hsiang Hsiao, Yi-Cheng Wang, Tzung-Sheng Lin, Yi-Ren Yeh, Chu-Song Chen), 2025.
• arXiv:2502.01113v3 GFM-RAG: Graph Foundation Model for Retrieval Augmented Generation (Linhao Luo, Zicheng Zhao, Gholamreza Haffari, Dinh Phung, Chen Gong, Shirui Pan), 2025.
• arXiv:2502.06864v1 Knowledge Graph-Guided Retrieval Augmented Generation (Xiangrong Zhu, Yuexiang Xie, Yi Liu, Yaliang Li, Wei Hu), 2025.
• arXiv:2505.09945v1 Personalizing Large Language Models using Retrieval Augmented Generation and Knowledge Graph (Deeksha Prahlad, Chanhee Lee, Dongha Kim, Hokeun Kim), 2025.

What To Build

• Name the key service/user journey SLOs and their required dimensions.
• Emit metrics/log fields for success, failure, cost/latency, and reasoned fallback.
• Add a dashboard/runbook stub or CLI report that makes the new signals operator-visible.

Acceptance Criteria

• A short design note names the repo-specific workflow, threat or correctness model, and the research assumptions being adopted.
• A runnable check, fixture, or verifier exercises the new contract in CI or an equivalent local command documented in the repo.
• The implementation emits or stores enough evidence for a downstream agent/operator to cite inputs, decisions, and outputs.
• At least one negative/degraded-mode case is covered so failures are observable rather than silently accepted.
• Documentation links the new behavior to the relevant EvalOps platform primitive or explicitly records why this repo remains standalone.

Notes

• Generated issue 4/5 for evalops/agent-harness by evalops_org_miner.py.
• Before implementation, confirm the sampled latent-spec snippets still match main; this issue intentionally cites exact file paths/lines where the mining pass saw them.