Idea: static reliability analysis for the agent-orchestration layer

[surajkumar811]

Floating this to gauge interest before anything formal.

Odysseus runs real agent orchestration under agent mode + sub-agents + MCP tools. As that layer grows, some failures aren't single-component bugs — they're structural: one agent's failure cascading downstream, loops with no exit condition burning tokens, untrusted tool output reaching an agent with sensitive access. They're hard to catch by testing components in isolation.

I build swarm-test — open-source static analysis on agent topologies (single points of failure, cascade paths, unbounded loops, injection-convergence points). No live LLM calls, runs at design time. Could be an optional check on agent configs, not a core dependency.

Came here via a thread with Divyesh on DEV, who suggested raising it. Mostly want to know: is structural reliability of the agent layer something the team sees as worth a design-time check? Happy to prototype an adapter if there's appetite.

github.com/surajkumar811/swarm-test

[alteixeira20]

This is an interesting direction, especially as Odysseus grows more agent/tool orchestration paths.

I would be cautious about adding a new reliability-analysis dependency or framework to core before it proves value on a real Odysseus flow, but I do think an optional design-time check could be useful if it stays external/prototype-first.

The most useful next step, in my opinion, would be a very small adapter proof of concept:

1. Pick one real Odysseus agent path or config shape.
2. Convert it into whatever topology representation swarm-test expects.
3. Run the static checks without live LLM calls.
4. Show the output on concrete risks: unbounded loops, cascade paths, single points of failure, or untrusted tool-output paths.
5. Document what the tool caught, what it missed, and whether the result is actionable for maintainers.

That would make the discussion much easier to evaluate. The key question is not only “can this analyze agent graphs?”, but whether it can produce findings that are specific and useful enough to guide Odysseus development without creating noise or a new maintenance burden.

So I would be interested in seeing this as an external/optional prototype first. If it can catch one or two real structural risks in the current orchestration layer, then it becomes much easier to discuss whether it should become a dev tool, CI-adjacent check, or stay as a separate audit helper.

[surajkumar811]

This is exactly the right framing, thanks for taking the time. Agreed on all of it — external/prototype-first, prove value on a real flow before anything touches core, and the bar is "specific, actionable findings, not noise." That's the right bar.

I'll do exactly what you outlined: take one real Odysseus agent path, convert it to swarm-test's topology model, run the static checks (no live LLM calls), and report back transparently — what it caught, what it missed, and whether the findings are actually actionable for a maintainer vs. just noise. If it can't surface at least one or two real structural risks in the current orchestration layer, that's a useful negative result too.

Give me a bit to put together the PoC against a real path and I'll follow up here with the concrete output. Appreciate the clear direction.

[surajkumar811]

Thanks for the clear bar — "specific and useful, not noise" is the right test, so I held the PoC to it. I traced agent_loop.py and I'm going to give you the honest result rather than a manufactured one.
The finding that matters: swarm-test isn't a fit for Odysseus's agent layer today. stream_agent_loop is a single-agent, multi-round tool loop — one model emitting tool calls, results fed back, repeat to MAX_AGENT_ROUNDS. swarm-test models multi-agent topologies (agents as nodes, hand-offs as edges) and checks for cross-agent cascade, collusion, and inter-agent context leakage. None of those classes exist in a single-agent loop, so running those checks here would produce exactly the noise you're worried about. I'm not going to invent a topology that isn't in the code.
A few things I'll note from the trace, in case they're useful regardless:

1. The one genuine structural edge is _run_verifier_subagent — it reuses the same endpoint_url/model as the primary agent. So the independent completion check shares a failure mode with the thing it's checking: if that endpoint degrades, the agent and its verifier fail together. Minor, and arguably acceptable, but it's the single "shared dependency" point a static pass would flag.

2. The loop already handles the runtime failure classes a static checker would care about — _detect_runaway_call, the _stuck_rounds → _force_answer convergence, the round cap, and tool output wrapped via untrusted_context_message (trusted=False) per your THREAT_MODEL. That's good defense-in-depth and it's why a static loop-detector wouldn't add much here.

So my honest recommendation is not to adopt swarm-test for Odysseus as it stands — there's no real surface for it. Where it'd become relevant is if the agent layer grows into actual multi-agent orchestration (parallel specialized agents handing off work, sub-agent fan-out beyond the single verifier). At that point the cascade/SPOF/context-leakage checks start mapping to real structure, and I'd be glad to revisit with a concrete adapter.
Appreciate you taking the proposal seriously enough to set a real bar — that's rarer than it should be.