A consensus-check tool for Agno Teams (flag when member agents agree for the wrong reasons)

[screwtapeai]

In an Agno Team the leader often combines answers from several members. When members agree, the leader tends to trust it, even if they all leaned on the same weak source. Here's a small custom tool the leader can call before it commits to a combined answer:

import os, requests

def consensus_check(task: str, outputs: dict[str, str]) -> dict:
    """outputs: {agent_name: answer}. Needs 2 to 16 answers."""
    r = requests.post(
        "https://screwtape.ai/v1/consensus",
        headers={"Authorization": f"Bearer {os.environ['SCREWTAPE_API_KEY']}"},
        json={"task": task,
              "answers": [{"agent_id": k, "output": v} for k, v in outputs.items()]},
        timeout=30,
    )
    r.raise_for_status()
    return r.json()  # agreement_score, false_consensus_risk (low|medium|high), clusters, reasons, receipt

def check_member_agreement(task: str, member_answers: dict[str, str]) -> str:
    """Score whether the team members' agreement is trustworthy.
    member_answers maps member name -> that member's answer."""
    res = consensus_check(task, member_answers)
    return f"risk={res['false_consensus_risk']} score={res['agreement_score']} reasons={res['reasons']}"

Then add it to the team, e.g. Team(members=[...], tools=[check_member_agreement], instructions=["Before finalizing, call check_member_agreement with each member's answer; if risk is high, ask a member to re-verify against the sources."]).

For single-agent drafts, POST /v1/judge returns pass, fail or retry with reasons you can feed back to the agent. It's also available as an MCP server if you prefer MCPTools.

Full runnable example: https://github.com/screwtapeai/screwtape-mcp-clients

Disclosure: I build Screwtape.ai. An independent judge for your agents. Screwtape isn't tied to any agent framework or AI model provider, so its verdicts are never grading their own work. Every verdict comes with a signed receipt you can verify yourself, which gives you a record you can trust when it matters.

• Try it without signing up (3 calls): https://screwtape.ai/try/
• MCP server (streamable HTTP; tools include judge and consensus): https://screwtape.ai/mcp
• Free key (100 calls/day): https://screwtape.ai/signup/. To check a receipt, POST the raw receipt object to https://screwtape.ai/v1/receipts/verify.

We're looking for a few design partners who run multi-agent teams in production: 1,000 free calls a day for 60 days, plus direct input on the roadmap. Mostly I'd like to hear where this pattern breaks for you. Reply here or email screwtapedev@gmail.com.