Scoring agreement in concurrent / group-chat orchestrations with an external judge (MCP)

[screwtapeai]

Concurrent and group-chat orchestrations give you several agent answers to the same task. A useful last step is to check whether their agreement is real (not just shared context or one agent priming the rest) before you return a result.

The judge is a remote MCP server, so in Python it plugs in with MCPStreamableHTTPTool:

import os
from agent_framework import MCPStreamableHTTPTool

screwtape = MCPStreamableHTTPTool(
    name="screwtape",
    url="https://screwtape.ai/mcp",
    static_headers={"Authorization": f"Bearer {os.environ['SCREWTAPE_API_KEY']}"},
)
# give `screwtape` to the agent that aggregates results; the relevant tools:
#   consensus(task, answers[2..16])           -> agreement_score, false_consensus_risk (low|medium|high)
#   judge(task, agent_output, evidence?)      -> verdict pass|fail|retry, score, reasons

Or call it directly on the collected outputs of a concurrent run:

import os, requests

def consensus_check(task: str, outputs: dict[str, str]) -> dict:
    """outputs: {agent_name: answer}. Needs 2 to 16 answers."""
    r = requests.post(
        "https://screwtape.ai/v1/consensus",
        headers={"Authorization": f"Bearer {os.environ['SCREWTAPE_API_KEY']}"},
        json={"task": task,
              "answers": [{"agent_id": k, "output": v} for k, v in outputs.items()]},
        timeout=30,
    )
    r.raise_for_status()
    return r.json()  # agreement_score, false_consensus_risk (low|medium|high), clusters, reasons, receipt

If false_consensus_risk comes back high, a reasonable policy is to route to a fresh agent with the original evidence rather than accept the majority.

Full runnable example: https://github.com/screwtapeai/screwtape-mcp-clients

Disclosure: I build Screwtape.ai. An independent judge for your agents. Screwtape isn't tied to any agent framework or AI model provider, so its verdicts are never grading their own work. Every verdict comes with a signed receipt you can verify yourself, which gives you a record you can trust when it matters.

• Try it without signing up (3 calls): https://screwtape.ai/try/
• MCP server (streamable HTTP; tools include judge and consensus): https://screwtape.ai/mcp
• Free key (100 calls/day): https://screwtape.ai/signup/. To check a receipt, POST the raw receipt object to https://screwtape.ai/v1/receipts/verify.

We're looking for a few design partners who run multi-agent orchestrations in production: 1,000 free calls a day for 60 days, plus direct input on the roadmap. Mostly I'd like to hear where this pattern breaks for you. Reply here or email screwtapedev@gmail.com.