feat: Make Mixture-of-Agents (moa) fully configurable via config.yaml for cost/quality tuning

[apoapostolov]

feat: Make Mixture-of-Agents (moa / mixture_of_agents) fully configurable via config.yaml

Summary

The current mixture_of_agents implementation (the moa toolset) is hardcoded to use a specific set of expensive frontier models with maximum reasoning effort on every call. This makes it "stupidly expensive" for anything beyond occasional use and prevents users from tuning it for cost/quality tradeoffs.

Request: Add first-class support in config.yaml (under a moa: section) so users can define their own reference models, aggregator, temperatures, reasoning effort levels, and minimum successful references. Also expose the customization parameters in the tool schema so the agent (or user via /steer etc.) can override per-call when desired.

Additionally, provide session-level commands to turn MoA routing on and off dynamically, plus explicit slash commands that let a user force a specific request through the MoA path instead of the normal single-model route — all without leaving the current session or editing config files.

This would turn MoA from an all-or-nothing expensive hammer into a tunable, practical tool while preserving the high-end experience.

Current State (as of 2026-06)

• Hardcoded in tools/mixture_of_agents_tool.py:
  • REFERENCE_MODELS = ["anthropic/claude-opus-4.6", "google/gemini-2.5-pro", "openai/gpt-5.4-pro", "deepseek/deepseek-v3.2"]
  • AGGREGATOR_MODEL = "anthropic/claude-opus-4.6"
  • Fixed REFERENCE_TEMPERATURE = 0.6, AGGREGATOR_TEMPERATURE = 0.4
  • Every reference + aggregator call forces extra_body: { "reasoning": { "enabled": true, "effort": "xhigh" } }
  • MIN_SUCCESSFUL_REFERENCES = 1
• Tool schema (MOA_SCHEMA + registry.register) only declares user_prompt. The Python function accepts optional reference_models/aggregator_model but the registration lambda strips them:

  handler=lambda args, **kw: mixture_of_agents_tool(user_prompt=args.get("user_prompt", "")),

• No integration with ~/.hermes/config.yaml (or profile-specific configs).
• moa is deliberately in _DEFAULT_OFF_TOOLSETS in several places for cost reasons.
• OpenRouter-only via tools/openrouter_client.py (with OPENROUTER_API_KEY).

See:

• tools/mixture_of_agents_tool.py (lines ~64-78 for constants, ~519-542 for schema/registration)
• toolsets.py (moa definition)
• hermes_cli/tools_config.py and hermes_cli/config.py
• Existing related issues: feat: MoA customizável com multi-key NVIDIA NIM (interesse em PR?) #15101 (customizável with NVIDIA NIM), test coverage work on the tool, doctor/tools list inconsistency.

The module docstring itself admits the only customization path is "modify the configuration constants at the top of this file".

Proposed Design

1. Config schema (new moa: section)

moa:
  # Global defaults (used when moa toolset is enabled)
  reference_models:
    - "anthropic/claude-3.5-sonnet"      # much cheaper
    - "google/gemini-2.0-flash"
    - "deepseek/deepseek-chat"
  aggregator_model: "anthropic/claude-3.5-sonnet"
  reference_temperature: 0.7
  aggregator_temperature: 0.5
  reasoning_effort: "high"               # none | low | medium | high | xhigh (maps to extra_body)
  min_successful_references: 2
  max_tokens_per_reference: 16000
  max_tokens_aggregator: 32000

  # Optional: named profiles for different cost/quality tiers
  profiles:
    lite:
      reference_models:
        - "deepseek/deepseek-chat"
        - "google/gemini-2.0-flash"
      aggregator_model: "deepseek/deepseek-chat"
      reasoning_effort: "medium"
    research:
      reference_models:
        - "anthropic/claude-opus-4.6"
        - "openai/o1-preview"   # if/when available
      aggregator_model: "anthropic/claude-opus-4.6"
      reasoning_effort: "xhigh"

• If moa: section is absent, fall back to current expensive defaults (for backward compat).
• Support per-profile selection via a new optional tool param or future /moa profile foo.
• Validation on load (at least 1 reference model, valid effort levels).

2. Tool schema updates

Extend MOA_SCHEMA to accept optional overrides (so the agent can request a specific ensemble for a sub-task):

"properties": {
  "user_prompt": { ... },
  "reference_models": { "type": "array", "items": { "type": "string" } },
  "aggregator_model": { "type": "string" },
  "reasoning_effort": { "type": "string", "enum": ["none", "low", "medium", "high", "xhigh"] },
  "profile": { "type": "string" }   // e.g. "lite"
}

Update the handler lambda to forward the new fields to mixture_of_agents_tool(...).

3. Implementation in the tool

• Move constants into a get_moa_defaults() function that reads from Hermes config (reuse patterns from hermes_cli/config.py, agent/config, or get_hermes_home() + yaml load).
• Merge: call-time overrides > named profile > top-level moa: section > hardcoded legacy defaults.
• Map reasoning_effort string to the extra_body structure (with graceful fallback).
• Update _run_reference_model_safe and _run_aggregator_model to respect per-model or global settings.
• Keep the debug / get_moa_configuration helpers and extend them.
• Add clear logging: "Using MoA config from ~/.hermes/config.yaml (profile: lite, 3 refs @ medium reasoning)"

4. Other changes

• toolsets.py, hermes_cli/tools_config.py: surface the new config keys.
• hermes config edit / setup wizard: optional prompt for MoA tier.
• Update hermes doctor to report effective MoA config.
• Docs: website/docs/reference/tools-reference.md, configuration guide, and an example in cli-config.yaml.example.
• Tests: add cases for config loading, profile merging, schema overrides, and cost-related assertions.
• Optional future: support non-OpenRouter providers if the client layer generalizes.

5. Session-level control and explicit per-request routing (slash commands)

To make MoA usable in mixed workloads without constant config changes or session restarts, add two complementary control mechanisms:

A. Dynamic session toggles (in-memory state for the current conversation)

• /moa on — Enables MoA-aware routing for the rest of the session. The agent will now consider calling mixture_of_agents for sufficiently hard or complex sub-tasks, using the loaded moa: config (or active profile).
• /moa off — Disables MoA routing and returns to normal single-model behavior.
• /moa status — Shows current mode (on/off), active profile, effective reference/aggregator models, reasoning effort, and a rough cost estimate relative to the base model.
• /moa profile <name> — Switches the active profile for the remainder of the session (e.g. /moa profile lite).

These toggles are conversation-scoped and do not modify config.yaml. They survive normal resets within the same session but are cleared on a full new session.

B. Explicit slash-command routing

Users can force a specific request through the MoA path regardless of the current on/off state or whether the moa toolset is globally enabled:

• /moa <prompt> — Directly routes the prompt through Mixture-of-Agents using the current effective config/profile.
  Example:

  /moa Design a correct and efficient algorithm for finding the minimum spanning tree in a graph with 10^6 nodes
  

• With profile override:

  /moa --profile lite "Quick sanity-check this refactor for obvious bugs"
  /moa profile:research "Produce a rigorous proof for this conjecture"
  

The explicit form bypasses the agent's normal tool-selection heuristics and calls mixture_of_agents_tool directly. This is extremely useful for one-off hard problems.

Implementation notes for the control layer

• These can be implemented as standard slash commands (similar to the existing /compress, /status, etc.).
• Session state (moa_enabled, active_profile) can live alongside existing per-session state used by the context engine and memory system.
• When /moa on is active, the system can surface MoA capability more visibly in the prompt or tool list.
• The explicit /moa command should work even if the moa toolset is disabled in the global toolsets (it can still load the config section and enforce the OPENROUTER_API_KEY requirement).
• Good error messages: "MoA is currently off. Use /moa on or /moa <prompt> to force a single request."
• These commands should be available as soon as the MoA code is loaded (no hard requirement that the whole toolset be enabled).

Backward Compatibility

• Existing behavior unchanged when no moa: section is present.
• Current hardcoded models remain the implicit "full" default.
• Existing calls with only user_prompt continue to work.
• All new slash commands are additive; old sessions and configs are unaffected.

Life Use Cases (Realistic & Valuable)

1. Cost-Conscious Professional Development
   A full-time engineer enables a lite MoA profile (Sonnet + Flash + Deepseek, high effort) for every significant code review or refactoring PR. Previously $1–3 per invocation made it unusable. Now ~$0.05–0.15. The multi-model consensus still catches subtle bugs better than a single model while staying inside monthly API budget.

2. Autonomous Research / Daily Synthesis Agents
   A cron job runs every morning: "Read the latest arXiv papers in my field and produce a synthesized briefing." Using full opus MoA for 30 days would be hundreds of dollars. With a configurable research_lite profile using efficient models + medium reasoning, the user gets diverse perspectives and aggregation quality at sustainable cost. The agent can still escalate to the research profile (full opus) for a particularly important paper via per-call override.

3. Indie Hacker / Bootstrapped Side Projects
   Someone building a niche AI coding assistant on limited credits wants occasional "super reasoning" without switching providers. They define a 2-model cheap MoA (Deepseek + Gemini Flash) and only invoke it for the final architecture decision or tricky algorithm. This gives most of the MoA benefit at a fraction of the price.

4. Domain-Specific or Multilingual Ensembles

   • Math-heavy work: Prioritize models strong in formal reasoning (specific Deepseek variant + Gemini + a code model).
   • Non-English technical work: Mix models known to perform well in the target language.
   • Users can maintain and share "best known MoA configs" for particular domains (e.g., via skills or gists), something impossible today.

5. Budgeted Long-Running or Multi-Tenant Agents
   In a Kanban worker or gateway session handling many users, the main loop uses a cheap single model. Only when a task is marked "high complexity" does it call mixture_of_agents with a conservative profile that caps references at 2 and reasoning at "medium". Per-call profile selection + config defaults gives fine-grained cost control inside one long-lived agent.

6. A/B Testing, Experimentation & Education
   Researchers or students can rapidly iterate: "What happens to consensus quality if I drop temperature to 0.3 and use only 2 references?" or "Does adding a weaker but very different model improve results?" Per-call overrides + config profiles make this trivial. Great for teaching MoA concepts without burning real money.

7. Enterprise / Team Guardrails
   A company sets a conservative moa: section in their shared Hermes profile (max 2 refs, no xhigh, specific approved models only). Junior developers or automated agents cannot accidentally trigger $2+ calls. Overrides can be allowed only for power users.

8. Progressive Adoption Path
   New users start with moa disabled. They enable a very cheap 2-model profile first, get comfortable with the quality improvement, then gradually upgrade individual models in their config as budget allows. This lowers the activation energy dramatically compared to "either use the $2 opus version or nothing."

New use cases enabled by session commands and explicit routing:

9. Mixed-workload interactive sessions
   User is doing normal development with a cheap base model. They hit a genuinely hard algorithmic problem and type:

   /moa Design an optimal solution for the following constraint satisfaction problem with 50 variables...
   

   MoA runs once with the expensive (or custom) profile. After getting the answer they type /moa off and continue the rest of the session normally. No config edit, no restart.

10. Quick escalation inside long agent runs
    While working on a big task the agent says "This sub-problem looks particularly difficult." The user replies:

    /moa profile:research Solve the sub-problem above with maximum rigor
    

    The explicit command forces MoA on that specific piece while the rest of the plan continues with the normal model.

11. Temporary "thinking cap" for a whole session
    User starts a deep technical discussion and wants higher quality for the entire conversation:

    /moa on
    /moa profile lite
    

    They work for an hour with MoA routing enabled, then /moa off before switching to casual chat or cheaper tasks.

Why This Fits Hermes

• Hermes already has rich config for model, compression, memory providers, delegation, auxiliary models, etc.
• It aligns with the "profiles" system and the philosophy of user-controlled, environment-specific behavior.
• Similar to how context.engine or memory backends are pluggable/configurable.
• Would make the existing "customize at top of file" comment obsolete and reduce the need for source edits.
• Adding slash-command control follows the existing pattern of powerful in-session commands (/compress, etc.).

This would make advanced multi-model reasoning accessible to far more users without sacrificing the "maximum quality" path for those who need (and can afford) it.

Optional / Low-Priority Nice-to-Haves

These are not required for the core value of the feature but would be valuable follow-up improvements:

• Pre-flight estimation / dry-run: A /moa estimate "..." (or /moa --dry-run) command that reports the models that would be used, number of API calls, active profile, and a rough cost multiplier before actually running MoA. This directly reduces the risk of unexpected expense.

• Basic guardrails in config: Simple safety options such as max_references, max_reasoning_effort, and warn_if_cost_exceeds to prevent overly expensive profiles from being used accidentally.

• Delegation / sub-agent integration: Session MoA state (/moa on + active profile) should be inherited by sub-agents created via delegate_task, or at minimum be easy to pass explicitly when spawning sub-agents.

• Verification / second-opinion workflow: Treat MoA as a dedicated "final check" or red-team tool for high-stakes outputs (security-sensitive code, financial models, production infrastructure changes, legal/medical reasoning, etc.). Users could do the bulk of the work with a normal model and only run MoA at the end for consensus validation.

These can be added incrementally after the main configuration and slash-command controls are in place.

References

• Original MoA paper: https://arxiv.org/abs/2406.04692
• Current implementation: tools/mixture_of_agents_tool.py
• Related: feat: MoA customizável com multi-key NVIDIA NIM (interesse em PR?) #15101 (customizable MoA interest)

[ab-qf]

+1 — the "edit source code" workaround is not viable long-term

I hit this today: wanted to customize the MoA reference/aggregator models, and the module docstring itself says "modify the configuration constants at the top of this file." That's the only documented customization path.

The problem is obvious: hermes update overwrites the source file, so any edits to REFERENCE_MODELS / AGGREGATOR_MODEL are lost on every update. This mixes configuration with code — exactly the anti-pattern that config.yaml exists to prevent.

Every other configurable subsystem in Hermes (model, compression, delegation, memory, auxiliary, TTS, etc.) reads from config.yaml. MoA should too.

The proposed moa: section in the OP is the right design. A minimal first step — just reading reference_models and aggregator_model from config with today's hardcoded values as fallback — would already solve the core problem without any of the more advanced features (profiles, slash commands, etc.).

[teknium1]

Closing as implemented by the MoA redesign (June 2026). MoA is now fully configurable via config.yaml moa: presets — per-preset reference_models (a list of {provider, model} slots), aggregator, reference_temperature, aggregator_temperature, and max_tokens, all editable via hermes moa configure or by hand. Cost/quality tuning is exactly per-preset model + temperature selection. Thanks for pushing on this — it's the shape MoA shipped in.

[070freebird070-ctrl]

For anyone landing here wanting the other half of this — the model escalating to MoA on its own judgment mid-run (rather than the user typing /moa) — I built a small user plugin that does exactly that, no core changes:

https://github.com/070freebird070-ctrl/hermes-consult-committee

It registers a consult_committee(question) tool that fans the question out to your configured moa.presets reference models in parallel and returns their labelled answers as the tool result, so the acting model synthesizes with full conversation context (same aggregator-is-the-acting-model shape as upstream MoA). Budget-capped, graceful on partial advisor failures.

One honest finding from testing worth sharing: small local models under-call help tools badly (0/4 self-escalations in our tests, consistent with the 2026 escalation literature) — until you add an explicit "call consult_committee FIRST on expert-level questions" rule to the system prompt, after which unprompted escalation worked reliably. Details and the exact prompt rule are in the README.