Skip to content

[Bug]: Cron job main model silently falls back on 404 (tool-use unsupported) — run recorded as ok, no disclosure of substitution in last_error, audit log, or any surfaced event #41377

Open

Description

@nujovich

Bug Description

When a cron job declares a main model that exists on its provider but returns HTTP 404 "No endpoints found that support tool use" (because tools are present in the payload and no endpoint for that model supports tool-use), Hermes correctly falls back to the configured fallback_providers chain (openrouter/auto → resolves to google/gemini-2.5-flash-lite). The fallback itself is expected behavior per the provider-runtime docs (404 is a non-retryable error that triggers in-place swap).
The bug is the total absence of disclosure. The run is recorded as last_status: ok, last_error: null, still declaring the original model. There is no surfaced fallback event, no audit log entry, and no record anywhere of:

(a) that the declared model returned 404,
(b) that the reason was unsupported tool-use,
(c) which model actually executed.

The only way I detected the substitution was an independent telemetry plugin (hermes-telemetry) that records the executed model per LLM call and compares it against the declared model. Hermes' own state shows a clean, successful run on the model I never actually used.
This contradicts the v0.5.0 release note for #3153: "All retry, fallback, and compression events now surface to the user as formatted messages." That guarantee does not hold for a 404-triggered fallback on a cron job's main model.

Environment

Hermes Agent v0.16.0 (2026.6.5) · upstream a317e549
Python: 3.11.15 · OpenAI SDK: 2.24.0

Steps to Reproduce

Configure a cron job with a main model that exists on OpenRouter but has no tool-use-capable endpoint, e.g. nousresearch/hermes-4-70b, with provider: openrouter.
Ensure the job sends tools (e.g. enabled_toolsets: [web, terminal, file]).
Have fallback_providers configured (e.g. openrouter/auto).
Let the cron run.

Confirmed root cause, isolated from Hermes — direct OpenRouter call with one tool:

bash
curl -s https://openrouter.ai/api/v1/chat/completions \
  -H "Authorization: Bearer $OPENROUTER_API_KEY" -H "Content-Type: application/json" \
  -d '{"model":"nousresearch/hermes-4-70b",
       "messages":[{"role":"user","content":"hi"}],
       "tools":[{"type":"function","function":{"name":"t","description":"x","parameters":{"type":"object","properties":{}}}}]}'

Returns:

json
{"error":{"message":"No endpoints found that support tool use. Try disabling \"t\".","code":404}}
The model is in the OpenRouter catalog (nousresearch/hermes-4-70b resolves), so this is not a missing-model or bad-slug case — it is a capability mismatch (model exists, no tool-use endpoint) that surfaces only when tools are in the payload.

The model is in the OpenRouter catalog (nousresearch/hermes-4-70b resolves), so this is not a missing-model or bad-slug case — it is a capability mismatch (model exists, no tool-use endpoint) that surfaces only when tools are in the payload.

Expected Behavior

When the main model of a cron job is swapped via fallback due to a non-retryable error:

  • the run record should reflect the substitution (e.g., last_error populated with the 404 reason, or a dedicated fallback_reason / executed_model field),
  • the fallback event should surface per the #3153 guarantee,
  • An audit log entry should be written for the failed attempt.

Actual Behavior

  • jobs.json: model: nousresearch/hermes-4-70b, last_status: ok, last_error: null, last_delivery_error: null.
  • Executed model (per independent telemetry, all calls): google/gemini-2.5-flash-lite, 100% of LLM calls, real cost, estimated=0.
  • No request_dump_cron_* and no audit/cron_* artifact written for this job's runs.
  • No surfaced fallback message.

Affected Component

Tools (terminal, file ops, web, code execution, etc.)

Messaging Platform (if gateway-related)

N/A (CLI only)

Debug Report

Debug report uploaded:
  Report       https://paste.rs/bh7ws
  agent.log    https://paste.rs/EwkrE
  gateway.log  https://paste.rs/xPuBc
  desktop.log  https://paste.rs/QnFnZ

Operating System

Ubuntu 24.04.2 LTS (GNU/Linux 6.6.87.1-microsoft-standard-WSL2 x86_64)

Python Version

3.11.15

Hermes Version

0.16.0

Additional Logs / Traceback (optional)

Root Cause Analysis (optional)

How this differs from existing issues (not a duplicate)

#7559 / #24029 / #22201 — all about the auxiliary model chain (auxiliary_client.py, hardcoded gemini-*-flash for compression/vision/title). This is the main cron model path, different code, different trigger (404 tool-use, not key-limit/connection-error).
#5358 — about ignoring model.provider when OPENROUTER_API_KEY is set. Here the provider IS openrouter and is respected; the problem is post-routing disclosure.
#6673 — reports cron has no fallback chain. The opposite is observed here: the fallback did fire (swap to gemini succeeded). If #6673's fix landed, this is the disclosure gap that the fix introduced/left open.

Impact
Silent main-model substitution means cron output is generated by a different (often cheaper/weaker) model than declared, with no signal. For scheduled intelligence/analysis jobs, this is a correctness and cost-attribution problem: the operator believes results came from the declared model. Detectable today only with an external telemetry layer.
Detection
I found this with hermes-telemetry, a local SQLite telemetry plugin that records each LLM call's executed model/provider/cost and lets you reconcile declared-vs-executed per run. The declared model came from the job definition; the executed model came from the per-call records:

declared (runs table):      nousresearch/hermes-4-70b   status=ok
executed (llm_calls table):  google/gemini-2.5-flash-lite   (100% of calls, real cost)

Note this also exposes a blind spot worth flagging: the plugin's hooks (post_api_request) only observe the successful call to the fallback model — the failed 404 attempt on the declared model never produces a hook event, because it dies in Hermes' routing layer before any call record is created. So even external telemetry sees the result of the substitution, not the cause. The cause (the 404) is only recoverable by reproducing the OpenRouter call by hand, as shown above. This reinforces the core request: the 404 and the swap need to be surfaced by Hermes itself, since no downstream observer can see them.

Proposed Fix (optional)

The disclosure mechanism already exists — it just doesn't reach the cron record. _try_activate_fallback() in run_agent.py already emits a user-facing message on a successful swap (per #17446, around run_agent.py:7178):

🔄 Primary model failed — switching to fallback: <fb_model> via <fb_provider>

In an interactive session, this surfaces. In a cron run, it goes nowhere observable: it isn't persisted to the job record, written to an audit log, or attached to the delivered output. So the fix is not new disclosure logic — it's routing the existing fallback event into the cron run's persisted state.
Concretely, when _try_activate_fallback() performs the in-place swap, captures the transition and persists it on the run, so a fallback never leaves the record claiming the original model with status=ok / last_error=null. Minimal shape:

  • record the original (model, provider), the activated (model, provider), and the triggering error (status code + message) at the swap site;
  • write that onto the cron run record — e.g. populate last_error (or, better, a dedicated fallback_reason / executed_model field so a clean status can coexist with an honest substitution note);
  • emit an audit log entry for the failed attempt, matching the artifacts other run types produce.

This dovetails with the error-flagging hook some contributors are working on: the swap site in _try_activate_fallback() is the natural place to fire an error/transition hook, so the same event feeds (a) interactive surfacing, (b) cron persistence, and (c) any downstream observer — instead of the current state where the event only exists for interactive sessions and is lost everywhere else.

(Happy to take this as a PR if the maintainers agree on whether the substitution should land in last_error vs. a dedicated field — that's the one design decision I don't want to assume.)

Are you willing to submit a PR for this?

  • I'd like to fix this myself and submit a PR

Activity

  1. added
    type/bugSomething isn't working
    comp/cronCron scheduler and job management
    P2Medium — degraded but workaround exists
    on Jun 7, 2026
  2. liuhao1024 commented on Jun 7, 2026

    @liuhao1024

    This is fixed by #41421 — records the executed model in cron job metadata when provider-level fallback occurs, so
    ┌─────────────────────────────────────────────────────────────────────────┐
    │ Scheduled Jobs │
    └─────────────────────────────────────────────────────────────────────────┘

    15fcf6c5edca [active]
    Name: 每日日报生成
    Schedule: 0 9 * * *
    Repeat: ∞
    Next run: 2026-06-08T09:00:00+08:00
    Deliver: origin
    Skills: llm-wiki-ingest
    Workdir: /Users/liuhao/llm-wiki
    Last run: 2026-06-07T09:10:56.852347+08:00 ok
    ⚠ Delivery failed: delivery error: Weixin send failed: iLink sendmessage rate limited: ret=-2 errcode=None errmsg=rate limited

    1bdb58f5b361 [active]
    Name: 每周知识库周报
    Schedule: 0 9 * * 1
    Repeat: ∞
    Next run: 2026-06-08T09:00:00+08:00
    Deliver: origin
    Skills: llm-wiki-ingest
    Workdir: /Users/liuhao/llm-wiki
    Last run: 2026-06-01T09:09:17.208419+08:00 ok

    9f474479d762 [active]
    Name: hermes-bugfix-contribution
    Schedule: every 5m
    Repeat: ∞
    Next run: 2026-06-08T05:28:07.437117+08:00
    Deliver: local
    Skills: high-caution-hermes-upstream-contribution-lite, hermes-pr-tracker
    Script: upstream_contribution_preflight.sh
    Workdir: /Users/liuhao/.hermes/workdir/hermes-agent
    Last run: 2026-06-08T05:12:43.479917+08:00 ok

    aaf7642d874f [active]
    Name: hermes-pr-tracker-sync
    Schedule: every 30m
    Repeat: ∞
    Next run: 2026-06-08T05:46:39.827725+08:00
    Deliver: local
    Skills: hermes-pr-tracker
    Script: pr_tracker_sync.sh
    Mode: no-agent (script stdout delivered directly)
    Workdir: /Users/liuhao/.hermes/pr-tracker
    Last run: 2026-06-08T05:16:39.827725+08:00 ok

    3cbb2ca3002c [active]
    Name: hermes-pr-review
    Schedule: every 5m
    Repeat: ∞
    Next run: 2026-06-08T05:28:58.001500+08:00
    Deliver: local
    Skills: github-code-review-lite, hermes-pr-tracker
    Last run: 2026-06-08T05:13:58.001500+08:00 ok

    ed57c700c945 [active]
    Name: hermes-feat-contribution
    Schedule: every 10m
    Repeat: ∞
    Next run: 2026-06-08T05:25:35.744039+08:00
    Deliver: local
    Skills: hermes-pr-tracker
    Script: hermes_feat_preflight.sh
    Workdir: /Users/liuhao/.hermes/workdir/hermes-agent-feat
    Last run: 2026-06-08T05:15:35.744039+08:00 ok shows which model actually ran.

  3. GottZ commented on Jul 29, 2026

    @GottZ

    This was generated by AI during triage.

    Summary

    Two PRs were triaged in this complex, but only #41421 targets the reported silent provider fallback: it adds persisted executed-model and fallback-origin metadata, whereas #43956 removes per-job profile support and does not address fallback disclosure. #41421 is only a partial fix because its propagation is wired through an obsolete cron runner rather than the current shared execution path.

    Related pull requests

    Suggested consolidation

    Merge #41421 after reworking it onto the current run_one_job() path, adding an end-to-end runner regression test that persists fallback execution metadata, and resolving the documented maintainer schema decision; this follows rather than overrides its keep_open review. #43956 is already merged and unrelated, so there is no duplicate PR to close.

    Complex graph

    flowchart LR
        classDef open fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a
        classDef merged fill:#dcfce7,stroke:#15803d,color:#14532d
        classDef closed fill:#e5e7eb,stroke:#6b7280,color:#1f2937
        classDef unverified fill:#f3f4f6,stroke:#9ca3af,color:#374151
        classDef best stroke-width:3px,stroke:#b45309
        classDef target stroke-width:3px,stroke:#4338ca
        I41377(["issue #41377 (open)"])
        P41421["PR #41421 (open)"]
        P41421 -->|best fix| I41377
        class I41377 open
        class I41377 target
        class P41421 open
        class P41421 best
        click I41377 "https://github.com/NousResearch/hermes-agent/issues/41377"
        click P41421 "https://github.com/NousResearch/hermes-agent/pull/41421"
    
    Loading

    Graph: solid arrow = fixes / best fix, dashed arrow = partial or unverified (see edge label); boxed group = PRs duplicating each other; amber border = best fix; indigo border = target; gray node = closed or no verify verdict yet (state tag in the node label).

    Cross-PR triage: Reviewed 2 pull requests and 1 issue in this complex. Each diff was read against this issue; Assessment working set: 81 kB of PR diffs, 14 kB of issue/PR text, 9 kB of discussion (6 comments), 3 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Medium — degraded but workaround existscomp/cronCron scheduler and job managementtype/bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions