Repository navigation
Description
Bug Description
When a cron job declares a main model that exists on its provider but returns HTTP 404 "No endpoints found that support tool use" (because tools are present in the payload and no endpoint for that model supports tool-use), Hermes correctly falls back to the configured fallback_providers chain (openrouter/auto → resolves to google/gemini-2.5-flash-lite). The fallback itself is expected behavior per the provider-runtime docs (404 is a non-retryable error that triggers in-place swap).
The bug is the total absence of disclosure. The run is recorded as last_status: ok, last_error: null, still declaring the original model. There is no surfaced fallback event, no audit log entry, and no record anywhere of:
(a) that the declared model returned 404,
(b) that the reason was unsupported tool-use,
(c) which model actually executed.
The only way I detected the substitution was an independent telemetry plugin (hermes-telemetry) that records the executed model per LLM call and compares it against the declared model. Hermes' own state shows a clean, successful run on the model I never actually used.
This contradicts the v0.5.0 release note for #3153: "All retry, fallback, and compression events now surface to the user as formatted messages." That guarantee does not hold for a 404-triggered fallback on a cron job's main model.
Environment
Hermes Agent v0.16.0 (2026.6.5) · upstream a317e549
Python: 3.11.15 · OpenAI SDK: 2.24.0
Steps to Reproduce
Configure a cron job with a main model that exists on OpenRouter but has no tool-use-capable endpoint, e.g. nousresearch/hermes-4-70b, with provider: openrouter.
Ensure the job sends tools (e.g. enabled_toolsets: [web, terminal, file]).
Have fallback_providers configured (e.g. openrouter/auto).
Let the cron run.
Confirmed root cause, isolated from Hermes — direct OpenRouter call with one tool:
bash
curl -s https://openrouter.ai/api/v1/chat/completions \
-H "Authorization: Bearer $OPENROUTER_API_KEY" -H "Content-Type: application/json" \
-d '{"model":"nousresearch/hermes-4-70b",
"messages":[{"role":"user","content":"hi"}],
"tools":[{"type":"function","function":{"name":"t","description":"x","parameters":{"type":"object","properties":{}}}}]}'
Returns:
json
{"error":{"message":"No endpoints found that support tool use. Try disabling \"t\".","code":404}}
The model is in the OpenRouter catalog (nousresearch/hermes-4-70b resolves), so this is not a missing-model or bad-slug case — it is a capability mismatch (model exists, no tool-use endpoint) that surfaces only when tools are in the payload.
The model is in the OpenRouter catalog (nousresearch/hermes-4-70b resolves), so this is not a missing-model or bad-slug case — it is a capability mismatch (model exists, no tool-use endpoint) that surfaces only when tools are in the payload.
Expected Behavior
When the main model of a cron job is swapped via fallback due to a non-retryable error:
- the run record should reflect the substitution (e.g.,
last_errorpopulated with the 404 reason, or a dedicatedfallback_reason/executed_modelfield), - the fallback event should surface per the #3153 guarantee,
- An audit log entry should be written for the failed attempt.
Actual Behavior
jobs.json:model: nousresearch/hermes-4-70b,last_status: ok,last_error: null,last_delivery_error: null.- Executed model (per independent telemetry, all calls):
google/gemini-2.5-flash-lite,100% of LLM calls, real cost,estimated=0. - No
request_dump_cron_*and noaudit/cron_*artifact written for this job's runs. - No surfaced fallback message.
Affected Component
Tools (terminal, file ops, web, code execution, etc.)
Messaging Platform (if gateway-related)
N/A (CLI only)
Debug Report
Debug report uploaded:
Report https://paste.rs/bh7ws
agent.log https://paste.rs/EwkrE
gateway.log https://paste.rs/xPuBc
desktop.log https://paste.rs/QnFnZOperating System
Ubuntu 24.04.2 LTS (GNU/Linux 6.6.87.1-microsoft-standard-WSL2 x86_64)
Python Version
3.11.15
Hermes Version
0.16.0
Additional Logs / Traceback (optional)
Root Cause Analysis (optional)
How this differs from existing issues (not a duplicate)
#7559 / #24029 / #22201 — all about the auxiliary model chain (auxiliary_client.py, hardcoded gemini-*-flash for compression/vision/title). This is the main cron model path, different code, different trigger (404 tool-use, not key-limit/connection-error).
#5358 — about ignoring model.provider when OPENROUTER_API_KEY is set. Here the provider IS openrouter and is respected; the problem is post-routing disclosure.
#6673 — reports cron has no fallback chain. The opposite is observed here: the fallback did fire (swap to gemini succeeded). If #6673's fix landed, this is the disclosure gap that the fix introduced/left open.
Impact
Silent main-model substitution means cron output is generated by a different (often cheaper/weaker) model than declared, with no signal. For scheduled intelligence/analysis jobs, this is a correctness and cost-attribution problem: the operator believes results came from the declared model. Detectable today only with an external telemetry layer.
Detection
I found this with hermes-telemetry, a local SQLite telemetry plugin that records each LLM call's executed model/provider/cost and lets you reconcile declared-vs-executed per run. The declared model came from the job definition; the executed model came from the per-call records:
declared (runs table): nousresearch/hermes-4-70b status=ok
executed (llm_calls table): google/gemini-2.5-flash-lite (100% of calls, real cost)
Note this also exposes a blind spot worth flagging: the plugin's hooks (post_api_request) only observe the successful call to the fallback model — the failed 404 attempt on the declared model never produces a hook event, because it dies in Hermes' routing layer before any call record is created. So even external telemetry sees the result of the substitution, not the cause. The cause (the 404) is only recoverable by reproducing the OpenRouter call by hand, as shown above. This reinforces the core request: the 404 and the swap need to be surfaced by Hermes itself, since no downstream observer can see them.
Proposed Fix (optional)
The disclosure mechanism already exists — it just doesn't reach the cron record. _try_activate_fallback() in run_agent.py already emits a user-facing message on a successful swap (per #17446, around run_agent.py:7178):
🔄 Primary model failed — switching to fallback: <fb_model> via <fb_provider>
In an interactive session, this surfaces. In a cron run, it goes nowhere observable: it isn't persisted to the job record, written to an audit log, or attached to the delivered output. So the fix is not new disclosure logic — it's routing the existing fallback event into the cron run's persisted state.
Concretely, when _try_activate_fallback() performs the in-place swap, captures the transition and persists it on the run, so a fallback never leaves the record claiming the original model with status=ok / last_error=null. Minimal shape:
- record the original
(model, provider), the activated(model, provider), and the triggering error (status code + message) at the swap site; - write that onto the cron run record — e.g. populate
last_error(or, better, a dedicatedfallback_reason/executed_modelfield so a clean status can coexist with an honest substitution note); - emit an audit log entry for the failed attempt, matching the artifacts other run types produce.
This dovetails with the error-flagging hook some contributors are working on: the swap site in _try_activate_fallback() is the natural place to fire an error/transition hook, so the same event feeds (a) interactive surfacing, (b) cron persistence, and (c) any downstream observer — instead of the current state where the event only exists for interactive sessions and is lost everywhere else.
(Happy to take this as a PR if the maintainers agree on whether the substitution should land in last_error vs. a dedicated field — that's the one design decision I don't want to assume.)
Are you willing to submit a PR for this?
- I'd like to fix this myself and submit a PR
Activity
liuhao1024 commentedon Jun 7, 2026
This is fixed by #41421 — records the executed model in cron job metadata when provider-level fallback occurs, so
┌─────────────────────────────────────────────────────────────────────────┐
│ Scheduled Jobs │
└─────────────────────────────────────────────────────────────────────────┘
15fcf6c5edca [active]
Name: 每日日报生成
Schedule: 0 9 * * *
Repeat: ∞
Next run: 2026-06-08T09:00:00+08:00
Deliver: origin
Skills: llm-wiki-ingest
Workdir: /Users/liuhao/llm-wiki
Last run: 2026-06-07T09:10:56.852347+08:00 ok
⚠ Delivery failed: delivery error: Weixin send failed: iLink sendmessage rate limited: ret=-2 errcode=None errmsg=rate limited
1bdb58f5b361 [active]
Name: 每周知识库周报
Schedule: 0 9 * * 1
Repeat: ∞
Next run: 2026-06-08T09:00:00+08:00
Deliver: origin
Skills: llm-wiki-ingest
Workdir: /Users/liuhao/llm-wiki
Last run: 2026-06-01T09:09:17.208419+08:00 ok
9f474479d762 [active]
Name: hermes-bugfix-contribution
Schedule: every 5m
Repeat: ∞
Next run: 2026-06-08T05:28:07.437117+08:00
Deliver: local
Skills: high-caution-hermes-upstream-contribution-lite, hermes-pr-tracker
Script: upstream_contribution_preflight.sh
Workdir: /Users/liuhao/.hermes/workdir/hermes-agent
Last run: 2026-06-08T05:12:43.479917+08:00 ok
aaf7642d874f [active]
Name: hermes-pr-tracker-sync
Schedule: every 30m
Repeat: ∞
Next run: 2026-06-08T05:46:39.827725+08:00
Deliver: local
Skills: hermes-pr-tracker
Script: pr_tracker_sync.sh
Mode: no-agent (script stdout delivered directly)
Workdir: /Users/liuhao/.hermes/pr-tracker
Last run: 2026-06-08T05:16:39.827725+08:00 ok
3cbb2ca3002c [active]
Name: hermes-pr-review
Schedule: every 5m
Repeat: ∞
Next run: 2026-06-08T05:28:58.001500+08:00
Deliver: local
Skills: github-code-review-lite, hermes-pr-tracker
Last run: 2026-06-08T05:13:58.001500+08:00 ok
ed57c700c945 [active]
Name: hermes-feat-contribution
Schedule: every 10m
Repeat: ∞
Next run: 2026-06-08T05:25:35.744039+08:00
Deliver: local
Skills: hermes-pr-tracker
Script: hermes_feat_preflight.sh
Workdir: /Users/liuhao/.hermes/workdir/hermes-agent-feat
Last run: 2026-06-08T05:15:35.744039+08:00 ok shows which model actually ran.
GottZ commentedon Jul 29, 2026
Summary
Two PRs were triaged in this complex, but only #41421 targets the reported silent provider fallback: it adds persisted executed-model and fallback-origin metadata, whereas #43956 removes per-job profile support and does not address fallback disclosure. #41421 is only a partial fix because its propagation is wired through an obsolete cron runner rather than the current shared execution path.
Related pull requests
best fix— (+118/-63) — partial fix: Addslast_executed_modelandlast_fallback_fromstorage and derives them from the agent result, directly addressing which model executed after a fallback. Consistent with the keep_open review on fix(cron): record executed model in job metadata on provider-level fallback #41421, the diff targets the superseded six-valuerun_job()path and tests only direct metadata storage, so it must be rebased onto currentrun_one_job()with a runner-level persistence test and maintainer confirmation of the schema.unrelated— (+19/-672) — unrelated: Removes per-job profile support across cron storage, scheduling, CLI, tooling, tests, and documentation; it neither captures provider fallback events nor records the executed model. Although already merged, it remains relevant as cron execution-path context for rebasing fix(cron): record executed model in job metadata on provider-level fallback #41421, not as an implementation of [Bug]: Cron job main model silently falls back on 404 (tool-use unsupported) — run recorded as ok, no disclosure of substitution in last_error, audit log, or any surfaced event #41377.Suggested consolidation
Merge #41421 after reworking it onto the current
run_one_job()path, adding an end-to-end runner regression test that persists fallback execution metadata, and resolving the documented maintainer schema decision; this follows rather than overrides its keep_open review. #43956 is already merged and unrelated, so there is no duplicate PR to close.Complex graph
flowchart LR classDef open fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a classDef merged fill:#dcfce7,stroke:#15803d,color:#14532d classDef closed fill:#e5e7eb,stroke:#6b7280,color:#1f2937 classDef unverified fill:#f3f4f6,stroke:#9ca3af,color:#374151 classDef best stroke-width:3px,stroke:#b45309 classDef target stroke-width:3px,stroke:#4338ca I41377(["issue #41377 (open)"]) P41421["PR #41421 (open)"] P41421 -->|best fix| I41377 class I41377 open class I41377 target class P41421 open class P41421 best click I41377 "https://github.com/NousResearch/hermes-agent/issues/41377" click P41421 "https://github.com/NousResearch/hermes-agent/pull/41421"Graph: solid arrow = fixes / best fix, dashed arrow = partial or unverified (see edge label); boxed group = PRs duplicating each other; amber border = best fix; indigo border = target; gray node = closed or no verify verdict yet (state tag in the node label).
Cross-PR triage: Reviewed 2 pull requests and 1 issue in this complex. Each diff was read against this issue; Assessment working set: 81 kB of PR diffs, 14 kB of issue/PR text, 9 kB of discussion (6 comments), 3 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.