Skip to content

[Bug]: cronjob tool can remain hidden if tool definitions are cached before gateway env is set #35561

Open

Description

@moonbite-benie

Bug Description

This is likely an edge case rather than a common plain-gateway startup bug. I am filing it because the repro uses only upstream modules and points to a cache invalidation boundary: tool availability can depend on environment variables, but the quiet-mode tool-definition cache does not include those env values.

Specifically, the cronjob tool can remain hidden if tool definitions are inspected before gateway.run sets HERMES_EXEC_ASK=1.

I hit this in a downstream setup that performs tool-visibility/status checks around the gateway. A plain hermes gateway run startup may not hit this ordering if nothing probes tool definitions before the gateway module finishes initializing.

However, the minimal repro below uses only upstream modules. The issue is not specific to any private tool or custom configuration.

Steps to Reproduce

  1. In a clean detached v2026.5.28 worktree, run this script:
import os

for key in ("HERMES_GATEWAY_SESSION", "HERMES_EXEC_ASK", "HERMES_INTERACTIVE"):
    os.environ.pop(key, None)

from model_tools import get_tool_definitions

pre = [
    tool["function"]["name"]
    for tool in get_tool_definitions(
        enabled_toolsets=["cronjob"],
        quiet_mode=True,
    )
]

import gateway.run

post = [
    tool["function"]["name"]
    for tool in get_tool_definitions(
        enabled_toolsets=["cronjob"],
        quiet_mode=True,
    )
]

print({
    "pre": pre,
    "exec_ask": os.getenv("HERMES_EXEC_ASK"),
    "post": post,
})
  1. Observe that importing gateway.run sets HERMES_EXEC_ASK=1.
  2. Observe that cronjob is still absent from the second tool-definition lookup.
  3. I also reproduced the same ordering on current origin/main (b1a2540) using skip_tool_search_assembly=True.

Expected Behavior

After gateway.run marks the process as a messaging gateway approval context via HERMES_EXEC_ASK=1, cronjob should become visible, or any stale pre-gateway tool availability cache should be invalidated.

Actual Behavior

cronjob remains hidden because the earlier quiet_mode=True tool-definition result is reused.

Observed output from a clean detached v2026.5.28 worktree:

{'pre': [], 'exec_ask': '1', 'post': []}

Observed output from a clean detached current origin/main worktree:

{'pre': [], 'exec_ask': '1', 'post': []}

Affected Component

Tools (terminal, file ops, web, code execution, etc.)

Messaging Platform (if gateway-related)

No response

Debug Report

Not included. I did not run `hermes debug share` because the repro is a minimal import/cache-order script using upstream modules, and my live deployment contains private/customized configuration.

Operating System

macOS 26.3

Python Version

Python 3.11.15

Hermes Version

v2026.5.28; also reproduced on origin/main b1a2540

Additional Logs / Traceback (optional)

No traceback. Minimal repro output:


{'pre': [], 'exec_ask': '1', 'post': []}

Root Cause Analysis (optional)

tools.cronjob_tools.check_cronjob_requirements() enables the tool only when one of these env vars is truthy:

  • HERMES_INTERACTIVE
  • HERMES_GATEWAY_SESSION
  • HERMES_EXEC_ASK

model_tools.get_tool_definitions(..., quiet_mode=True) memoizes the filtered tool list.

In gateway/run.py, HERMES_EXEC_ASK=1 is set during module initialization, but not before every possible prior import/probe of model_tools. If another path in the same process calls get_tool_definitions() first, cronjob can be cached as unavailable. Importing gateway.run later sets HERMES_EXEC_ASK=1, but the stale cached tool list is still reused.

Proposed Fix (optional)

A few possible approaches:

  1. Set the regular gateway env markers (HERMES_QUIET, HERMES_EXEC_ASK) earlier, before any imports/config validation paths can probe tool availability, and invalidate already-loaded tool/check caches after setting them.
  2. Include the relevant tool-availability env fingerprint in the get_tool_definitions cache key, so env changes that affect check functions cannot reuse stale results.
  3. Expose a small explicit cache invalidation call for gateway startup after it sets process-level tool context.

Are you willing to submit a PR for this?

  • I'd like to fix this myself and submit a PR

Activity

  1. added
    type/bugSomething isn't working
    comp/toolsTool registry, model_tools, toolsets
    comp/cronCron scheduler and job management
    P2Medium — degraded but workaround exists
    on May 30, 2026
  2. alt-glitch commented on May 30, 2026

    @alt-glitch

    Related: #33423 (fix: invalidate tool caches on dotenv reload — same _tool_defs_cache code path, different trigger), #22349 (quiet-mode cache not scoped by sender identity — closed).

    This issue identifies a new invalidation boundary: env-var-gated tool availability (HERMES_EXEC_ASK) is not reflected in the quiet-mode tool-definition cache after the env changes mid-process.

  3. bbopen commented on Jul 5, 2026

    @bbopen

    Opened draft PR #58635 for this: #58635

    The patch covers two repro paths from this issue class:

    • no-session quiet-mode lookup caches cronjob as absent, then HERMES_EXEC_ASK=1 appears and cronjob becomes visible
    • no-session quiet-mode lookup caches cronjob as absent, then a Matrix-style gateway ContextVar session appears and cronjob becomes visible

    It also keeps the fix narrower than broad dotenv/runtime env reload invalidation by keying availability caches on explicit gateway/session context plus the existing explicit invalidation path.

  4. GottZ commented on Jul 28, 2026

    @GottZ

    This was generated by AI during triage.

    Summary

    Two PRs address #35561’s stale quiet-mode tool-availability cache after gateway/session context changes. #35605 covers the reported process-environment flip but misses current task-local ContextVar state and outer-cache invalidation boundaries, while #58635 coordinates both cache layers across environment, ContextVar, profile, TTL, and explicit-invalidation transitions.

    Related pull requests

    • fix(tools): refresh cache when session env changes #35605 partial — (+62/-3) — partial fix: Tracks process-level availability environment variables and adds the focused HERMES_EXEC_ASK regression, but it does not cover the current ContextVar gateway-session path or fully coordinate registry and outer schema-cache invalidation. Consistent with the contributor keep_open review, its regression should be salvaged into fix(tools): scope availability cache by session context #58635 rather than merging this branch as-is; the author also accepted that disposition.
    • fix(tools): scope availability cache by session context #58635 best fix — (+486/-35) — fixes: Keys registry availability by the relevant profile/session context, propagates availability identity and expiry into the outer schema cache, advances an epoch on explicit invalidation, and adds env-flip, ContextVar-transition, TTL/grace, profile-isolation, and explicit-invalidation regressions. The contributor keep_open review rates this approach highly salvageable and confirms that the diff directly covers both stale-cache boundaries.

    Duplicates

    #35605 and #58635 target the same HERMES_EXEC_ASK stale-cache cause, but #58635 supersedes #35605 by also covering task-local ContextVar sessions and coordinated registry/outer-cache invalidation.

    Suggested consolidation

    Merge #58635 as the comprehensive fix for #35561; close #35605 as superseded after retaining its focused HERMES_EXEC_ASK regression coverage. This follows the contributor review on #35605, which explicitly recommends consolidation into #58635, while the keep_open review on #58635 confirms that its diff addresses the relevant cache layers and current ContextVar path.

    Complex graph

    flowchart LR
        classDef open fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a
        classDef merged fill:#dcfce7,stroke:#15803d,color:#14532d
        classDef closed fill:#e5e7eb,stroke:#6b7280,color:#1f2937
        classDef unverified fill:#f3f4f6,stroke:#9ca3af,color:#374151
        classDef best stroke-width:3px,stroke:#b45309
        classDef target stroke-width:3px,stroke:#4338ca
        I35561(["issue #35561 (open)"])
        subgraph Dup35605 ["PRs duplicating each other"]
            P35605["PR #35605 (open)"]
            P58635["PR #58635 (open)"]
        end
        P35605 -.->|partial| I35561
        P58635 -->|best fix| I35561
        class I35561 open
        class I35561 target
        class P35605 open
        class P58635 open
        class P58635 best
        click I35561 "https://github.com/NousResearch/hermes-agent/issues/35561"
        click P35605 "https://github.com/NousResearch/hermes-agent/pull/35605"
        click P58635 "https://github.com/NousResearch/hermes-agent/pull/58635"
    
    Loading

    Graph: solid arrow = fixes / best fix, dashed arrow = partial or unverified (see edge label); boxed group = PRs duplicating each other; amber border = best fix; indigo border = target; gray node = closed or no verify verdict yet (state tag in the node label).

    Cross-PR triage: Reviewed 2 pull requests and 1 issue in this complex. Each diff was read against this issue; Assessment working set: 37 kB of PR diffs, 11 kB of issue/PR text, 7 kB of discussion (12 comments), 4 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.

  5. GottZ commented on Jul 30, 2026

    @GottZ

    This was generated by AI during triage.

    Delta since our previous triage comment

    Since the previous comment, @kiranmagic7 closed #35605 as stale WIP and confirmed that its process-environment fingerprint is not suitable for the current availability-context and outer-cache-epoch architecture. This reinforces rather than changes the earlier partial-fix assessment; #58635 remains the best existing fix.

    Changed pull requests

    Suggested consolidation

    Recommendation updated for the current backlog lane: keep #58635 open, consistent with its keep_open review, as the best existing fix pending maintainer/author activation rather than merge it now; #35605 is already closed as superseded.

    Complex graph

    flowchart LR
        classDef open fill:#dbeafe,stroke:#1d4ed8,color:#1e3a8a
        classDef merged fill:#dcfce7,stroke:#15803d,color:#14532d
        classDef closed fill:#e5e7eb,stroke:#6b7280,color:#1f2937
        classDef unverified fill:#f3f4f6,stroke:#9ca3af,color:#374151
        classDef best stroke-width:3px,stroke:#b45309
        classDef target stroke-width:3px,stroke:#4338ca
        I35561(["issue #35561 (open)"])
        subgraph Dup35605 ["PRs duplicating each other"]
            P35605["PR #35605 (closed)"]
            P58635["PR #58635 (open)"]
        end
        P35605 -.->|partial| I35561
        P58635 -->|best fix| I35561
        class I35561 open
        class I35561 target
        class P35605 closed
        class P58635 open
        class P58635 best
        click I35561 "https://github.com/NousResearch/hermes-agent/issues/35561"
        click P35605 "https://github.com/NousResearch/hermes-agent/pull/35605"
        click P58635 "https://github.com/NousResearch/hermes-agent/pull/58635"
        classDef changedSince stroke:#dc2626,stroke-width:3px,stroke-dasharray: 6 3
        class P35605 changedSince
    
    Loading

    Graph: solid arrow = fixes / best fix, dashed arrow = partial or unverified (see edge label); boxed group = PRs duplicating each other; amber border = best fix; indigo border = target; gray node = closed (state tag in the node label).
    Dashed red border = changed since our previous triage comment (1 added/updated, 1 removed graph elements).

    Cross-PR triage: Reviewed 2 pull requests and 1 issue in this complex. Each diff was read against this issue; Assessment working set: 37 kB of PR diffs, 11 kB of issue/PR text, 6 kB of discussion (11 comments), 3 verify verdicts. verdicts reflect diff content, not PR titles. Part of an automated triage batch.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Medium — degraded but workaround existscomp/cronCron scheduler and job managementcomp/toolsTool registry, model_tools, toolsetssweeper:risk-session-stateSweeper risk: may lose/corrupt/mis-associate session or context statetype/bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions