Skip to content

[Bug] Switching memory provider leaves stale sub-keys in merged config — causes gateway instability and crashes #58072

Description

@wyl1988

Bug Description

When a user switches memory.provider from one provider to another (e.g., hindsight → openviking), the config deep-merge logic updates the top-level provider key but does not remove the old provider's sub-keys.

For example, after changing memory.provider from hindsight to openviking:

# Profile config.yaml (correctly updated)
memory:
  provider: openviking

# But main config.yaml still has:
memory:
  provider: hindsight      # ← overridden by profile
  hindsight:                # ← stale sub-key NOT removed
    base_url: http://localhost:36813

Deep merge result: provider=openviking (from profile) + hindsight.base_url (from main, stale). Some code paths that read main config for provider initialization probe the stale hindsight.base_url, attempt to connect to a non-existent service (port 36813), fail, and cause:

  1. Gateway WSS disconnections (keepalive ping timeout)
  2. SIGTERM with exit code 1 (FAILURE)
  3. Automatic restart loops

Impact

Observed in production (20-profile deployment):

  • Gateway WSS disconnected twice in 40 minutes after switching from hindsight to openviking
  • Gateway crashed with exit code 1 and auto-restarted
  • Root cause was stale hindsight.base_url in main config.yaml that was never cleaned up

This affects any memory provider switch, not just hindsight → openviking.

Steps to Reproduce

  1. Configure memory.provider: hindsight with hindsight.base_url in main config.yaml
  2. In a profile config, set memory.provider: openviking
  3. Uninstall hindsight service (stop + disable)
  4. Restart gateway
  5. Observe WSS disconnections and gateway crashes

Expected Behavior

When memory.provider changes, the merge logic should either:

  1. Remove the old provider's sub-keys from the merged result
  2. Or warn the user about stale sub-keys

Workaround Applied

Manually remove stale hindsight sub-keys from main config.yaml.

Environment

  • Hermes Agent: 0.18.0
  • 20 profiles, switched from hindsight to openviking

Metadata

Metadata

Assignees

No one assigned

    Labels

    P2Medium — degraded but workaround existsarea/configConfig system, migrations, profilescomp/gatewayGateway runner, session dispatch, deliverysweeper:cannot-reproduceSweeper: could not reproduce on current maintool/memoryMemory tool and memory providerstype/bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions