Skip to content

Memory release strategy for memory guard #1945

Description

@Alfabeta01

I use hermes agent 0.17.0 along with oMLX v 0.4.4 (on Mac Mini M4 with 24GB Ram).

Regularly I start conversations without troubles for multiple iterations of chat, tool call etc.
At a certain moment I got errors dependent if I activate compression or not:

  • Error: Context length exceeded (8,273 tokens). Cannot compress further.
  • [Error: Request aborted: process memory limit exceeded (usage 22.3 GB, ceiling 17.0 GB). Reduce context size or lower memory_guard_tier.
  • Error: Context overflow and auto-compaction is disabled (compression.enabled: false). Run /compress to compact manually, /new to start fresh, or switch to a larger-context model.

Of course it might be in relation with hermes-agent but there is a reason why I think oMLX is involved.

The situation facing at the moment the error appears is always similar to following picture.

Image
  • The used memory temporarily exceeds the hard limit.
  • The above picture shows the memory within the range between soft and hard limit but there are also situations where the situation is constantly showing memory usage higher than the hard limit.
  • There is enough room in the SSD cache I gave.
  • The memory is never be released automatically

If I chat in omlx I get following or similar message

Error: {"error":{"message":"oMLX prefill memory guard rejected this prompt: Prefill context too large for available memory (preflight safety guard, kv_len=9, min_chunk=10): predicted peak would require ~18.54 GB (current 18.53 GB + KV 640.00 KB + min-chunk transient 1.02 MB) but prefill safety cap is 18.52 GB (95% of effective ceiling 19.50 GB). Reduce context length, free system memory, or loosen memory_guard_tier (safe → balanced → aggressive). To continue, set Memory Guard to aggressive, raise the custom memory guard ceiling, free system memory, or compact/reduce context.","type":"invalid_request_error","param":null,"code":"prefill_memory_exceeded","omlx_code":"prefill_memory_exceeded","estimated_bytes":19901816936,"limit_bytes":19891067289},"type":"error"}

Only way out is to restart the server.

What I think sould happen is:

  1. Full usage of SSD cache
  2. Release of memory after hitting the hard limit

I experience this for a longer period also with older versions. A few releases had already provided a better result but it might also be a necessary feature required.

If there are also some tipps for settings in omlx or even for hermes-agent I highly appreciate all support.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions