I use hermes agent 0.17.0 along with oMLX v 0.4.4 (on Mac Mini M4 with 24GB Ram).
Regularly I start conversations without troubles for multiple iterations of chat, tool call etc.
At a certain moment I got errors dependent if I activate compression or not:
- Error: Context length exceeded (8,273 tokens). Cannot compress further.
- [Error: Request aborted: process memory limit exceeded (usage 22.3 GB, ceiling 17.0 GB). Reduce context size or lower memory_guard_tier.
- Error: Context overflow and auto-compaction is disabled (compression.enabled: false). Run /compress to compact manually, /new to start fresh, or switch to a larger-context model.
Of course it might be in relation with hermes-agent but there is a reason why I think oMLX is involved.
The situation facing at the moment the error appears is always similar to following picture.
- The used memory temporarily exceeds the hard limit.
- The above picture shows the memory within the range between soft and hard limit but there are also situations where the situation is constantly showing memory usage higher than the hard limit.
- There is enough room in the SSD cache I gave.
- The memory is never be released automatically
If I chat in omlx I get following or similar message
Error: {"error":{"message":"oMLX prefill memory guard rejected this prompt: Prefill context too large for available memory (preflight safety guard, kv_len=9, min_chunk=10): predicted peak would require ~18.54 GB (current 18.53 GB + KV 640.00 KB + min-chunk transient 1.02 MB) but prefill safety cap is 18.52 GB (95% of effective ceiling 19.50 GB). Reduce context length, free system memory, or loosen memory_guard_tier (safe → balanced → aggressive). To continue, set Memory Guard to aggressive, raise the custom memory guard ceiling, free system memory, or compact/reduce context.","type":"invalid_request_error","param":null,"code":"prefill_memory_exceeded","omlx_code":"prefill_memory_exceeded","estimated_bytes":19901816936,"limit_bytes":19891067289},"type":"error"}
Only way out is to restart the server.
What I think sould happen is:
- Full usage of SSD cache
- Release of memory after hitting the hard limit
I experience this for a longer period also with older versions. A few releases had already provided a better result but it might also be a necessary feature required.
If there are also some tipps for settings in omlx or even for hermes-agent I highly appreciate all support.
I use hermes agent 0.17.0 along with oMLX v 0.4.4 (on Mac Mini M4 with 24GB Ram).
Regularly I start conversations without troubles for multiple iterations of chat, tool call etc.
At a certain moment I got errors dependent if I activate compression or not:
Of course it might be in relation with hermes-agent but there is a reason why I think oMLX is involved.
The situation facing at the moment the error appears is always similar to following picture.
If I chat in omlx I get following or similar message
Error: {"error":{"message":"oMLX prefill memory guard rejected this prompt: Prefill context too large for available memory (preflight safety guard, kv_len=9, min_chunk=10): predicted peak would require ~18.54 GB (current 18.53 GB + KV 640.00 KB + min-chunk transient 1.02 MB) but prefill safety cap is 18.52 GB (95% of effective ceiling 19.50 GB). Reduce context length, free system memory, or loosen memory_guard_tier (safe → balanced → aggressive). To continue, set Memory Guard to aggressive, raise the custom memory guard ceiling, free system memory, or compact/reduce context.","type":"invalid_request_error","param":null,"code":"prefill_memory_exceeded","omlx_code":"prefill_memory_exceeded","estimated_bytes":19901816936,"limit_bytes":19891067289},"type":"error"}
Only way out is to restart the server.
What I think sould happen is:
I experience this for a longer period also with older versions. A few releases had already provided a better result but it might also be a necessary feature required.
If there are also some tipps for settings in omlx or even for hermes-agent I highly appreciate all support.