Skip to content

claude-cli-worker leaks spawned processes on LRU cell eviction #41

@adiled

Description

@adiled

Summary

When the worker's cell table is full and a new prompt arrives, it evicts the least-recently-used cell (worker.cell.evict.lru). The eviction removes the cell from the worker's tracking table but does not kill the already-spawned claude process. That process continues running — making API calls, consuming memory — indefinitely until it self-exits or the OS reclaims it.

In practice the process never self-exits because it is mid-bootstrap and waiting for an API response. With 30 bees all bootstrapping simultaneously (e.g. after a humd restart), the worker fills its cell table, evicts roughly 15-20 sessions, and those 15-20 orphaned claude processes keep running. On the next tick (75s later) the cycle repeats. Within an hour: 100+ orphaned claude processes, 4GB+ RAM consumed, system crawl.

Evidence

Worker log shows evictions with no corresponding kill:
```
worker.cell.evict.lru evicted_sid=sid-daman-follower-v1-5
worker.cell.evict.lru evicted_sid=sid-daman-operator
worker.cell.evict.lru evicted_sid=sid-daman-watchdog-v1-1
```

After each eviction the evicted pid continues appearing in ps aux. Observed accumulation across batches:

Start time Process count RSS
11:45AM 9 245MB
12:05PM 21 642MB
12:08PM 7 201MB
12:24PM 22 671MB
12:29PM 23 805MB
Total 99 ~3.5GB

All 99 were orphaned (worker had evicted their cells). Killing them manually restored the system.

Reproduction

  1. Run 30+ bees that all reconnect simultaneously (e.g. restart humd).
  2. Watch worker.cell.evict.lru events in the worker log.
  3. Note that the evicted PIDs remain in ps aux indefinitely.

Expected

When worker.cell.evict.lru fires, the worker should kill the child process associated with that cell before dropping it from the table.

Environment

  • humd 0.31.16 / thrum 0.7.0
  • claude-cli-worker (same release)
  • macOS aarch64
  • 30 forager bees

Metadata

Metadata

Assignees

No one assigned

    Labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions