Repository navigation
v0.3.0
Apple Silicon: GridCore runs real models on a Mac, through llama.cpp's
Metal backend. Measured on an M4 Pro 24 GB with macOS 15.8 and llama.cpp
b11146 (Homebrew). And a dashboard at /admin/ui. On the RTX 4060 Ti the
v0.2 load tests (mixed, thrash, ambient) give the same results as v0.2.0.
Added
gpu.device: apple(andautoon macOS): the device total is Metal's
working-set limit (llama-server --list-devices, else
iogpu.wired_limit_mb, else two thirds of RAM); memory in use comes
fromioreg, a model's size fromfootprint(all the RAM its process
takes, CPU-side tensors included), memory pressure and swap from
sysctl. No cgo, no new dependencies.gridcore checknames the device
and where its limit came from.- Estimates on unified memory count the host side too (embeddings, a tied
model's input copy oftoken_embd, weights left on the CPU) with Metal's
overhead instead of CUDA's: -2..+4 % against measured on Gemma 4 12B, E4B
and E2B.models --explainsays that-ngl/--cpu-moesave no memory
there. - Memory pressure: at "warn" background and batch work may not load
models (family requests take a resident variant), at "critical" cold
models are unloaded one every 5 s, ignoringmin_residency; a minute of
no lower-class loads follows each such unload. Interactive work is never
held.gridcore_memory_pressure_level, pressure and swap in
/admin/stateandgridcore status. - Device out of memory while serving: llama.cpp on an overcommitted Metal
device keeps answering/healthand fails every request, neighbours
included. Running instances' OOM errors are counted
(gridcore_device_oom_total); the model loaded within the last minute is
unloaded with its measurement discarded, and background loads wait 30 s. examples/apple-24gb.yaml,deploy/gridcore.plist(launchd agent).- Dashboard at
/admin/ui(/redirects there): the memory budget split
by model, memory / queue / utilisation charts over ten minutes, resident
models, running and queued jobs with why they wait, and every scheduler
event since the page was opened (the daemon keeps only its last 100).
One page compiled into the binary that polls/admin/state: no external
resources (a CSP forbids them), read-only.
Changed
- llama.cpp launch defaults on unified memory, each only if the binary
knows the flag and the model'sargsdo not set it:--load-mode none
(or--no-mmapon older builds),--cache-ram 0,--ctx-checkpoints 0.
Each of them otherwise grows a process after it was measured: the RAM
prompt cache alone took a 12B from 8.8 to 18.2 GB in a dozen prompts. - Default
headroom_mbis 1024 on Apple Silicon (512 elsewhere). - On unified memory models load one at a time.
- While an interactive request is freeing memory for its model or waiting
for the load slot, background and batch work start no loads (all
devices). On a 24 GB Mac a starved background job kept slipping its load
in first: 64 loads in 3 minutes and no chat answered.
Fixed
- A load whose warmup ran out of memory is a failed (OOM) load even when
/healthalready answers. - Orphan reaping after a crash checks reused PIDs through
pswhere there
is no/proc(macOS); before, it could kill an unrelated process. gridcore benchmeasures on monitors that must be told the PID.