LocalLlmMemorySizing
Local LLM Memory Sizing is a technique for deciding, before downloading a model, whether a given large language model will actually run well on a specific machine or whether it will grind to a halt swapping to disk. The method is simple arithmetic: take the quantized model's file size, subtract what the operating system needs to keep running (roughly 6 to 8 GB on macOS), reserve additional memory for context window usage (8 to 16 GB depending on how long the conversations will be), and check that what remains fits comfortably within the machine's available unified memory -- ideally leaving 10 to 15% of memory free rather than running right up against the limit.
The technique was documented in a write-up of one person's local inference setup on an M4 Pro Mac mini with 48 GB of RAM, connected to a phone and laptop over a private mesh network so all devices could share the same local inference server. That setup ran two differently sized models side by side: a large reasoning-focused model using a mixture-of-experts architecture (only a fraction of its parameters activate per token, keeping actual compute cost down despite its large total size), and a small model for lightweight chat and formatting tasks. Because swapping a model in or out only requires downloading a new file and restarting the inference server, the sizing check becomes a fast way to try new models as they are released without committing to a lengthy trial-and-error process.
The broader value of the technique is that it lets local inference cover the bulk of everyday requests -- the write-up cited roughly 80% -- while still keeping cloud API access available for demanding tasks, trading some capability for cost predictability and keeping data on-device.
This technique is placed in Assess. It is a reusable, generalizable method for evaluating local model feasibility, not tied to any single tool, but it has not yet been applied first-person to size a local model deployment in this wiki owner's own setup.
