Mac Available Memory might be slightly inaccurate

[kgibm]

psutil's virtual_memory on Mac calculates avail = inactive + free. free comes from psutil_virtual_mem and is calculated as (vm_statistics_data_t.free_count - vm_statistics_data_t.speculative_count) * PAGE_SIZE. This means that avail = (vm_statistics_data_t.inactive_count + vm_statistics_data_t.free_count - vm_statistics_data_t.speculative_count) * PAGE_SIZE. However, I can't find any evidence that speculative_count is effectively unavailable memory. An Apple engineer defines speculative_count as:

on Mac OS X 10.5 we introduced a new, fifth category of memory, speculative memory, used to hold pages that have been read from disk speculatively

The way I interpret this is that the effective amount of free RAM is free - speculative; but, speculative, just by its definition, is probably something that is effectively available for program demands (just like inactive). So I think psutil is calculating free correctly, but avail incorrectly and avail should add back in speculative.

[giampaolo]

I'm not a mac user but in general available should report the same value as the system task manager (the GUI one) or "top" cmdline utility at the very least. Perhaps you can check that out?

[kgibm]

Activity Monitor does not have an available statistic:

top also does not:

$ top -l 1; vm_stat
Processes: 334 total, 3 running, 331 sleeping, 1296 threads                                                                                                                                                                                                          09:07:10
Load Avg: 3.83, 3.33, 2.71  CPU usage: 17.78% user, 15.22% sys, 66.99% idle  SharedLibs: 213M resident, 58M data, 26M linkedit. MemRegions: 40154 total, 3917M resident, 206M private, 1387M shared. PhysMem: 11G used (2130M wired), 5313M unused.
VM: 1469G vsize, 1114M framework vsize, 0(0) swapins, 0(0) swapouts. Networks: packets: 415308/560M in, 119176/11M out. Disks: 1808179/25G read, 285028/5119M written.
[...]
Mach Virtual Memory Statistics: (page size of 4096 bytes)
Pages free:                                6018.
Pages active:                           1744278.
Pages inactive:                          822029.
Pages speculative:                      1082146.
Pages throttled:                              0.
Pages wired down:                        539463.
Pages purgeable:                           6720.
"Translation faults":                  15928515.
Pages copy-on-write:                     947314.
Pages zero filled:                      7114084.
Pages reactivated:                        15064.
Pages purged:                             44681.
File-backed pages:                      2061989.
Anonymous pages:                        1586464.
Pages stored in compressor:                   0.
Pages occupied by compressor:                 0.
Decompressions:                               0.
Compressions:                                 0.
Pageins:                                3667459.
Pageouts:                                   170.
Swapins:                                      0.
Swapouts:                                     0.

[giampaolo]

available mem should be 16 - 7.93.

[kgibm]

I'll close this issue out since I'm also not really a Mac guy and I'm not totally sure of all the details, but for posterity, I'll just assert, that given my reading, and if we want to match the Linux behavior (i.e. subtracting parts of RAM that are available for program demands), then I believe the correct behavior is the equivalent from vm_stat of Pages free + Pages inactive + Pages speculative + File-backed pages.

[giampaolo]

Please let's keep this open.

[giampaolo]

OK I think I fixed this in 2f9dcf3. It turns out also used memory was not properly calculated.

[kgibm]

@giampaolo That's a very unintuitive Mac API - it seems like the conclusion is that speculative is included in free, but then reported separately, thus why you're not adding speculative to avail and instead subtracting speculative from free.

[giampaolo]

The point here is that we have no authoritative source establishing how available memory should be calculated. As such I decided to assume two authoritative sources:

• for available memory we do what zabbix does (avail = inactive + free)
• for free memory we do what "free" cmdline utility does (free = free - speculative)

I would say the most authoritative source should be the Activity Monitor GUI tool but AFAIK there's no source code for it and it's difficult to guess what it really does under the hoods.

[kgibm]

@giampaolo I agree, and thanks for your work on this. I wonder whether speculative should be added to avail, but like you point out maybe that's implicitly in there.

[xyzzy-plugh]

Something seems amiss here; the info from activity monitor doesn't really support the 83.9% coming back from NCPA. What other info can I provide?

macOS 14.4.1, M2 CPU, NCPA 3.0.2

https://xxxxxxxxxxxx:5693/api/memory/virtual?units=Gi :
{ "virtual": { "available": [ 2.57, "GiB" ], "total": [ 16, "GiB" ], "percent": [ 83.9, "%" ], "free": [ 1.63, "GiB" ], "used": [ 1.3, "GiB" ] } }