Join GitHub today
Tag nodes with virtual memory footprints #857
Right now we're tagging high-memory usage nodes with their maximum demands for resident memory, which is appropriate for systems that permit the kernel to overcommit memory. However, in strict overcommit mode, the kernel counts all virtual memory against its total limits.
For non-strict mode, using resident memory requirements will allow us to better utilize a system's resources, while for strict mode, using virtual memory requirements will prevent us from crashing. So it seems sensible to allow both options, which will adjust the tagging.
I suspect we'll want a function or an object to handle the memory options. If we make some rough assumptions of VM/RSS ratios, we can simply keep the numbers we have in the case of non-strict, and apply a scaling factor for strict. e.g.
MEMORY_MODE = 'strict' VM_RSS_RATIO = 2 def scale_mem(rss, vm=None): if MEMORY_MODE == 'nonstrict': return rss if vm is not None: return vm return rss * VM_RSS_RATIO # Profiled RSS only node = pe.Node(Interface(), mem_gb=scale_mem(3)) # Profiled RSS AND VM node = pe.Node(Interface(), mem_gb=scale_mem(3, 5))
We can check for