New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tag nodes with virtual memory footprints #857

Open
effigies opened this Issue Nov 29, 2017 · 3 comments

Comments

Projects
None yet
2 participants
@effigies
Collaborator

effigies commented Nov 29, 2017

Right now we're tagging high-memory usage nodes with their maximum demands for resident memory, which is appropriate for systems that permit the kernel to overcommit memory. However, in strict overcommit mode, the kernel counts all virtual memory against its total limits.

For non-strict mode, using resident memory requirements will allow us to better utilize a system's resources, while for strict mode, using virtual memory requirements will prevent us from crashing. So it seems sensible to allow both options, which will adjust the tagging.

I suspect we'll want a function or an object to handle the memory options. If we make some rough assumptions of VM/RSS ratios, we can simply keep the numbers we have in the case of non-strict, and apply a scaling factor for strict. e.g.

MEMORY_MODE = 'strict'
VM_RSS_RATIO = 2

def scale_mem(rss, vm=None):
    if MEMORY_MODE == 'nonstrict':
        return rss
    if vm is not None:
        return vm
    return rss * VM_RSS_RATIO

# Profiled RSS only
node = pe.Node(Interface(), mem_gb=scale_mem(3))

# Profiled RSS AND VM
node = pe.Node(Interface(), mem_gb=scale_mem(3, 5))
@oesteban

This comment has been minimized.

Contributor

oesteban commented Nov 29, 2017

We can check for /proc/sys/vm/overcommit_memory < 2 to see if overcommit is allowed at all (default would be yes, in case the file can't be read). And a default RSS_VM_RATIO (I switched the order, so ratios will be <1.0 in that case) can be read from /proc/sys/vm/overcommit_ratio. A default of 0.5 makes sense.

@oesteban

This comment has been minimized.

Contributor

oesteban commented Nov 29, 2017

Actually, this is probably something we may want to integrate directly into nipype, provide nipype config options etc.

@oesteban

This comment has been minimized.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment