Improve memory available detection

[asyd]

In

distributed/distributed/worker.py

Line 63 in 5ef9c80

TOTAL_MEMORY = psutil.virtual_memory().total

you use psutil to known the amount of memory available.

However, in the case of the worker is launched in a cgroup with a memory limitation (memory.limit_in_bytes) the value returned by psutil is the amount of the host, not the amount available in the cgroups, there fore the worker try to use more memory than it can and the linux kernel kill it.

[mrocklin]

You can solve this manually by telling dask workers to limit their memory usage with --memory-limit NBYTES

To make this more automatic is there a nice way to get the cgroups memory limit from python with a library like psutil?

[asyd]

I'm looking for it!

[mrocklin]

Any update here @asyd ?

[asyd]

Hi @mrocklin,

what do you think about asyd@e6f1883?

[mrocklin]

Interesting. Is this accurate for all linux distributions, or do some systems behave differently?

[asyd]

Since it's a Linux kernel feature it should work on every linux distributions.

However, while this workaround will work on my use case (worker runs inside a docker container) I need to improve it to check if the worker runs on root cgroup or a subone by checking /prod/self/cgroup but I wanted have feedbacks before