Enforce resource requirements with c-groups?

[mrocklin]

Currently the dask-worker process runs a nanny that then creates a new process. We currently track things like memory use and kill the process if it exceeds that limit. In systems that support it, we might also consider using linux c-groups. Should we? Or should we say that that is orthogonal?

[jacobtomlinson]

I'm curious how things behave today.

If the nanny is started within a control group does it still have visibility of the full resources of the system? Or just the allocation within its group?

If it sees the whole system then it is likely the process will be OOM killed before the nanny decides to handle its own memory. So we may want to support this in that case.

[mrocklin]

I think that @jcrist may have modified things at one point to respect cgroups limits with regards to auto-inspection

[mrocklin]

To be clear, what I'm suggesting is that the nanny actually invokes cgroups machinery itself

[jacobtomlinson]

Oh interesting! I hadn't considered that.

What use cases do you see benefit from using this? The only thing I can think of is when multiple users share a single linux system with no other containment tools like Docker, or a batch scheduler like SLURM/PBS/etc.

[jakirkham]

Yeah I think this is a good idea.

To Jacob's point, this often gets setup on sys admins on shared cluster resources for processes. One thing that could be interesting would be if the Nanny could inspect and see how cgroups have been set for it and use a more constrained cgroup for the Worker itself. This would be handy in that the Nanny should then not killed by the cluster, but the Worker (if misbehaving) would be (well before the Nanny would have been killed). Idk how difficult this would be to do

[guillaumeeb]

Hi everyone,

My 2 cents: I would advise against getting into c-groups into Dask, and leave it up to resources scheduler (e.g. Kubernetes or job queuing systems).

C-groups is really hard to get right, and even buggy in some old kernels (although now even with RedHat based distrib it seems it has stabilized). It took several years for PBS software to have a reliable implementation of c-groups. I agree this could prevent in some case Nanny to be killed instead of workers as mentioned by @jakirkham, but I think this is not worth the effort, Linux Kernel usually kills the process that uses memory.

[MatthewLennie]

My one cent.
I am running on a slurm batch system (using sshcluster and slurmcluster for difference scenarios). The workers get killed by the batch system when you exceed the memory usage set by the cgroups. I think that @guillaumeeb is probably right about the various job queuing systems. Some smaller research institutes have small adhoc clusters without a queuing system or containerization which would be an example of what @jacobtomlinson mentioned, but that's probably a boundary case. Interesting idea though.