distributed can accumulate lots of old `worker-XXX` directories

[pitrou]

I've found that, either when running the test suite or short experiments, distributed leaves lots of (empty) worker directories around in my work directory:

$ tree worker-*
worker-1nsxxt70
└── storage
worker-2zdjkg_k
└── storage
worker-4agvlevk
└── storage
worker-4jv0zy__
└── storage
worker-4mue6z3j
└── storage
[etc.]

It would be nice to be able to clean them up when possible.

[pitrou]

One possibility would be for worker directories to all reside in a top dask-worker-space directory, so that obsolete subdirectories there can be pruned easily. What do you think @mrocklin ?

[mrocklin]

That sounds reasonable for testing. Another approach would be to try to discover the original cause of why they're being left around.

[pitrou]

I presume a crash or SIGKILL can prevent the worker from cleaning up?

[mrocklin]

Yes, that's probably it. I suppose there isn't much we can do there.

[pitrou]

The locket library implements a portable file-based lock which is automatically released when the holding process is killed. So we could use that to find out which directories are stale.

One thing we may want to be careful about is to avoid permission errors, i.e. only try to clean up directories which belong to the current user?

[jakirkham]

Have seen this as well. Are directories being registered for removal with atexit or similar?

[mrocklin]

They are, but if the process is SIGTERMed then it can stick around. This is resolved by #1543 . This PR is pretty much ready to go in, but could use some detailed review if you're interested @jakirkham . I know that this isn't the usual part of Dask that you are engaged in, but might be an interesting introduction.

[jakirkham]

Guessing this can be closed?