Multiple Future submissions return status of cancelled

[ameetshah1983]

We are submitting multiple futures / tasks from a given worker using worker_client context manager. The function makes use of client.submit to submit list of futures.

We break down the execution into batches and after each batch run is complete, we run client.cancel to do a cleanup and then submit the next batch. Time and again, entire batch of futures (lot of 95) immediately return status of cancelled.

We checked the worker logs and not much there and also the scheduler logs. Also tried to enable scheduler logs timestamp by adding the following lines to yaml config but did not help.

distributed:
  admin:
    log-format: '%(name)s - %(levelname)s - %(message)s'

We are using DASK - 0.17.2 and Distributed - 1.21.6 running on Windows 2018R2 servers. Also the we have 128 worker processes scattered over 16 nodes and each batch size for execution is around 95 futures at a time.

Can you please help with what we can look into which could be causing this.

[martindurant]

It would be better if you showed us a minimal version of the code you are running - but generally, there should not be a need for you to explicitly cancel your futures.

[ameetshah1983]

We cancel the futures after each batch to cleanup any memory utilization immediately. We can avoid that and then try but the submissions of next batch is after the cancel of each batch. We are submitting close to 4000 tasks overall but only 1 or 2 batches face this issue and again its random.

[martindurant]

Again, it would be much easier for us to diagnose if we could see the code. I could make a guess, but that would probably be premature.

[ameetshah1983]

Really sorry, in this case unable to provide code as it has business logic. However this does not occur for the same set of inputs and is very random as close to 3800 - 3600 runs work just fine running the same piece of code.
Was trying to debug it by enabling the timestamp in scheduler logs to see if there was any worker restart or client shutdown etc at around the time we received the issue, but the config settings did not work. Also not much data being passed on through the scheduler either.
But if there are any other reasons that can cause this, can look into this.

[ameetshah1983]

The code is mainly running the following scipy optimization -
scipy.minimize, quadratic function minimization, COBYLA solver, bunch of constraints (non linear), max_iter:1e7

However the cancels are received quite quickly and we dont think we even reach the optimization part.

[martindurant]

Without some form of the code, we are in the dark.
My best guess is, that some of the function inputs are similar enough that you happen to get key clashes, so the new result find the previously cancelled future. You could try investigating the keys of your futures.

[ameetshah1983]

Thank you Martin. We will try with pure=False and see if that helps. We did check and out of list of inputs atleast one of them should be different but makes sense what you said, and thinking pure=False on submit calls should help right in that case.
Also we will try and get some code snippet out soon like you mentioned which better helps define the problem.

[ameetshah1983]

We tried pure=False and also removed the cancellation of the dask futures after each batch. Instead we now use a for loop to delete each future after batch completion to test if that helps.

We still see the cancellations after these changes.

[martindurant]

use a for loop to delete each future

So what really is the problem if you don't do this? Futures should clean themselves up when they go out of scope and release memory without ever having to be cancelled/deleted explicitly.

[ameetshah1983]

Reason we were cleaning up the futures is that the futures show up in the scheduler status page when viewed using the console on port 8787. They tend to pile up till the client is closed as we use the worker_client with context manager. So say if we are running 300 tasks which execute function test_func, if we dont delete or cancel the future, the status page show 300 runs, while with deletion, only the given batch of 100 are displayed and then dropped.

Only reason we are doing this is to help release memory or unwanted objects sooner so that prev batch run has no effect on any future batches that will be executed. If this is not recommended, we remove that and test it out.

[martindurant]

If you can demonstrate that futures are not cleaned up upon exiting the context of worker_client, than that would be a bug, and give us something concrete to work on.

[ameetshah1983]

We use the same worker_client client for all the batches. As the total number of tasks can be much greater than number of worker processes, we break them in batches and all batches execute using the same client connection and not creating new one for each batch, thus we were performing the delete / cancel. Sorry for the confusion on that one.

[martindurant]

This raises the question of why you are creating all these batches rather than simply executing the work in your given batch.

If you can demonstrate that futures are not cleaned up upon exiting the context of worker_client

PLEASE, please try to show some code. I don't believe this conversation is heading anywhere otherwise.

[mrocklin]

Closing for now. @ameetshah1983 if you're able to help provide a minimal example then please let us know and we'll reopen. There isn't much we can do until then.

See https://matthewrocklin.com/blog/work/2018/02/28/minimal-bug-reports