Work stealing issue

[rbubley]

I experienced a work-stealing issue with one task out of a batch of a couple of thousand sent to distributed. I don't believe there to be anything exceptional about this task. For this task, a summary of the logs is as follows:

11:36 - 12:06 : distributed.stealing: Moved task 45 times.
12:07         : stimulus task finished + unexpected worker completed task
12:15 - 12:18 : distributed.stealing: Moved task 3 times.
12:19         : stimulus task finished + unexpected worker completed task
12:20         : stimulus task finished + received already computed task
12:25 - 12:30 : Communication failed during replication (Repeated in batches of 10, about once every 0.2 seconds - so roughly 15,000 messages)
12:32         : Couldn’t gather keys
12:36         : Stimulus task finished

Any thoughts on what might be happening?

(distributed==1.19.3)

@mrocklin, do you think this may have been addressed (inter alia) by #1489?

[mrocklin]

Is this on latest release or on master?

…

On Fri, Nov 3, 2017 at 9:15 AM, rbubley ***@***.***> wrote: I experienced a work-stealing issue with one task out of a batch of a couple of thousand sent to distributed. I don't believe there to be anything exceptional about this task. For this task, a summary of the logs is as follows: 11:36 - 12:06 : distributed.stealing: Moved task 45 times. 12:07 : stimulus task finished + unexpected worked completed task 12:15 - 12:18 : distributed.stealing: Moved task 3 times. 12:19 : stimulus task finished + unexpected worked completed task 12:20 : stimulus task finished + received already computed task 12:25 - 12:30 : Communication failed during replication (Repeated in batches of 10, about once every 0.2 seconds - so roughly 15,000 messages) 12:32 : Couldn’t gather keys 12:36 : Stimulus task finished Any thoughts on what might be happening? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#1526>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AASszBIqdLjXMahUCwgNEi1_JWQ0yxefks5syxH4gaJpZM4QREdn> .

[rbubley]

Latest release.

[rbubley]

I presume that under master the task wouldn't be moved and calculated multiple times, which would be better than what I am currently seeing, but is that likely to be related to 15k messages on "communication failed during replication"?

[mrocklin]

It looks like you changed your original response from "master" to "latest release". Would you mind seeing if you can replicate it under master? If so, it would be useful to have more details, including logs if available.

[rbubley]

This no longer seems to be a problem in recent versions.