Outstanding task on client but hub says completed

[drevicko]

Edit: Looks like there's a race condition where Client.outstanding is updated with a message id after the reply has already been received and processed, probalby here and here. See second comment below...

I was trying to purge_everything but got a RuntimeError: Can't purge outstanding tasks: ....

Here's an investigation of the offending task:

print(cli.outstanding)
print('6bee8602-ca584ac7aa23ca16738f43a0' in cli.history, '6bee8602-ca584ac7aa23ca16738f43a0' in cli.results)
print(cli.result_status('6bee8602-ca584ac7aa23ca16738f43a0'))
print(cli.get_result('6bee8602-ca584ac7aa23ca16738f43a0'))
print(cli.db_query({"msg_id":'6bee8602-ca584ac7aa23ca16738f43a0'}))

produces:

{'6bee8602-ca584ac7aa23ca16738f43a0'}
True False
{'pending': [], 'status': 'ok', 'completed': ['6bee8602-ca584ac7aa23ca16738f43a0']}
<AsyncHubResult: unknown>
[]

I'm happy to attempt more detailed debug info if someone can provide suggestions on how to do it.

[drevicko]

A little more info on this:

Earlier in execution (presumably during a lbview.map() call, where lbview is a load balanced view), I got this message with the offending msg_id:

got unknown result: 6bee8602-ca584ac7aa23ca16738f43a0

And since ignoring the above exception, I get this message regularly. The message is generated by Client._handle_execute_reply(self, msg).

This suggests a race condition where the message id is getting added to Client.outstanding after the reply is received. Looking at the Client source here and here and noting that Client.outstanding is an instance of traitlet.Set which isn't threadsafe largely confirms this.

(btw: I have ipyparallel version 6.0.2 installed from conda-forge.)

[drevicko]

fwiw, my conda install looks like this:

               platform : linux-64
          conda version : 4.3.29
       conda is private : False
      conda-env version : 4.3.29
    conda-build version : 1.18.2
         python version : 3.5.3.final.0
       requests version : 2.14.2
       root environment : /home/ianwoo/anaconda3  (writable)
    default environment : /home/ianwoo/anaconda3
       envs directories : /home/ianwoo/anaconda3/envs
                          /home/ianwoo/.conda/envs
          package cache : /home/ianwoo/anaconda3/pkgs
                          /home/ianwoo/.conda/pkgs
           channel URLs : https://conda.anaconda.org/menpo/linux-64
                          https://conda.anaconda.org/menpo/noarch
                          https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.continuum.io/pkgs/main/linux-64
                          https://repo.continuum.io/pkgs/main/noarch
                          https://repo.continuum.io/pkgs/free/linux-64
                          https://repo.continuum.io/pkgs/free/noarch
                          https://repo.continuum.io/pkgs/r/linux-64
                          https://repo.continuum.io/pkgs/r/noarch
                          https://repo.continuum.io/pkgs/pro/linux-64
                          https://repo.continuum.io/pkgs/pro/noarch
            config file : /home/ianwoo/.condarc
             netrc file : None
           offline mode : False
             user-agent : conda/4.3.29 requests/2.14.2 CPython/3.5.3 Linux/4.4.0-97-generic debian/stretch/sid glibc/2.23    
                UID:GID : 1011:1012

[minrk]

Thanks, I think you've tracked it down. I'll have to think carefully abouthow to fix it.

For debugging, you can set client.debug = True to log all messages as they go by (it's a lot!).

[ruiwang2uber]

Got the same error due to the same reason.

Is there any updates on this? or Can anyone please offer a workaround?

[FabioLuporini]

This might be the cause for the bug described here

basically, it happens in Continuous Integration + pytest + nbval + ipyparallel , apparently only on non-output cells. I think this is fairly relevant

[FabioLuporini]

also, silly question due to my poor knowledge of ipyparallel: what threads are competing for the same data structures ? I can easily see one thread dealing with socket I/O , how many others ? and where are they spawned?

[dalbabur]

i think i may be running into a similar issue, this time with broadcast view...

client.outstanding returns several tasks, but view.queue_status() shows all are complete

[minrk]

@dalbabur can you open a new Issue with more information about what you are doing? A reliable reproduction would be a huge help, too, if you can make one.