-
-
Notifications
You must be signed in to change notification settings - Fork 41
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
"Stream is closed" error when creating dask client in Python 2 #25
Comments
My guess is that this is due to mismatched versions between the scheduler and client. Can you verify that your client process has the same version of distributed that is in the environment that you're giving to the |
I just checked the distributed versions on client and environment sides and they appear to be the same pip freeze | grep distributed
distributed==1.23.3 As a matter of fact I made the anaconda environment tar.gz file few days ago so it's fresh. I also checked the tornado and skein versions and they are also the same. |
Update. I tried the same code under Python 3 and It works great. The scenario I tried initially was as follows: I installed Anaconda2 (python 2), created environment for YARN from this anaconda, called the client from anaconda's python 2 and got the error I described above. Repeating the same steps but with Anaconda 3 doens't produce any errors. So the problem is with Python 2. |
Ah, that's it then. This has nothing to do with
An environment created with |
Well, I think the environment was packed correctly in some sense. I just discovered that anaconda2 creates environment with Python 3 being the default version even though Python 2 seems to be more logical default version created from anaconda 2. |
Glad to hear you figured things out. |
Is it possible to install dask-yarn on python2? |
I'm trying to test dask-yarn installation on my Hadoop cluster. I started with simple example:
the YARN application starts successfully, I have 1 scheduler and 4 workers. One thing is I have the following error in the scheduler's container log
(zzz is just scheduler's host IP); and the output log in the python shell:
Once I'm getting to the dask client:
I receive the stream closed error
Interestingly, that
<Future finished exception=KeyError('op',)>
appears again in the scheduler log every time I'm trying to create the client.The environment:
pip freeaze
):The text was updated successfully, but these errors were encountered: