Windows Subsystem for Linux "Could not set timeout on TCP stream: [Errno 92] Protocol not available"

[jwilson8767]

distributed/comm/tcp.py gives the warning "Could not set timeout on TCP stream: [Errno 92] Protocol not available" when spawning workers using Windows Subsystem for Linux as it expects "TCP_USER_TIMEOUT" to work -- and it doesn't appear to in WSL. I have created an issue over here: microsoft/WSL#3873 as I think this is an OS type issue, but I just wanted to bring it up over here to discuss workarounds/proper documentation. Currently it appears (though I'm having trouble confirming conclusively) that this allows worker processes to be orphaned following exceptions, and while some get cleaned up with the message "distributed.comm.tcp - WARNING - Closing dangling stream in
" many others remain running after the program has terminated.

What can we do to mitigate this or how do we better document this limitation?

Host OS: Windows 10.0.17134.619
Container OS: Ubuntu 18.04.2 LTS

jwilson@M:~$ python3
Python 3.6.7 (default, Oct 22 2018, 11:32:17)
[GCC 8.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import platform
>>> platform.version()
'#523-Microsoft Mon Dec 31 17:49:00 PST 2018'
>>> import sys
>>> sys.platform
'linux'

Thanks

[jwilson8767]

Could this be related to reusing sockets as py-amqp was over in this issue: celery/py-amqp#145?

[mrocklin]

Thanks for raising the issue @jwilson8767 .

I personally have no particular experience here and am unlikely to be able to help. I just wanted to make sure that your well-worded issue here was acknowledged and appreciated.

Perhaps @beberg has suggestions? (no worries if not)

[jwilson8767]

Just to follow up, this issue hasn't progressed on the WSL side. I'm hoping the recently announced WSL2 will offer some changes that will make multiprocessing as a whole a lot more efficient, but I'm unsure what changes to the network stack may help this particular issue.

[jurreht]

After encoutering this issue myself, I have added a workaround in #2802. This resolves the issues on WSL, as can be verified by running the test suite on a WSL setup. (This is not available in Travis, correct?)

[JSKenyon]

#2802 has been closed for now, but it would be particularly awesome if the distributed scheduler could work under WSL. Unfortunately, I lack the expertise to be of much assistance. Did you manage to find a workaround @jwilson8767?

[jwilson8767]

No, because I couldn't use dask reliably in my development environment I haven't used it at all.

I do still have hope that WSL2 will not have the same issues at the networking level as WSL1.

[StianOvrevage]

What is the status of this?

I'm getting

distributed.comm.tcp - WARNING - Could not set timeout on TCP stream: [Errno 92] Protocol not available

on dask 2.9.2 and it does not appear to be working.

[jwilson8767]

@StianOvrevage It hasn't been touched. I'm waiting for WSL2 to be fully released before I even attempt to use Dask on it as their networking stack appears to be both more robust and more complex which has been causing some people issues migrating tools/workflows from WSL1 to WSL2.

[StianOvrevage]

Ok, thanks for the update! Looking forward to WSL2 then :)

[aidanmontare-edu]

WSL 2 is available for some people:
https://docs.microsoft.com/en-us/windows/wsl/install-win10#update-to-wsl-2

I can't install it since I don't yet have the newest Windows 10 build, but has anyone tried this?

[dazzag24]

I have WSL2 and I just re-ran a notebook in which I had previously observed this problem.

Whereas previously with with WSL v1 I would observe this:

from dask.distributed import Client
client = Client()
----------------
distributed.comm.tcp - WARNING - Could not set timeout on TCP stream: [Errno 92] Protocol not available
distributed.comm.tcp - WARNING - Could not set timeout on TCP stream: [Errno 92] Protocol not available
distributed.comm.tcp - WARNING - Could not set timeout on TCP stream: [Errno 92] Protocol not available
distributed.comm.tcp - WARNING - Could not set timeout on TCP stream: [Errno 92] Protocol not available
distributed.comm.tcp - WARNING - Could not set timeout on TCP stream: [Errno 92] Protocol not available

with WSLv2 I now observe this:

from dask.distributed import Client
client = Client()
client
-----------------------------
Client    Scheduler: tcp://127.0.0.1:33239   Dashboard: http://127.0.0.1:8787/status 
Cluster    Workers: 4   Cores: 8   Memory: 13.22 GB

[aidanmontare-edu]

That's cool!

I'm considering upgrading to WSL 2, the only problem is that I store all my files in the Windows system, and the FAQ from Microsoft says that version 2 doesn't have great performance across file systems.