sockets still blocked for tests that use bokeh server when exception is specified in command line

[orionlee]

For tests that involve creating a bokeh server, if I pass the command line options:

• --allow-unix-socket and / or,
• --allow-hosts=127.0.0.1,localhost,::1
  the test still failed complaining that the sockets are blocked.

A workaround: if I mark individual tests with @pytest.mark.enable_socket, then the test will correctly pass.

---

Example test:

import pytest
import bokeh
from bokeh.io import show, output_notebook, push_notebook
from bokeh.layouts import layout, Spacer
from bokeh.models.widgets import Button, Div


def test_bokeh_server():
    def create_interact_ui(doc):
        widgets_and_figures = layout([Div(text="test bokeh server")])
        doc.add_root(widgets_and_figures)

    output_notebook(verbose=False, hide_banner=True)
    show(create_interact_ui)

If I run the command

> pytest tests/test_bokeh.py  --disable-socket --allow-unix-socket --allow-hosts=127.0.0.1,localhost,::1

The test will fail with the following exception:

tests\test_bokeh.py:16: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
c:\anaconda3\envs\lkv2_dev\lib\site-packages\bokeh\io\showing.py:137: in show
    return run_notebook_hook(state.notebook_type, 'app', obj, state, notebook_url, **kw)
c:\anaconda3\envs\lkv2_dev\lib\site-packages\bokeh\io\notebook.py:298: in run_notebook_hook
    return _HOOKS[notebook_type][action](*args, **kw)
c:\anaconda3\envs\lkv2_dev\lib\site-packages\bokeh\io\notebook.py:467: in show_app
    loop = IOLoop.current()
c:\anaconda3\envs\lkv2_dev\lib\site-packages\tornado\ioloop.py:263: in current
    loop = asyncio.get_event_loop()
c:\anaconda3\envs\lkv2_dev\lib\asyncio\events.py:640: in get_event_loop
    self.set_event_loop(self.new_event_loop())
c:\anaconda3\envs\lkv2_dev\lib\asyncio\events.py:660: in new_event_loop
    return self._loop_factory()
c:\anaconda3\envs\lkv2_dev\lib\asyncio\selector_events.py:60: in __init__
    self._make_self_pipe()
c:\anaconda3\envs\lkv2_dev\lib\asyncio\selector_events.py:107: in _make_self_pipe
    self._ssock, self._csock = socket.socketpair()
c:\anaconda3\envs\lkv2_dev\lib\socket.py:514: in socketpair
    lsock = socket(family, type, proto)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

cls = <class 'pytest_socket.disable_socket.<locals>.GuardedSocket'>, args = (<AddressFamily.AF_INET: 2>, <SocketKind.SOCK_STREAM: 1>, 0)
kwargs = {}, is_unix_socket = False

    def __new__(cls, *args, **kwargs):
        try:
            is_unix_socket = args[0] == socket.AF_UNIX
        except AttributeError:
            # AF_UNIX not supported on Windows https://bugs.python.org/issue33408
            is_unix_socket = False
    
        if is_unix_socket and allow_unix_socket:
            return super().__new__(cls, *args, **kwargs)
    
>       raise SocketBlockedError()
E       pytest_socket.SocketBlockedError: A test tried to use socket.socket.

---

pytest: 0.4.0
bokeh: 2.2.3
Platform + Python: Windows + Python 3.7. Also seen the same error (in non isolated setup) in Linux + Python 3.6, and OSX + Python 3.8 CI environments.

[miketheman]

Hi @orionlee !

Reading the stack trace, it looks like bokeh is using asyncio under the hood. Preliminary support for asyncio was added in #63 - and we specifically don't have full support for Windows yet, since Windows doesn't use UNIX sockets, and we excluded the async tests from the Windows test suite.

I haven't had time to fully check out the provided test case and reproduce yet, but will try to get to it soon!

[orionlee]

I had some suspicion that the problem could be Windows-specific, but am not sure.

The isolated test supplied here was based on an actual test that had more moving parts. For the actual test, it failed with the same error on Windows, Linux and OSX CI environments. (I did not get to run the isolated test in CI, however.)

So while it was plausible that the original actual test failed in CI for some other reasons, it could also be an indicator that the problem is not Windows-specific. I just don't know.

E.g., in this test run on linux with python 3.6. The test was started by

poetry run pytest --disable-socket --allow-unix-socket

Tests that need bokeh server still failed. The same SocketBlockedError was raised when bokeh's show() function was called. See line 216 of the log for an example.

[miketheman]

Hi @orionlee !

I have been reworking some of the dependency logic in the plugin, and have used your example to confirm that the changes should now work for your example.

in bokeh.py:

import pytest

import bokeh
from bokeh.io import show, output_notebook, push_notebook
from bokeh.layouts import layout, Spacer
from bokeh.models.widgets import Button, Div


def test_bokeh_server():
    def create_interact_ui(doc):
        widgets_and_figures = layout([Div(text="test bokeh server")])
        doc.add_root(widgets_and_figures)

    output_notebook(verbose=False, hide_banner=True)
    show(create_interact_ui)

$ pytest bokeh.py --disable-socket --allow-unix-socket --allow-hosts=127.0.0.1,localhost,::1
...
====== 1 passed ... =======

I'll be looking to release 0.50.0 with the changes soon! Closing this, but if it's still an issue with 0.50.0 (or you can try the main branch right now), please reopen!