Bug: tmpdir fixture in pytest 3.7.4

[mmrahn]

Hi,

there seems to be an issue with the tmpdir fixture in pytest 3.7.4. I submitted it as a pytest issue but they claimed it was an eventlet bug (see pytest-dev/pytest#3951).

When running

def test_foo(tmpdir):
    pass

I get the error

test setup failed
self = <_pytest.tmpdir.TempdirFactory object at 0x0000023C49299B70>

    def getbasetemp(self):
        """ return base temporary directory. """
        try:
>           return self._basetemp
E           AttributeError: 'TempdirFactory' object has no attribute '_basetemp'

..\..\..\Miniconda3\envs\io\lib\site-packages\_pytest\tmpdir.py:49: AttributeError

During handling of the above exception, another exception occurred:

fd = <eventlet.greenio.py3.GreenFileIO object at 0x0000023C493C1B70>

    def set_nonblocking(fd):
        """
        Sets the descriptor to be nonblocking.  Works on many file-like
        objects as well as sockets.  Only sockets can be nonblocking on
        Windows, however.
        """
        try:
>           setblocking = fd.setblocking
E           AttributeError: 'GreenFileIO' object has no attribute 'setblocking'

..\..\..\Miniconda3\envs\io\lib\site-packages\eventlet\greenio\base.py:86: AttributeError

During handling of the above exception, another exception occurred:

fd = <eventlet.greenio.py3.GreenFileIO object at 0x0000023C493C1B70>

    def set_nonblocking(fd):
        """
        Sets the descriptor to be nonblocking.  Works on many file-like
        objects as well as sockets.  Only sockets can be nonblocking on
        Windows, however.
        """
        try:
            setblocking = fd.setblocking
        except AttributeError:
            # fd has no setblocking() method. It could be that this version of
            # Python predates socket.setblocking(). In that case, we can still set
            # the flag "by hand" on the underlying OS fileno using the fcntl
            # module.
            try:
>               import fcntl
E               ModuleNotFoundError: No module named 'fcntl'

..\..\..\Miniconda3\envs\io\lib\site-packages\eventlet\greenio\base.py:93: ModuleNotFoundError

During handling of the above exception, another exception occurred:

fd = 8, args = ('w',), kw = {}

    def fdopen(fd, *args, **kw):
        """fdopen(fd [, mode='r' [, bufsize]]) -> file_object
    
        Return an open file object connected to a file descriptor."""
        if not isinstance(fd, int):
            raise TypeError('fd should be int, not %r' % fd)
        try:
>           return greenio.GreenPipe(fd, *args, **kw)

..\..\..\Miniconda3\envs\io\lib\site-packages\eventlet\green\os.py:28: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
..\..\..\Miniconda3\envs\io\lib\site-packages\eventlet\greenio\py3.py:212: in GreenPipe
    return _open(name, mode, buffering, encoding, errors, newline, closefd, opener)
..\..\..\Miniconda3\envs\io\lib\_pyio.py:210: in open
    closefd, opener=opener)
..\..\..\Miniconda3\envs\io\lib\site-packages\eventlet\greenio\py3.py:49: in __init__
    set_nonblocking(self)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

fd = <eventlet.greenio.py3.GreenFileIO object at 0x0000023C493C1B70>

    def set_nonblocking(fd):
        """
        Sets the descriptor to be nonblocking.  Works on many file-like
        objects as well as sockets.  Only sockets can be nonblocking on
        Windows, however.
        """
        try:
            setblocking = fd.setblocking
        except AttributeError:
            # fd has no setblocking() method. It could be that this version of
            # Python predates socket.setblocking(). In that case, we can still set
            # the flag "by hand" on the underlying OS fileno using the fcntl
            # module.
            try:
                import fcntl
            except ImportError:
                # Whoops, Windows has no fcntl module. This might not be a socket
                # at all, but rather a file-like object with no setblocking()
                # method. In particular, on Windows, pipes don't support
                # non-blocking I/O and therefore don't have that method. Which
                # means fcntl wouldn't help even if we could load it.
>               raise NotImplementedError("set_nonblocking() on a file object "
                                          "with no setblocking() method "
                                          "(Windows pipes don't support non-blocking I/O)")
E               NotImplementedError: set_nonblocking() on a file object with no setblocking() method (Windows pipes don't support non-blocking I/O)

..\..\..\Miniconda3\envs\io\lib\site-packages\eventlet\greenio\base.py:100: NotImplementedError
                                                   [100%]

The environment I'm using is

amqp                2.3.2
amqplib             1.0.2
anyjson             0.3.3
appdirs             1.4.3
asn1crypto          0.24.0
astral              1.6.1
astroid             2.0.4
callisto     2.1.2
atomicwrites        1.2.0
attrs               18.1.0
Automat             0.7.0
backcall            0.1.0
bcrypt              3.1.4
bleach              2.1.4
Cerberus            1.2
certifi             2018.8.24
cffi                1.11.5
chardet             3.0.4
click               6.7
climatecontrol      0.7.1
colorama            0.3.9
colorlover          0.2.1
constantly          15.1.0
coverage            4.5.1
cryptography        2.3.1
cufflinks           0.8.2
cycler              0.10.0
dataclasses         0.6
decorator           4.3.0
dnspython           1.15.0
entrypoints         0.2.3
et-xmlfile          1.0.1
eventlet            0.24.1
fancycompleter      0.8
filelock            3.0.6
flake8              3.5.0
flake8-docstrings   1.3.0
flake8-import-order 0.18
flake8-polyfill     1.0.2
flake8-quotes       1.0.0
future              0.16.0
geocoder            1.38.1
germanholidays      0.3
graypy              0.2.14
greenlet            0.4.14
html5lib            1.0.1
hyperlink           18.0.0
hypothesis          3.59.1
idna                2.7
incremental         17.5.0
influxdb            5.2.0
ipdb                0.11
ipykernel           4.8.2
ipython             6.5.0
ipython-genutils    0.2.0
isort               4.3.4
jdcal               1.4
jedi                0.12.1
Jinja2              2.10
jsonschema          2.6.0
jupyter-client      5.2.3
jupyter-core        4.4.0
kiwisolver          1.0.1
kombu               4.2.1
lazy-object-proxy   1.3.1
line-profiler       2.1.2
MarkupSafe          1.0
matplotlib          2.2.3
mccabe              0.6.1
memory-profiler     0.54.0
mistune             0.8.3
mkl-fft             1.0.4
mkl-random          1.0.1
mock                2.0.0
monotonic           1.5
more-itertools      4.3.0
mypy                0.620
nameko              2.11.0
nbconvert           5.3.1
nbformat            4.4.0
notebook            5.6.0
numexpr             2.6.8
numpy               1.15.1
openpyxl            2.5.6
pandas              0.23.4
pandocfilters       1.4.2
paramiko            2.4.1
parso               0.3.1
path.py             11.0.1
patsy               0.5.0
pbr                 4.2.0
pendulum            1.5.1
pickleshare         0.7.4
Pint                0.8.1
pip                 18.0
plotly              2.7.0
pluggy              0.7.1
prometheus-client   0.3.1
prompt-toolkit      1.0.15
psutil              5.4.7
py                  1.6.0
pyasn1              0.4.4
pyasn1-modules      0.2.2
pycodestyle         2.3.1
pycparser           2.18
pydocstyle          2.1.1
pyflakes            1.6.0
Pygments            2.2.0
pylint              2.1.1
PyNaCl              1.2.1
pyOpenSSL           18.0.0
pyparsing           2.2.0
pyreadline          2.1
pytest              3.7.4
pytest-cov          2.5.1
pytest-mock         1.10.0
pytest-runner       4.2
python-dateutil     2.7.3
pytz                2018.5
pytzdata            2018.5
pywin32             223
pywinpty            0.5.4
PyYAML              3.13
pyzmq               17.1.2
ratelim             0.1.6
requests            2.19.1
retrying            1.3.3
scipy               1.1.0
Send2Trash          1.5.0
service-identity    17.0.0
setuptools          40.2.0
simplegeneric       0.8.1
sip                 4.19.12
six                 1.11.0
snowballstemmer     1.2.1
statsmodels         0.9.0
tables              3.4.4
terminado           0.8.1
testpath            0.3.1
toml                0.9.4
tornado             5.1
traitlets           4.3.2
Twisted             18.7.0
typed-ast           1.1.0
typing              3.6.6
tzlocal             1.5.1
urllib3             1.23
vine                1.1.4
wcwidth             0.1.7
webencodings        0.5.1
Werkzeug            0.14.1
wheel               0.31.1
wincertstore        0.2
wmctrl              0.3
wrapt               1.10.11
xarray              0.10.8
xdg                 3.0.2
xlrd                1.1.0
XlsxWriter          1.0.8
zope.interface      4.5.0

Operating system is Windows 10. This bug does not happen up to pytest == 3.2.4 and py == 1.4.34.

[temoto]

It is Eventlet bug all right.

Please show full test module code. In what you provided def test_foo(tmpdir) I don't see a reason to call any eventlet code.

[mmrahn]

I can't give you the full module code, but minimal example showing the bug is:

conda create -n eventlet-test python=3.7 pytest=3.7 py=1.6
conda activate eventlet-test
pip install eventlet

test.py:

import eventlet 
eventlet.monkey_patch()

def test_foo(tmpdir):
    pass

pytest test.py

[temoto]

That's full module code, thanks.

[alessandroRoaro]

Hi, I'm having the same issue, with the exact same monkey-patching. Did you find a solution for this?

[temoto]

There is workaround, somewhat ugly, but also providing more test isolation, which is always helpful. Also it will break pytest error values introspection (expanding expressions in assert).

def test_foo(tmpdir):
    run_python("real_test_code.py")  # eventlet.monkey_patch() inside there

See run_python() function code here

eventlet/tests/__init__.py

Line 324 in 3a01b33

def run_python(path, env=None, args=None, timeout=None, pythonpath_extend=None, expect_pass=False):

For anyone willing to take on this issue:

• find why pytest tmpdir opens pipe in the first place
• handling Windows here https://github.com/eventlet/eventlet/blob/master/eventlet/greenio/base.py#L95 may be the best solution
• make GreenPipe skip set_nonblocking under Windows could be a working crutch
• new GreenPipe implementation using socketpair could be the best solution too