Eventlet broke pathlib on Python 3.7 when patches os

[orsinium]

Script to reproduce:

import eventlet
eventlet.monkey_patch(os=True)
from pathlib import Path
Path('.').open()

# Python 3.x: IsADirectoryError: [Errno 21] Is a directory: '.'
# ^ it's ok
# Python 3.7.1: TypeError: open: path should be string, bytes or os.PathLike, not _NormalAccessor
# ^ it's not ok

As you can see, Path.open doesn't work on Python 3.7. It's because pathlib uses _NormalAccessor instance, that stores functions from os as attributes:

class _NormalAccessor(_Accessor):
    open = os.open

Python <3.7 wraps this functions in staticmethod like this:

class _NormalAccessor(_Accessor):
    open = staticmethod(os.open)

However Python 3.7 doesn't. Python doesn't bound os.open to _NormalAccessor because os.open isn't a function. But when eventlet patches os.open by the custom python function then interpreter bound this function (eventlet.green.os.open) to _NormalAccessor.

I can't report this bug to Python because I can't register on bugs.python.org >.<

Possible solution: patch _NormalAccessor too. But don't forget: in Python 3.6 passed argument has to converted to string:

class _NormalAccessor(_Accessor):

    def _wrap_strfunc(strfunc):
        @functools.wraps(strfunc)
        def wrapped(pathobj, *args):
            return strfunc(str(pathobj), *args)
        return staticmethod(wrapped)

    def _wrap_binary_strfunc(strfunc):
        @functools.wraps(strfunc)
        def wrapped(pathobjA, pathobjB, *args):
            return strfunc(str(pathobjA), str(pathobjB), *args)
        return staticmethod(wrapped)

    open = _wrap_strfunc(os.open)

References:

• Python 3.6 pathlib source
• Python 3.7 pathlib source

[nazariydolfin]

Here is my traceback:

  File "/app/backend/app/core/portfolio_factor_risk/risk.py", line 9, in <module>
    from sklearn.linear_model import LassoLarsCV, LassoCV
  File "/usr/local/lib/python3.7/site-packages/sklearn/linear_model/__init__.py", line 12, in <module>
    from .base import LinearRegression
  File "/usr/local/lib/python3.7/site-packages/sklearn/linear_model/base.py", line 38, in <module>
    from ..preprocessing.data import normalize as f_normalize
  File "/usr/local/lib/python3.7/site-packages/sklearn/preprocessing/__init__.py", line 6, in <module>
    from ._function_transformer import FunctionTransformer
  File "/usr/local/lib/python3.7/site-packages/sklearn/preprocessing/_function_transformer.py", line 5, in <module>
    from ..utils.testing import assert_allclose_dense_sparse
  File "/usr/local/lib/python3.7/site-packages/sklearn/utils/testing.py", line 751, in <module>
    import pytest
  File "/usr/local/lib/python3.7/site-packages/pytest.py", line 7, in <module>
    from _pytest.assertion import register_assert_rewrite
  File "/usr/local/lib/python3.7/site-packages/_pytest/assertion/__init__.py", line 12, in <module>
    from _pytest.assertion import rewrite
  File "/usr/local/lib/python3.7/site-packages/_pytest/assertion/rewrite.py", line 23, in <module>
    from _pytest.assertion import util
  File "/usr/local/lib/python3.7/site-packages/_pytest/assertion/util.py", line 10, in <module>
    import _pytest._code
  File "/usr/local/lib/python3.7/site-packages/_pytest/_code/__init__.py", line 6, in <module>
    from .code import Code  # noqa
  File "/usr/local/lib/python3.7/site-packages/_pytest/_code/code.py", line 15, in <module>
    import pluggy
  File "/usr/local/lib/python3.7/site-packages/pluggy/__init__.py", line 16, in <module>
    from .manager import PluginManager, PluginValidationError
  File "/usr/local/lib/python3.7/site-packages/pluggy/manager.py", line 6, in <module>
    import importlib_metadata
  File "/usr/local/lib/python3.7/site-packages/importlib_metadata/__init__.py", line 466, in <module>
    __version__ = version(__name__)
  File "/usr/local/lib/python3.7/site-packages/importlib_metadata/__init__.py", line 433, in version
    return distribution(package).version
  File "/usr/local/lib/python3.7/site-packages/importlib_metadata/__init__.py", line 222, in version
    return self.metadata['Version']
  File "/usr/local/lib/python3.7/site-packages/importlib_metadata/__init__.py", line 210, in metadata
    self.read_text('METADATA')
  File "/usr/local/lib/python3.7/site-packages/importlib_metadata/__init__.py", line 393, in read_text
    return self._path.joinpath(filename).read_text(encoding='utf-8')
  File "/usr/local/lib/python3.7/pathlib.py", line 1199, in read_text
    with self.open(mode='r', encoding=encoding, errors=errors) as f:
  File "/usr/local/lib/python3.7/pathlib.py", line 1186, in open
    opener=self._opener)
  File "/usr/local/lib/python3.7/pathlib.py", line 1039, in _opener
    return self._accessor.open(self, flags, mode)
  File "/usr/local/lib/python3.7/site-packages/eventlet/green/os.py", line 107, in open
    fd = __original_open__(file, flags, mode, dir_fd=dir_fd)
TypeError: open: path should be string, bytes or os.PathLike, not _NormalAccessor

[nazariydolfin]

fixed my issue by disabling eventlet patching os: eventlet.monkey_patch(os=False)

[orsinium]

Of course it doesn't appear if you don't patch os. However, sometimes you have no choice. For example, nameko patches os inside and you can't configure it.