artifacts_folder FileNotFoundError in combination with pytest-parallel

[geiregjo]

A FileNotFoundError is thrown during the test cleanup phase. This error has no impact on the result. I tracked the issue down to the tempfile.TemporaryDirectory artifacts_folder. The TemporyDirectory is explicitly cleaned with a cleanup() call. It is also implicitly cleaned up when the object is garbage-collected or during interpreter shutdown.

I'm not sure how pytest-parallel works. I think at some point the process is forked. This creates multiple TemporaryDirectory instances and thus multiple clean up attempts? In Python 3.10 these errors can easily be ignored using the TemporaryDirectory ignore_cleanup_errors kwarg. Not sure how it should be handled in Python 3.7.

Successful Test Traceback:

Traceback (most recent call last):
  File "/usr/lib/python3.7/weakref.py", line 648, in _exitfunc
    f()
  File "/usr/lib/python3.7/weakref.py", line 572, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/usr/lib/python3.7/tempfile.py", line 936, in _cleanup
    _rmtree(name)
  File "/usr/lib/python3.7/shutil.py", line 485, in rmtree
    onerror(os.lstat, path, sys.exc_info())
  File "/usr/lib/python3.7/shutil.py", line 483, in rmtree
    orig_st = os.lstat(path)
FileNotFoundError: [Errno 2] No such file or directory: '/tmp/playwright-pytest-7rzu1l9o'

[mxschmitt]

Should be fixed by #97

[geiregjo]

We've been running the tests in Ubuntu for a couple of months. We decided to start running them on Windows because of numerous reasons. And we noticed that we have the same issue. And it's fixed the same way as suggested in my first post. Fix #97 does not capture the permission issue.

___________________________________ ERROR at teardown of test_upload_file[ActionOrigin.TOOLBAR-temp_dirs0] ____________________________________ 

path = 'C:\\Users\\ubuntu\\AppData\\Local\\Temp\\playwright-pytest-vs4qcn13'
onerror = <function TemporaryDirectory._rmtree.<locals>.onerror at 0x00000257D0DE1FC0>

    def _rmtree_unsafe(path, onerror):
        try:
            with os.scandir(path) as scandir_it:
                entries = list(scandir_it)
        except OSError:
            onerror(os.scandir, path, sys.exc_info())
            entries = []
        for entry in entries:
            fullname = entry.path
            if _rmtree_isdir(entry):
                try:
                    if entry.is_symlink():
                        # This can only happen if someone replaces
                        # a directory with a symlink after the call to
                        # os.scandir or entry.is_dir above.
                        raise OSError("Cannot call rmtree on a symbolic link")
                except OSError:
                    onerror(os.path.islink, fullname, sys.exc_info())
                    continue
                _rmtree_unsafe(fullname, onerror)
            else:
                try:
>                   os.unlink(fullname)
E                   PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\u
buntu\\AppData\\Local\\Temp\\playwright-pytest-vs4qcn13\\83cde3cf4414efd29848e7ff566f2d6e.webm'

C:\Python310\lib\shutil.py:615: PermissionError

During handling of the above exception, another exception occurred:

func = <built-in function unlink>
path = 'C:\\Users\\ubuntu\\AppData\\Local\\Temp\\playwright-pytest-vs4qcn13\\83cde3cf4414efd29848e7ff566f2d6e.webm'
exc_info = (<class 'PermissionError'>, PermissionError(13, 'The process cannot access the file because it is being used by another process'), <t
raceback object at 0x00000257D07B1200>)

    def onerror(func, path, exc_info):
        if issubclass(exc_info[0], PermissionError):
            def resetperms(path):
                try:
                    _os.chflags(path, 0)
                except AttributeError:
                    pass
                _os.chmod(path, 0o700)

            try:
                if path != name:
                    resetperms(_os.path.dirname(path))
                resetperms(path)

                try:
>                   _os.unlink(path)
E                   PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\u
buntu\\AppData\\Local\\Temp\\playwright-pytest-vs4qcn13\\83cde3cf4414efd29848e7ff566f2d6e.webm'

C:\Python310\lib\tempfile.py:820: PermissionError

During handling of the above exception, another exception occurred:

launch_browser = <function launch_browser.<locals>.launch at 0x00000257D042AC20>

    @pytest.fixture(scope="session")
    def browser(launch_browser: Callable[[], Browser]) -> Generator[Browser, None, None]:
        browser = launch_browser()
        yield browser
        browser.close()
>       artifacts_folder.cleanup()

C:\Python310\lib\site-packages\pytest_playwright\pytest_playwright.py:186:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
C:\Python310\lib\tempfile.py:850: in cleanup
    self._rmtree(self.name, ignore_errors=self._ignore_cleanup_errors)
C:\Python310\lib\tempfile.py:832: in _rmtree
    _shutil.rmtree(name, onerror=onerror)
C:\Python310\lib\shutil.py:747: in rmtree
    return _rmtree_unsafe(path, onerror)
C:\Python310\lib\shutil.py:617: in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
C:\Python310\lib\tempfile.py:823: in onerror
    cls._rmtree(path, ignore_errors=ignore_errors)
C:\Python310\lib\tempfile.py:832: in _rmtree
    _shutil.rmtree(name, onerror=onerror) 
C:\Python310\lib\shutil.py:747: in rmtree
    return _rmtree_unsafe(path, onerror)
C:\Python310\lib\shutil.py:598: in _rmtree_unsafe
    onerror(os.scandir, path, sys.exc_info())
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
 
path = 'C:\\Users\\ubuntu\\AppData\\Local\\Temp\\playwright-pytest-vs4qcn13\\83cde3cf4414efd29848e7ff566f2d6e.webm'
onerror = <function TemporaryDirectory._rmtree.<locals>.onerror at 0x00000257D10CDAB0>

    def _rmtree_unsafe(path, onerror):
        try:
>           with os.scandir(path) as scandir_it:
E           NotADirectoryError: [WinError 267] The directory name is invalid: 'C:\\Users\\ubuntu\\AppData\\Local\\Temp\\playwright-pytest-vs4qcn
13\\83cde3cf4414efd29848e7ff566f2d6e.webm'

C:\Python310\lib\shutil.py:595: NotADirectoryError 
----------------------------------------------------

[geiregjo]

Can this issue be reopened?

[mxschmitt]

Can this issue be reopened?

Can you file a separate issue and reference to this one? With a repro etc.

[geiregjo]

I'll take another look. I worked around the issue by fixing it in a fork. But this was only temporary. I also stopped using pytest-parallel and was surprised that I'm still experiencing this issue... Maybe the cleanup starts before the process writing the .webm file releases the file?