New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[3.4.3] Multiple tests in TestCase fail when not run in debug - shutil.make_archive #427

Closed
inkychris opened this Issue Sep 7, 2018 · 5 comments

Comments

Projects
None yet
2 participants
@inkychris

inkychris commented Sep 7, 2018

I'm not sure if this is a bug or an issue with my approach but the following minimal test example only appears to run both tests successfully if run with the debugger using PyCharm 2018.2.3 CE on Windows 10 (Python 3.7.0).

from pyfakefs.fake_filesystem_unittest import TestCase
import shutil


class Test(TestCase):
    def setUp(self):
        self.setUpPyfakefs()
        self.fs.create_file('src/file_1.txt')
        self.fs.create_file('src/a/file_2.txt')
        shutil.make_archive('archive', 'zip', root_dir='src')

    def test_a(self):
        self.assertTrue(True)

    def test_b(self):
        self.assertTrue(True)

Stack trace:

.E
======================================================================
ERROR: test_b (scratchpad.Test)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "C:\Dev\venv\lib\site-packages\pyfakefs\fake_filesystem.py", line 1876, in get_object_from_normpath
    target_object = target_object.get_entry(component)
  File "C:\Dev\venv\lib\site-packages\pyfakefs\fake_filesystem.py", line 617, in get_entry
    return self.contents[pathname_name]
KeyError: 'a'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Dev\venv\lib\site-packages\pyfakefs\fake_filesystem.py", line 1062, in stat
    entry_path, follow_symlinks, allow_fd=True)
  File "C:\Dev\venv\lib\site-packages\pyfakefs\fake_filesystem.py", line 1921, in resolve
    return self.get_object_from_normpath(self.resolve_path(file_path))
  File "C:\Dev\venv\lib\site-packages\pyfakefs\fake_filesystem.py", line 1878, in get_object_from_normpath
    self.raise_io_error(errno.ENOENT, file_path)
  File "C:\Dev\venv\lib\site-packages\pyfakefs\fake_filesystem.py", line 878, in raise_io_error
    raise IOError(errno, self._error_message(errno), filename)
FileNotFoundError: [Errno 2] No such file or directory in the fake filesystem: '\\a'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Dev\scratchpad.py", line 10, in setUp
    shutil.make_archive('archive', 'zip', root_dir='src')
  File "C:\Dev\venv\lib\shutil.py", line 813, in make_archive
    filename = func(base_name, base_dir, **kwargs)
  File "C:\Dev\venv\lib\shutil.py", line 705, in _make_zipfile
    zf.write(path, path)
  File "c:\dev\python37\Lib\zipfile.py", line 1688, in write
    zinfo = ZipInfo.from_file(filename, arcname)
  File "c:\dev\python37\Lib\zipfile.py", line 483, in from_file
    st = os.stat(filename)
  File "C:\Dev\venv\lib\site-packages\pyfakefs\fake_filesystem.py", line 3755, in stat
    return self.filesystem.stat(entry_path, follow_symlinks)
  File "C:\Dev\venv\lib\site-packages\pyfakefs\fake_filesystem.py", line 1070, in stat
    self.raise_os_error(io_error.errno, entry_path, winerror=winerror)
  File "C:\Dev\venv\lib\site-packages\pyfakefs\fake_filesystem.py", line 867, in raise_os_error
    raise OSError(errno, message, filename)
FileNotFoundError: [Errno 2] No such file or directory in the fake filesystem: 'a'

----------------------------------------------------------------------
Ran 2 tests in 0.016s

FAILED (errors=1)
@mrbean-bremen

This comment has been minimized.

Show comment
Hide comment
@mrbean-bremen

mrbean-bremen Sep 7, 2018

Collaborator

Hm, your code looks good to me. The only thing that I can think of are the relative paths - have you tried with absolute paths (e.g. c:/src) instead?
I will have a closer look tomorrow - thanks for the report!

Collaborator

mrbean-bremen commented Sep 7, 2018

Hm, your code looks good to me. The only thing that I can think of are the relative paths - have you tried with absolute paths (e.g. c:/src) instead?
I will have a closer look tomorrow - thanks for the report!

@inkychris

This comment has been minimized.

Show comment
Hide comment
@inkychris

inkychris Sep 8, 2018

Just tried with absolute paths but this didn't make a difference. Another interesting thing to note is that when run (for my particular use case) with other tests, it then works again, it's just in isolation that this fails. I'm just working on a minimal example to share with you.

inkychris commented Sep 8, 2018

Just tried with absolute paths but this didn't make a difference. Another interesting thing to note is that when run (for my particular use case) with other tests, it then works again, it's just in isolation that this fails. I'm just working on a minimal example to share with you.

@inkychris

This comment has been minimized.

Show comment
Hide comment
@inkychris

inkychris Sep 8, 2018

So another one of my tests imports zipfile. Adding this to the example above (even though I'm not using it directly) appears to fix the problem.

from pyfakefs.fake_filesystem_unittest import TestCase
import shutil, zipfile


class Test(TestCase):
    def setUp(self):
        self.setUpPyfakefs()
        self.fs.create_file('src/file_1.txt')
        self.fs.create_file('src/a/file_2.txt')
        shutil.make_archive('archive', 'zip', root_dir='src')

    def test_a(self):
        self.assertTrue(True)

    def test_b(self):
        self.assertTrue(True)

inkychris commented Sep 8, 2018

So another one of my tests imports zipfile. Adding this to the example above (even though I'm not using it directly) appears to fix the problem.

from pyfakefs.fake_filesystem_unittest import TestCase
import shutil, zipfile


class Test(TestCase):
    def setUp(self):
        self.setUpPyfakefs()
        self.fs.create_file('src/file_1.txt')
        self.fs.create_file('src/a/file_2.txt')
        shutil.make_archive('archive', 'zip', root_dir='src')

    def test_a(self):
        self.assertTrue(True)

    def test_b(self):
        self.assertTrue(True)
@mrbean-bremen

This comment has been minimized.

Show comment
Hide comment
@mrbean-bremen

mrbean-bremen Sep 8, 2018

Collaborator

Ok, I can confirm the bug, and it happens only on Python 3.7 - so my first guess is that the implementation of zipfile or shutil has changed that causes this. I will try to understand the cause.

Collaborator

mrbean-bremen commented Sep 8, 2018

Ok, I can confirm the bug, and it happens only on Python 3.7 - so my first guess is that the implementation of zipfile or shutil has changed that causes this. I will try to understand the cause.

@mrbean-bremen mrbean-bremen added the bug label Sep 8, 2018

@mrbean-bremen mrbean-bremen self-assigned this Sep 8, 2018

@mrbean-bremen

This comment has been minimized.

Show comment
Hide comment
@mrbean-bremen

mrbean-bremen Sep 8, 2018

Collaborator

Ok, this has to do with the fact that the zipfile module is imported in shutil inside a function to avoid cyclic dependencies. In the versions before 3.7, zipfile has been imported before (I didn't find out by whom, but that doesn't really matter), so pyfakefs has been able to stub out the os module in setUpPyfakefs. Your workaround to import zipfile in the test does the same, and you can use it safely for the time being.

As for the problem itself - I'm still not down to the root cause. If the module cannot be patched in setUpPyfakefs, it is patched later dynamically by another mechanism. This seems to work, but for some reason it does not get set back, so the time it is called in the second test, it still points to the fake filesystem instance from the first test. Will look into this another day...

Collaborator

mrbean-bremen commented Sep 8, 2018

Ok, this has to do with the fact that the zipfile module is imported in shutil inside a function to avoid cyclic dependencies. In the versions before 3.7, zipfile has been imported before (I didn't find out by whom, but that doesn't really matter), so pyfakefs has been able to stub out the os module in setUpPyfakefs. Your workaround to import zipfile in the test does the same, and you can use it safely for the time being.

As for the problem itself - I'm still not down to the root cause. If the module cannot be patched in setUpPyfakefs, it is patched later dynamically by another mechanism. This seems to work, but for some reason it does not get set back, so the time it is called in the second test, it still points to the fake filesystem instance from the first test. Will look into this another day...

mrbean-bremen added a commit to mrbean-bremen/pyfakefs that referenced this issue Sep 9, 2018

mrbean-bremen added a commit to mrbean-bremen/pyfakefs that referenced this issue Sep 9, 2018

@jmcgeheeiv jmcgeheeiv closed this in #428 Sep 9, 2018

jmcgeheeiv added a commit that referenced this issue Sep 9, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment