PermissionError raised, yet uncaught

[nicolashainaux]

Describe the bug
Trying to move a file to a directory without write permissions does raise a PermissionError (as expected) that cannot be caught (unexpected).

I'm puzzled about this, but cannot find out where I possibly could have wrongly written the tests, nor can I figure out how to get to catch the exception and why it does not get caught. I notice there's another exception raised "in the same time", that tells FileNotFoundError: [Errno 2] No such file or directory in the fake filesystem: PosixPath('/rootpath/dir2/f1.txt'), but this file has been created. I think it's maybe because it gets deleted at some point during the move() call, though I do not understand why it is deleted before the copy has been done.

How To Reproduce
Here are two tests failing exactly in the same way. fs is the regular pyfakefs's fs fixture.
The situation is the same in both tests, I only changed the test itself: in the first test I only directly try to catch the exception, in the second one, I use the pytest's raises() feature for that.
Both times, the exception is raised, as stated in the logs, but it is not caught.

def test_catch_permission_error01(fs):
    set_uid(1000)
    fs.create_dir('/rootpath/')
    dir1 = fs.create_dir('/rootpath/dir1')
    Path(dir1.path).chmod(0o555)  # remove write permissions
    fs.create_dir('/rootpath/dir2')
    fs.create_file('/rootpath/dir2/f1.txt')

    try:
        shutil.move(Path('/rootpath/dir2/f1.txt'),
                    Path('/rootpath/dir1/f1.txt'))  # PermissionError is raised, but not caught
    except PermissionError:
        print('PERMISSION ERROR')


def test_catch_permission_error02(fs):
    set_uid(1000)
    fs.create_dir('/rootpath/')
    dir1 = fs.create_dir('/rootpath/dir1')
    Path(dir1.path).chmod(0o555)  # remove write permissions
    fs.create_dir('/rootpath/dir2')
    fs.create_file('/rootpath/dir2/f1.txt')

    with pytest.raises(PermissionError) as excinfo:
        shutil.move(Path('/rootpath/dir2/f1.txt'),
                    Path('/rootpath/dir1/f1.txt'))  # PermissionError is raised, but not caught
    assert str(excinfo.value) \
        == "PermissionError: [Errno 13] Permission Denied: '/rootpath/dir1'"

Output of test_catch_permission_error01:

================================================================================== test session starts ==================================================================================
platform linux -- Python 3.8.6, pytest-5.4.3, py-1.10.0, pluggy-0.13.1 -- /home/nico/.local/dev/zano/.venv/py38/bin/python
cachedir: .pytest_cache
rootdir: /home/nico/.local/dev/zano
plugins: mock-3.6.1, pyfakefs-4.5.1
collected 1 item                                                                                                                                                                        

tests/04_tasks_test.py::test_catch_permission_error01 FAILED

======================================================================================= FAILURES ========================================================================================
_____________________________________________________________________________ test_catch_permission_error01 _____________________________________________________________________________

src = PosixPath('/rootpath/dir2/f1.txt'), dst = PosixPath('/rootpath/dir1/f1.txt'), copy_function = <function copy2 at 0x7f580ed268b0>

    def move(src, dst, copy_function=copy2):
        """Recursively move a file or directory to another location. This is
        similar to the Unix "mv" command. Return the file or directory's
        destination.
    
        If the destination is a directory or a symlink to a directory, the source
        is moved inside the directory. The destination path must not already
        exist.
    
        If the destination already exists but is not a directory, it may be
        overwritten depending on os.rename() semantics.
    
        If the destination is on our current filesystem, then rename() is used.
        Otherwise, src is copied to the destination and then removed. Symlinks are
        recreated under the new name if os.rename() fails because of cross
        filesystem renames.
    
        The optional `copy_function` argument is a callable that will be used
        to copy the source or it will be delegated to `copytree`.
        By default, copy2() is used, but any function that supports the same
        signature (like copy()) can be used.
    
        A lot more could be done here...  A look at a mv.c shows a lot of
        the issues this implementation glosses over.
    
        """
        sys.audit("shutil.move", src, dst)
        real_dst = dst
        if os.path.isdir(dst):
            if _samefile(src, dst):
                # We might be on a case insensitive filesystem,
                # perform the rename anyway.
                os.rename(src, dst)
                return
    
            real_dst = os.path.join(dst, _basename(src))
            if os.path.exists(real_dst):
                raise Error("Destination path '%s' already exists" % real_dst)
        try:
>           os.rename(src, real_dst)

/home/nico/.pyenv/versions/3.8.6/lib/python3.8/shutil.py:788: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pyfakefs.fake_filesystem.FakeOsModule object at 0x7f580da0edf0>, src = '/rootpath/dir2/f1.txt', dst = '/rootpath/dir1/f1.txt'

    def rename(self, src: AnyStr, dst: AnyStr, *,
               src_dir_fd: Optional[int] = None,
               dst_dir_fd: Optional[int] = None) -> None:
        """Rename a FakeFile object at old_file_path to new_file_path,
        preserving all properties.
        Also replaces existing new_file_path object, if one existed
        (Unix only).
    
        Args:
            src: Path to filesystem object to rename.
            dst: Path to where the filesystem object will live
                after this call.
            src_dir_fd: If not `None`, the file descriptor of a directory,
                with `src` being relative to this directory.
            dst_dir_fd: If not `None`, the file descriptor of a directory,
                with `dst` being relative to this directory.
    
        Raises:
            OSError: if old_file_path does not exist.
            OSError: if new_file_path is an existing directory.
            OSError: if new_file_path is an existing file (Windows only)
            OSError: if new_file_path is an existing file and could not
                be removed (Unix)
            OSError: if `dirname(new_file)` does not exist
            OSError: if the file would be moved to another filesystem
                (e.g. mount point)
        """
        src = self._path_with_dir_fd(src, self.rename, src_dir_fd)
        dst = self._path_with_dir_fd(dst, self.rename, dst_dir_fd)
>       self.filesystem.rename(src, dst)

/home/nico/.local/dev/zano/.venv/py38/lib/python3.8/site-packages/pyfakefs/fake_filesystem.py:4309: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pyfakefs.fake_filesystem.FakeFilesystem object at 0x7f580da90ee0>, old_file_path = '/rootpath/dir2/f1.txt', new_file_path = '/rootpath/dir1/f1.txt', force_replace = False

    def rename(self, old_file_path: AnyPath,
               new_file_path: AnyPath,
               force_replace: bool = False) -> None:
        """Renames a FakeFile object at old_file_path to new_file_path,
        preserving all properties.
    
        Args:
            old_file_path: Path to filesystem object to rename.
            new_file_path: Path to where the filesystem object will live
                after this call.
            force_replace: If set and destination is an existing file, it
                will be replaced even under Windows if the user has
                permissions, otherwise replacement happens under Unix only.
    
        Raises:
            OSError: if old_file_path does not exist.
            OSError: if new_file_path is an existing directory
                (Windows, or Posix if old_file_path points to a regular file)
            OSError: if old_file_path is a directory and new_file_path a file
            OSError: if new_file_path is an existing file and force_replace
                not set (Windows only).
            OSError: if new_file_path is an existing file and could not be
                removed (Posix, or Windows with force_replace set).
            OSError: if dirname(new_file_path) does not exist.
            OSError: if the file would be moved to another filesystem
                (e.g. mount point).
        """
        old_path = make_string_path(old_file_path)
        new_path = make_string_path(new_file_path)
        ends_with_sep = self.ends_with_path_separator(old_path)
        old_path = self.absnormpath(old_path)
        new_path = self.absnormpath(new_path)
        if not self.exists(old_path, check_link=True):
            self.raise_os_error(errno.ENOENT, old_path, 2)
        if ends_with_sep:
            self._handle_broken_link_with_trailing_sep(old_path)
    
        old_object = self.lresolve(old_path)
        if not self.is_windows_fs:
            self._handle_posix_dir_link_errors(
                new_path, old_path, ends_with_sep)
    
        if self.exists(new_path, check_link=True):
            renamed_path = self._rename_to_existing_path(
                force_replace, new_path, old_path,
                old_object, ends_with_sep)
    
            if renamed_path is None:
                return
            else:
                new_path = renamed_path
    
        old_dir, old_name = self.splitpath(old_path)
        new_dir, new_name = self.splitpath(new_path)
        if not self.exists(new_dir):
            self.raise_os_error(errno.ENOENT, new_dir)
        old_dir_object = self.resolve(old_dir)
        new_dir_object = self.resolve(new_dir)
        if old_dir_object.st_dev != new_dir_object.st_dev:
            self.raise_os_error(errno.EXDEV, old_path)
        if not S_ISDIR(new_dir_object.st_mode):
            self.raise_os_error(
                errno.EACCES if self.is_windows_fs else errno.ENOTDIR,
                new_path)
        if new_dir_object.has_parent_object(old_object):
            self.raise_os_error(errno.EINVAL, new_path)
    
        object_to_rename = old_dir_object.get_entry(old_name)
        old_dir_object.remove_entry(old_name, recursive=False)
        object_to_rename.name = new_name
        new_name = new_dir_object._normalized_entryname(new_name)
        if new_name in new_dir_object.entries:
            # in case of overwriting remove the old entry first
            new_dir_object.remove_entry(new_name)
>       new_dir_object.add_entry(object_to_rename)

/home/nico/.local/dev/zano/.venv/py38/lib/python3.8/site-packages/pyfakefs/fake_filesystem.py:2279: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pyfakefs.fake_filesystem.FakeDirectory object at 0x7f580da78ee0>, path_object = <pyfakefs.fake_filesystem.FakeFile object at 0x7f580e730cd0>

    def add_entry(self, path_object: FakeFile) -> None:
        """Adds a child FakeFile to this directory.
    
        Args:
            path_object: FakeFile instance to add as a child of this directory.
    
        Raises:
            OSError: if the directory has no write permission (Posix only)
            OSError: if the file or directory to be added already exists
        """
        if (not is_root() and not self.st_mode & PERM_WRITE and
                not self.filesystem.is_windows_fs):
>           raise OSError(errno.EACCES, 'Permission Denied', self.path)
E           PermissionError: [Errno 13] Permission Denied: '/rootpath/dir1'

/home/nico/.local/dev/zano/.venv/py38/lib/python3.8/site-packages/pyfakefs/fake_filesystem.py:698: PermissionError

During handling of the above exception, another exception occurred:

fs = <pyfakefs.fake_filesystem.FakeFilesystem object at 0x7f580da90ee0>

    def test_catch_permission_error01(fs):
        set_uid(1000)
        fs.create_dir('/rootpath/')
        dir1 = fs.create_dir('/rootpath/dir1')
        Path(dir1.path).chmod(0o555)  # remove write permissions
        fs.create_dir('/rootpath/dir2')
        fs.create_file('/rootpath/dir2/f1.txt')
    
        try:
>           shutil.move(Path('/rootpath/dir2/f1.txt'),
                        Path('/rootpath/dir1/f1.txt'))

/home/nico/.local/dev/zano/tests/04_tasks_test.py:590: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/home/nico/.pyenv/versions/3.8.6/lib/python3.8/shutil.py:802: in move
    copy_function(src, real_dst)
/home/nico/.pyenv/versions/3.8.6/lib/python3.8/shutil.py:432: in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
/home/nico/.pyenv/versions/3.8.6/lib/python3.8/shutil.py:261: in copyfile
    with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
/home/nico/.local/dev/zano/.venv/py38/lib/python3.8/site-packages/pyfakefs/fake_filesystem.py:4838: in open
    return fake_open(file, mode, buffering, encoding, errors,
/home/nico/.local/dev/zano/.venv/py38/lib/python3.8/site-packages/pyfakefs/fake_filesystem.py:5455: in __call__
    return self.call(*args, **kwargs)
/home/nico/.local/dev/zano/.venv/py38/lib/python3.8/site-packages/pyfakefs/fake_filesystem.py:5528: in call
    file_object = self._init_file_object(file_object,
/home/nico/.local/dev/zano/.venv/py38/lib/python3.8/site-packages/pyfakefs/fake_filesystem.py:5587: in _init_file_object
    self.filesystem.raise_os_error(errno.ENOENT, file_path)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pyfakefs.fake_filesystem.FakeFilesystem object at 0x7f580da90ee0>, err_no = 2, filename = PosixPath('/rootpath/dir2/f1.txt'), winerror = None

    def raise_os_error(self, err_no: int,
                       filename: Optional[AnyString] = None,
                       winerror: Optional[int] = None) -> NoReturn:
        """Raises OSError.
        The error message is constructed from the given error code and shall
        start with the error string issued in the real system.
        Note: this is not true under Windows if winerror is given - in this
        case a localized message specific to winerror will be shown in the
        real file system.
    
        Args:
            err_no: A numeric error code from the C variable errno.
            filename: The name of the affected file, if any.
            winerror: Windows only - the specific Windows error code.
        """
        message = self._error_message(err_no)
        if (winerror is not None and sys.platform == 'win32' and
                self.is_windows_fs):
            raise OSError(err_no, message, filename, winerror)
>       raise OSError(err_no, message, filename)
E       FileNotFoundError: [Errno 2] No such file or directory in the fake filesystem: PosixPath('/rootpath/dir2/f1.txt')

/home/nico/.local/dev/zano/.venv/py38/lib/python3.8/site-packages/pyfakefs/fake_filesystem.py:1057: FileNotFoundError
================================================================================ short test summary info ================================================================================
FAILED tests/04_tasks_test.py::test_catch_permission_error01 - FileNotFoundError: [Errno 2] No such file or directory in the fake filesystem: PosixPath('/rootpath/dir2/f1.txt')
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
=================================================================================== 1 failed in 0.58s ===================================================================================

Output of test_catch_permission_error02:

================================================================================== test session starts ==================================================================================
platform linux -- Python 3.8.6, pytest-5.4.3, py-1.10.0, pluggy-0.13.1 -- /home/nico/.local/dev/zano/.venv/py38/bin/python
cachedir: .pytest_cache
rootdir: /home/nico/.local/dev/zano
plugins: mock-3.6.1, pyfakefs-4.5.1
collected 1 item                                                                                                                                                                        

tests/04_tasks_test.py::test_catch_permission_error02 FAILED

======================================================================================= FAILURES ========================================================================================
_____________________________________________________________________________ test_catch_permission_error02 _____________________________________________________________________________

src = PosixPath('/rootpath/dir2/f1.txt'), dst = PosixPath('/rootpath/dir1/f1.txt'), copy_function = <function copy2 at 0x7f86dd23d8b0>

    def move(src, dst, copy_function=copy2):
        """Recursively move a file or directory to another location. This is
        similar to the Unix "mv" command. Return the file or directory's
        destination.
    
        If the destination is a directory or a symlink to a directory, the source
        is moved inside the directory. The destination path must not already
        exist.
    
        If the destination already exists but is not a directory, it may be
        overwritten depending on os.rename() semantics.
    
        If the destination is on our current filesystem, then rename() is used.
        Otherwise, src is copied to the destination and then removed. Symlinks are
        recreated under the new name if os.rename() fails because of cross
        filesystem renames.
    
        The optional `copy_function` argument is a callable that will be used
        to copy the source or it will be delegated to `copytree`.
        By default, copy2() is used, but any function that supports the same
        signature (like copy()) can be used.
    
        A lot more could be done here...  A look at a mv.c shows a lot of
        the issues this implementation glosses over.
    
        """
        sys.audit("shutil.move", src, dst)
        real_dst = dst
        if os.path.isdir(dst):
            if _samefile(src, dst):
                # We might be on a case insensitive filesystem,
                # perform the rename anyway.
                os.rename(src, dst)
                return
    
            real_dst = os.path.join(dst, _basename(src))
            if os.path.exists(real_dst):
                raise Error("Destination path '%s' already exists" % real_dst)
        try:
>           os.rename(src, real_dst)

/home/nico/.pyenv/versions/3.8.6/lib/python3.8/shutil.py:788: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pyfakefs.fake_filesystem.FakeOsModule object at 0x7f86dbf27e80>, src = '/rootpath/dir2/f1.txt', dst = '/rootpath/dir1/f1.txt'

    def rename(self, src: AnyStr, dst: AnyStr, *,
               src_dir_fd: Optional[int] = None,
               dst_dir_fd: Optional[int] = None) -> None:
        """Rename a FakeFile object at old_file_path to new_file_path,
        preserving all properties.
        Also replaces existing new_file_path object, if one existed
        (Unix only).
    
        Args:
            src: Path to filesystem object to rename.
            dst: Path to where the filesystem object will live
                after this call.
            src_dir_fd: If not `None`, the file descriptor of a directory,
                with `src` being relative to this directory.
            dst_dir_fd: If not `None`, the file descriptor of a directory,
                with `dst` being relative to this directory.
    
        Raises:
            OSError: if old_file_path does not exist.
            OSError: if new_file_path is an existing directory.
            OSError: if new_file_path is an existing file (Windows only)
            OSError: if new_file_path is an existing file and could not
                be removed (Unix)
            OSError: if `dirname(new_file)` does not exist
            OSError: if the file would be moved to another filesystem
                (e.g. mount point)
        """
        src = self._path_with_dir_fd(src, self.rename, src_dir_fd)
        dst = self._path_with_dir_fd(dst, self.rename, dst_dir_fd)
>       self.filesystem.rename(src, dst)

/home/nico/.local/dev/zano/.venv/py38/lib/python3.8/site-packages/pyfakefs/fake_filesystem.py:4309: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pyfakefs.fake_filesystem.FakeFilesystem object at 0x7f86dbfa7ee0>, old_file_path = '/rootpath/dir2/f1.txt', new_file_path = '/rootpath/dir1/f1.txt', force_replace = False

    def rename(self, old_file_path: AnyPath,
               new_file_path: AnyPath,
               force_replace: bool = False) -> None:
        """Renames a FakeFile object at old_file_path to new_file_path,
        preserving all properties.
    
        Args:
            old_file_path: Path to filesystem object to rename.
            new_file_path: Path to where the filesystem object will live
                after this call.
            force_replace: If set and destination is an existing file, it
                will be replaced even under Windows if the user has
                permissions, otherwise replacement happens under Unix only.
    
        Raises:
            OSError: if old_file_path does not exist.
            OSError: if new_file_path is an existing directory
                (Windows, or Posix if old_file_path points to a regular file)
            OSError: if old_file_path is a directory and new_file_path a file
            OSError: if new_file_path is an existing file and force_replace
                not set (Windows only).
            OSError: if new_file_path is an existing file and could not be
                removed (Posix, or Windows with force_replace set).
            OSError: if dirname(new_file_path) does not exist.
            OSError: if the file would be moved to another filesystem
                (e.g. mount point).
        """
        old_path = make_string_path(old_file_path)
        new_path = make_string_path(new_file_path)
        ends_with_sep = self.ends_with_path_separator(old_path)
        old_path = self.absnormpath(old_path)
        new_path = self.absnormpath(new_path)
        if not self.exists(old_path, check_link=True):
            self.raise_os_error(errno.ENOENT, old_path, 2)
        if ends_with_sep:
            self._handle_broken_link_with_trailing_sep(old_path)
    
        old_object = self.lresolve(old_path)
        if not self.is_windows_fs:
            self._handle_posix_dir_link_errors(
                new_path, old_path, ends_with_sep)
    
        if self.exists(new_path, check_link=True):
            renamed_path = self._rename_to_existing_path(
                force_replace, new_path, old_path,
                old_object, ends_with_sep)
    
            if renamed_path is None:
                return
            else:
                new_path = renamed_path
    
        old_dir, old_name = self.splitpath(old_path)
        new_dir, new_name = self.splitpath(new_path)
        if not self.exists(new_dir):
            self.raise_os_error(errno.ENOENT, new_dir)
        old_dir_object = self.resolve(old_dir)
        new_dir_object = self.resolve(new_dir)
        if old_dir_object.st_dev != new_dir_object.st_dev:
            self.raise_os_error(errno.EXDEV, old_path)
        if not S_ISDIR(new_dir_object.st_mode):
            self.raise_os_error(
                errno.EACCES if self.is_windows_fs else errno.ENOTDIR,
                new_path)
        if new_dir_object.has_parent_object(old_object):
            self.raise_os_error(errno.EINVAL, new_path)
    
        object_to_rename = old_dir_object.get_entry(old_name)
        old_dir_object.remove_entry(old_name, recursive=False)
        object_to_rename.name = new_name
        new_name = new_dir_object._normalized_entryname(new_name)
        if new_name in new_dir_object.entries:
            # in case of overwriting remove the old entry first
            new_dir_object.remove_entry(new_name)
>       new_dir_object.add_entry(object_to_rename)

/home/nico/.local/dev/zano/.venv/py38/lib/python3.8/site-packages/pyfakefs/fake_filesystem.py:2279: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pyfakefs.fake_filesystem.FakeDirectory object at 0x7f86dbf8fee0>, path_object = <pyfakefs.fake_filesystem.FakeFile object at 0x7f86dcc47cd0>

    def add_entry(self, path_object: FakeFile) -> None:
        """Adds a child FakeFile to this directory.
    
        Args:
            path_object: FakeFile instance to add as a child of this directory.
    
        Raises:
            OSError: if the directory has no write permission (Posix only)
            OSError: if the file or directory to be added already exists
        """
        if (not is_root() and not self.st_mode & PERM_WRITE and
                not self.filesystem.is_windows_fs):
>           raise OSError(errno.EACCES, 'Permission Denied', self.path)
E           PermissionError: [Errno 13] Permission Denied: '/rootpath/dir1'

/home/nico/.local/dev/zano/.venv/py38/lib/python3.8/site-packages/pyfakefs/fake_filesystem.py:698: PermissionError

During handling of the above exception, another exception occurred:

fs = <pyfakefs.fake_filesystem.FakeFilesystem object at 0x7f86dbfa7ee0>

    def test_catch_permission_error02(fs):
        set_uid(1000)
        fs.create_dir('/rootpath/')
        dir1 = fs.create_dir('/rootpath/dir1')
        Path(dir1.path).chmod(0o555)  # remove write permissions
        fs.create_dir('/rootpath/dir2')
        fs.create_file('/rootpath/dir2/f1.txt')
    
        with pytest.raises(PermissionError) as excinfo:
>           shutil.move(Path('/rootpath/dir2/f1.txt'),
                        Path('/rootpath/dir1/f1.txt'))

/home/nico/.local/dev/zano/tests/04_tasks_test.py:605: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
/home/nico/.pyenv/versions/3.8.6/lib/python3.8/shutil.py:802: in move
    copy_function(src, real_dst)
/home/nico/.pyenv/versions/3.8.6/lib/python3.8/shutil.py:432: in copy2
    copyfile(src, dst, follow_symlinks=follow_symlinks)
/home/nico/.pyenv/versions/3.8.6/lib/python3.8/shutil.py:261: in copyfile
    with open(src, 'rb') as fsrc, open(dst, 'wb') as fdst:
/home/nico/.local/dev/zano/.venv/py38/lib/python3.8/site-packages/pyfakefs/fake_filesystem.py:4838: in open
    return fake_open(file, mode, buffering, encoding, errors,
/home/nico/.local/dev/zano/.venv/py38/lib/python3.8/site-packages/pyfakefs/fake_filesystem.py:5455: in __call__
    return self.call(*args, **kwargs)
/home/nico/.local/dev/zano/.venv/py38/lib/python3.8/site-packages/pyfakefs/fake_filesystem.py:5528: in call
    file_object = self._init_file_object(file_object,
/home/nico/.local/dev/zano/.venv/py38/lib/python3.8/site-packages/pyfakefs/fake_filesystem.py:5587: in _init_file_object
    self.filesystem.raise_os_error(errno.ENOENT, file_path)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

self = <pyfakefs.fake_filesystem.FakeFilesystem object at 0x7f86dbfa7ee0>, err_no = 2, filename = PosixPath('/rootpath/dir2/f1.txt'), winerror = None

    def raise_os_error(self, err_no: int,
                       filename: Optional[AnyString] = None,
                       winerror: Optional[int] = None) -> NoReturn:
        """Raises OSError.
        The error message is constructed from the given error code and shall
        start with the error string issued in the real system.
        Note: this is not true under Windows if winerror is given - in this
        case a localized message specific to winerror will be shown in the
        real file system.
    
        Args:
            err_no: A numeric error code from the C variable errno.
            filename: The name of the affected file, if any.
            winerror: Windows only - the specific Windows error code.
        """
        message = self._error_message(err_no)
        if (winerror is not None and sys.platform == 'win32' and
                self.is_windows_fs):
            raise OSError(err_no, message, filename, winerror)
>       raise OSError(err_no, message, filename)
E       FileNotFoundError: [Errno 2] No such file or directory in the fake filesystem: PosixPath('/rootpath/dir2/f1.txt')

/home/nico/.local/dev/zano/.venv/py38/lib/python3.8/site-packages/pyfakefs/fake_filesystem.py:1057: FileNotFoundError
================================================================================ short test summary info ================================================================================
FAILED tests/04_tasks_test.py::test_catch_permission_error02 - FileNotFoundError: [Errno 2] No such file or directory in the fake filesystem: PosixPath('/rootpath/dir2/f1.txt')
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 1 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
=================================================================================== 1 failed in 0.58s ===================================================================================

Your environment

$ python -c "import platform; print(platform.platform())"
Linux-5.4.0-89-generic-x86_64-with-glibc2.29
$ python -c "import sys; print('Python', sys.version)"
Python 3.8.6 (default, Jan  9 2021, 22:33:39)
[GCC 9.3.0]
$ python -c "from pyfakefs.fake_filesystem import __version__; print('pyfakefs', __version__)"
pyfakefs 4.5.1

[nicolashainaux]

I've tried different "moves", only one works as expected: if one want to move a subdirectory of the one where permissions are insufficient (then the PermissionError is caught as expected). Otherwise: moving a file or a directory from "outside" to the 'forbidden' one, or renaming a file inside it, leads to the same errors as described in the first post.

On another hand, using copy2 and copytree does not create problems.

Here is a commented version of the second test, extended:

def test_catch_permission_error02(fs):
    set_uid(1000)
    fs.create_dir('/rootpath/')
    dir1 = fs.create_dir('/rootpath/dir1')
    fs.create_dir('/rootpath/dir1/dir3')  # one dir to rename inside (forbidden) dir1/
    fs.create_dir('/rootpath/dir2')
    fs.create_dir('/rootpath/dir2/dir5')  # one dir to move into (forbidden) dir1/
    fs.create_file('/rootpath/dir2/f1.txt')  # one file to move into (forbidden) dir1/

    Path(dir1.path).chmod(0o555)  # remove write permissions on dir1/

    # move directory inside 'dir1'
    with pytest.raises(PermissionError) as excinfo:
        shutil.move(Path('/rootpath/dir1/dir3'),
                    Path('/rootpath/dir1/dir4'))
    assert str(excinfo.value) \
        == "[Errno 13] Permission Denied: '/rootpath/dir1'"  # caught, as expected
    
    # copy2 a file into 'dir1'
    with pytest.raises(PermissionError) as excinfo:
        shutil.copy2(Path('/rootpath/dir2/f1.txt'),
                     Path('/rootpath/dir1/f1.copied2'))
    assert str(excinfo.value) \
        == "[Errno 13] Permission Denied: '/rootpath/dir1'"  # caught, as expected

    # copytree a directory into 'dir1'
    with pytest.raises(PermissionError) as excinfo:
        shutil.copytree(Path('/rootpath/dir2/dir5'),
                        Path('/rootpath/dir1/DIR5'))
    assert str(excinfo.value) \
        == "[Errno 13] Permission Denied: '/rootpath/dir1'"  # caught, as expected

    # move a directory into 'dir1'
    with pytest.raises(PermissionError) as excinfo:
        shutil.move(Path('/rootpath/dir2/dir5'),
                    Path('/rootpath/dir1/dir5'))
    assert str(excinfo.value) \
        == "[Errno 13] Permission Denied: '/rootpath/dir1'"  # NOT caught

    # move a file inside 'dir1'
    with pytest.raises(PermissionError) as excinfo:
        shutil.move(Path('/rootpath/dir1/f1.txt'),
                    Path('/rootpath/dir1/F1.txt'))
    assert str(excinfo.value) \
        == "[Errno 13] Permission Denied: '/rootpath/dir1'"  # NOT caught

    # move a file into 'dir1'
    with pytest.raises(PermissionError) as excinfo:
        shutil.move(Path('/rootpath/dir2/f1.txt'),
                    Path('/rootpath/dir1/f1.txt'))
    assert str(excinfo.value) \
        == "[Errno 13] Permission Denied: '/rootpath/dir1'"  # NOT caught

[mrbean-bremen]

I will have a closer look later, but have you checked how the real file system behaves in these cases?
From a cursory look I can see that the shutil.move code has its own exception handler around os.rename, so it may raise its own exception after it catches the PermissionError, which could explain the FileNotFound raised.
This is just a guess, but it would help to see the real fs behavior here.

[nicolashainaux]

Test number 04, using pathlib.Path and no fake file system, reproduces the 3 failing tests from test number 02 in real file system, but the exceptions are caught correctly, this time (so, this test passes successfully):

def test_catch_permission_error04():
    DATA = Path(__file__).parent / 'data'
    dir1 = DATA / 'dir1'
    dir3 = DATA / 'dir1/dir3'
    dir2 = DATA / 'dir2'
    dir5 = DATA / 'dir2/dir5'
    f1 = DATA / 'dir2/f1.txt'
    f2 = DATA / 'dir1/f2.txt'

    dir1.mkdir()
    dir3.mkdir()
    dir2.mkdir()
    dir5.mkdir()

    f1.write_text('')
    f2.write_text('')

    dest1 = DATA / 'dir1/f1.txt'
    dest2 = DATA / 'dir1/dir5'
    dest3 = DATA / 'dir1/F2.txt'

    dir1.chmod(0o555)

    with pytest.raises(PermissionError) as excinfo:
        shutil.move(f1, dest1)
    assert str(excinfo.value) \
        == f"[Errno 13] Permission denied: '{dest1}'"

    with pytest.raises(PermissionError) as excinfo:
        shutil.move(dir5, dest2)
    assert str(excinfo.value) \
        == f"[Errno 13] Permission denied: '{dest2}'"

    with pytest.raises(PermissionError) as excinfo:
        shutil.move(f2, dest3)
    assert str(excinfo.value) \
        == f"[Errno 13] Permission denied: '{dest3}'"

    dir1.chmod(0o777)
    f2.unlink()
    dir3.rmdir()
    dir1.rmdir()
    dir5.rmdir()
    f1.unlink()
    dir2.rmdir()

[mrbean-bremen]

Thank you, I guess in this case the permission error is raised inside the exception handler in another function, where pyfakefs raises FileNotFound instead - will have a look tonight. As I'm working mostly under Windows, it always helps to see the real output under Linux.

[nicolashainaux]

No problem! I hope I haven't missed anything obvious.
I can probably still provide more information tonight, but from tomorrow I'll be off for about a week, so I'll answer only later.
Thanks for your quick replies anyway!

[mrbean-bremen]

Ok, I think I found the problem: in the fake fs, renaming is done by removing the file from the old directory entry and adding it to the new one. If the latter fails due to a permission error, the file stays removed, e.g. it is effectively deleted.
The real fs behaves more consistent: if the rename fails, the whole rename is reverted and nothing is changed. What shutil.move does is first trying a rename, and if that fails, it tries a copy. This fails in the fake fs because the source has been removed during the rename, thus the FileNotFound exception.
This is clearly a bug in pyfakefs, that has not been found so far, thanks for reporting this!
It may take me some time to fix this properly - a naive fix that I just tried breaks other tests, and I have to do this with a fresh brain, maybe over the weekend. So you will (hopefully) get the fix after you return...

[nicolashainaux]

Happy if that leads to improve pyfakefs! And thank you for your work and quick answers, that's fantastic!

I do not want to urge you, there's anyway no hurry for me to get it fixed quickly (I think I can work around the bug making the move myself in two steps, copying to dest and then removing the source, as shutil.copy2 and shutil.copytree are not affected by the bug...). Thank you anyway!

[mrbean-bremen]

@nicolashainaux - please check if the master branch works for you now.

[nicolashainaux]

Yes, it works! Thanks a lot!
I close the issue, as it seems solved.

================================================================================== test session starts ==================================================================================
platform linux -- Python 3.8.6, pytest-5.4.3, py-1.10.0, pluggy-0.13.1 -- /home/nico/.local/dev/zano/.venv/py38/bin/python
cachedir: .pytest_cache
rootdir: /home/nico/.local/dev/zano
plugins: mock-3.6.1, pyfakefs-4.6.dev0
collected 31 items                                                                                                                                                                      

tests/04_tasks_test.py::test_retrieve_new_files_from_connected_replicas PASSED                                                                                                    [  3%]
tests/04_tasks_test.py::test_retrieve_new_files_from_connected_replicas_permission_errors PASSED                                                                                  [  6%]
tests/04_tasks_test.py::test_push_changes_to_replicas PASSED                                                                                                                      [  9%]
tests/04_tasks_test.py::test_sync_tree_structures_no_change PASSED                                                                                                                [ 12%]
tests/04_tasks_test.py::test_sync_tree_structures_new_dirs PASSED                                                                                                                 [ 16%]
tests/04_tasks_test.py::test_sync_tree_structures_renamed_dirs PASSED                                                                                                             [ 19%]
tests/04_tasks_test.py::test_sync_tree_structures_moved_dirs PASSED                                                                                                               [ 22%]
tests/04_tasks_test.py::test_sync_tree_structures_deleted_dirs PASSED                                                                                                             [ 25%]
tests/04_tasks_test.py::test_sync_tree_structures_rescue_files_in_deleted_dirs PASSED                                                                                             [ 29%]
tests/04_tasks_test.py::test_catch_permission_error01 PASSED                                                                                                                      [ 32%]
tests/04_tasks_test.py::test_catch_permission_error02a PASSED                                                                                                                     [ 35%]
tests/04_tasks_test.py::test_catch_permission_error02b PASSED                                                                                                                     [ 38%]
tests/04_tasks_test.py::test_catch_permission_error02c PASSED                                                                                                                     [ 41%]
tests/04_tasks_test.py::test_catch_permission_error02d PASSED                                                                                                                     [ 45%]
tests/04_tasks_test.py::test_catch_permission_error02e PASSED                                                                                                                     [ 48%]
tests/04_tasks_test.py::test_catch_permission_error02f PASSED                                                                                                                     [ 51%]
tests/04_tasks_test.py::test_catch_permission_error03 PASSED                                                                                                                      [ 54%]
tests/04_tasks_test.py::test_catch_permission_error04 PASSED                                                                                                                      [ 58%]
tests/04_tasks_test.py::test_sync_tree_structures_rescue_files_permission_errors PASSED                                                                                           [ 61%]
tests/04_tasks_test.py::test_sync_tree_structures_scenario_01 PASSED                                                                                                              [ 64%]
tests/04_tasks_test.py::test_sync_tree_structures_scenario_02 PASSED                                                                                                              [ 67%]
tests/04_tasks_test.py::test_sync_tree_structures_permission_error_on_new_dir PASSED                                                                                              [ 70%]
tests/04_tasks_test.py::test_sync_tree_structures_permission_error_on_deleted_dir PASSED                                                                                          [ 74%]
tests/04_tasks_test.py::test_sync_tree_structures_permission_error_on_moved_dir PASSED                                                                                            [ 77%]
tests/04_tasks_test.py::test_sync_tree_structures_filters PASSED                                                                                                                  [ 80%]
tests/04_tasks_test.py::test_sync_tree_structures_git_protection PASSED                                                                                                           [ 83%]
tests/04_tasks_test.py::test_update_paired_files PASSED                                                                                                                           [ 87%]
tests/04_tasks_test.py::test_update_paired_files_conflicts PASSED                                                                                                                 [ 90%]
tests/04_tasks_test.py::test_update_paired_files_no_update PASSED                                                                                                                 [ 93%]
tests/04_tasks_test.py::test_sync_files PASSED                                                                                                                                    [ 96%]
tests/04_tasks_test.py::test_sync_files_no_change PASSED                                                                                                                          [100%]

================================================================================== 31 passed in 0.83s ===================================================================================