Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

shutil.copytree fails when copying NFS to NFS #68752

Closed
jhamrick mannequin opened this issue Jul 4, 2015 · 13 comments
Closed

shutil.copytree fails when copying NFS to NFS #68752

jhamrick mannequin opened this issue Jul 4, 2015 · 13 comments
Labels
3.7 (EOL) end of life 3.8 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error

Comments

@jhamrick
Copy link
Mannequin

jhamrick mannequin commented Jul 4, 2015

BPO 24564
Nosy @larryhastings, @giampaolo, @bitdancer, @albertz, @hynek, @desbma, @minrk, @csabella, @cdyson37
PRs
  • bpo-24564: shutil.copystat(): ignore EINVAL on os.setxattr() #8601
  • bpo-24564: shutil.copystat(): ignore EINVAL on os.setxattr() #13369
  • [3.7] bpo-24564: shutil.copystat(): ignore EINVAL on os.setxattr() (GH-13369) #13673
  • Files
  • bug_demo.py: Reproduces the copytree error on Python 3.4
  • Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.

    Show more details

    GitHub fields:

    assignee = None
    closed_at = <Date 2019-05-30.06:03:47.433>
    created_at = <Date 2015-07-04.20:05:50.265>
    labels = ['3.7', '3.8', 'type-bug', 'library']
    title = 'shutil.copytree fails when copying NFS to NFS'
    updated_at = <Date 2019-09-09.21:58:13.583>
    user = 'https://bugs.python.org/jhamrick'

    bugs.python.org fields:

    activity = <Date 2019-09-09.21:58:13.583>
    actor = 'Michael Burt'
    assignee = 'none'
    closed = True
    closed_date = <Date 2019-05-30.06:03:47.433>
    closer = 'giampaolo.rodola'
    components = ['Library (Lib)']
    creation = <Date 2015-07-04.20:05:50.265>
    creator = 'jhamrick'
    dependencies = []
    files = ['39864']
    hgrepos = []
    issue_num = 24564
    keywords = ['patch']
    message_count = 13.0
    messages = ['246272', '246273', '246278', '246279', '280051', '304970', '322848', '322942', '341858', '342794', '343944', '343949', '351555']
    nosy_count = 13.0
    nosy_names = ['larry', 'giampaolo.rodola', 'r.david.murray', 'Albert.Zeyer', 'hynek', 'desbma', 'hans-meine', 'minrk', 'jhamrick', 'Gabriel Devenyi', 'cheryl.sabella', 'cdyson37', 'Michael Burt']
    pr_nums = ['8601', '13369', '13673']
    priority = 'normal'
    resolution = 'fixed'
    stage = 'resolved'
    status = 'closed'
    superseder = None
    type = 'behavior'
    url = 'https://bugs.python.org/issue24564'
    versions = ['Python 3.7', 'Python 3.8']

    @jhamrick
    Copy link
    Mannequin Author

    jhamrick mannequin commented Jul 4, 2015

    shutil.copytree seems to fail when copying files across NFS filesystems. In this example (see bug_demo.py), /tmp is a normal ext4 filesystem and the current working directory is NFS (version 4). Interestingly, it works fine to to copy between ext4 and NFS, just not NFS and NFS (even if it's the same NFS mount).

    Depending on the specific version of 3.4, there are slightly different errors. For 3.4.0:

    $ uname -a
    Linux compmodels-node 3.13.0-55-generic #94-Ubuntu SMP Thu Jun 18 00:27:10 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
    $ python3 bug_demo.py

    Python version info:
    3.4.0 (default, Jun 19 2015, 14:20:21)
    [GCC 4.8.2]
    --------------------------------------------------

    Copying NFS --> /tmp
    Copying /tmp --> NFS
    Copying NFS --> NFS
    Traceback (most recent call last):
      File "/usr/lib/python3.4/shutil.py", line 336, in copytree
        copystat(src, dst)
      File "/usr/lib/python3.4/shutil.py", line 212, in copystat
        _copyxattr(src, dst, follow_symlinks=follow)
      File "/usr/lib/python3.4/shutil.py", line 152, in _copyxattr
        os.setxattr(dst, name, value, follow_symlinks=follow_symlinks)
    OSError: [Errno 22] Invalid argument: 'demo_files3'
    
    During handling of the above exception, another exception occurred:
    
    Traceback (most recent call last):
      File "bug_demo.py", line 27, in <module>
        shutil.copytree('demo_files', 'demo_files3')
      File "/usr/lib/python3.4/shutil.py", line 339, in copytree
        if why.winerror is None:
    AttributeError: 'OSError' object has no attribute 'winerror'

    And for 3.4.3:

    $ uname -a
    Linux compmodels-node-02 3.19.0-21-generic #21-Ubuntu SMP Sun Jun 14 18:31:11 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
    $ python3 bug_demo.py

    Python version info:
    3.4.3 (default, Mar 26 2015, 22:03:40)
    [GCC 4.9.2]
    --------------------------------------------------

    Copying NFS --> /tmp
    Copying /tmp --> NFS
    Copying NFS --> NFS
    Traceback (most recent call last):
      File "bug_demo.py", line 27, in <module>
        shutil.copytree('demo_files', 'demo_files3')
      File "/usr/lib/python3.4/shutil.py", line 343, in copytree
        raise Error(errors)
    shutil.Error: [('demo_files', 'demo_files3', "[Errno 22] Invalid argument: 'demo_files3'")]

    This is probably related to https://bugs.python.org/issue17076 and my guess is that's also why there is a difference in the error messages between 3.4.0 and 3.4.3.

    @jhamrick jhamrick mannequin added the type-crash A hard crash of the interpreter, possibly with a core dump label Jul 4, 2015
    @jhamrick
    Copy link
    Mannequin Author

    jhamrick mannequin commented Jul 4, 2015

    Some further information: if I run copystat directly on 3.4.3, I get essentially the same error as on 3.4.0. So really it only looks like the difference is just in how the error is reported:

    Traceback (most recent call last):
      File "bug_demo.py", line 31, in <module>
        shutil.copystat('demo_files', 'demo_files3')
      File "/usr/lib/python3.4/shutil.py", line 213, in copystat
        _copyxattr(src, dst, follow_symlinks=follow)
      File "/usr/lib/python3.4/shutil.py", line 153, in _copyxattr
        os.setxattr(dst, name, value, follow_symlinks=follow_symlinks)
    OSError: [Errno 22] Invalid argument: 'demo_files3'

    Also, I should mention that I did test this on 2.7 as well, and it is not an issue there.

    @minrk
    Copy link
    Mannequin

    minrk mannequin commented Jul 4, 2015

    On a bit of further investigation, the NFS files have an xattr system.nfs4_acl. This can be read, but attempting to write it fails with EINVAL. Attempting to copy from NFS to non-NFS fails with ENOTSUP, which is caught and ignored, but copying from NFS to NFS raises EINVAL, which raises.

    Adding EINVAL to the ignored errnos would fix the problem, but might hide real failures (I'm not sure about the real failures, but it seems logical).

    Since the copy_function is customizable to switch between copy and copy2, making copystat optional on files, perhaps the copystat should be optional on directories, as well.

    @bitdancer
    Copy link
    Member

    There are a number of open issues with copytree originating from copystat. It would be great if someone could pull them all together and propose a solution. Making it optional might indeed be the best solution.

    @GabrielDevenyi
    Copy link
    Mannequin

    GabrielDevenyi mannequin commented Nov 4, 2016

    I'm running into this issue with python 3.5 and a ZFS-backed NFS4 mount.

    Strace of a setuptools install:
    chmod("/opt/quarantine/pydpiper/2.0/build/lib/python3.4/site-packages/pydpiper-2.0-py3.4.egg", 0644) = 0
    listxattr("dist/pydpiper-2.0-py3.4.egg", "system.nfs4_acl\0", 256) = 16
    getxattr("dist/pydpiper-2.0-py3.4.egg", "system.nfs4_acl", "\x00\x00\x00\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x16\x01\x87\x00\x00\x00\x06OWNER@\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x12\x00\x81\x00\x00\x00\x06GROUP@\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x12\x00\x81\x00\x00\x00\x09EVERYONE@\x00\x00", 128) = 80
    setxattr("/opt/quarantine/pydpiper/2.0/build/lib/python3.4/site-packages/pydpiper-2.0-py3.4.egg", "system.nfs4_acl", "\x00\x00\x00\x03\x00\x00\x00\x00\x00\x00\x00\x00\x00\x16\x01\x87\x00\x00\x00\x06OWNER@\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x12\x00\x81\x00\x00\x00\x06GROUP@\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x12\x00\x81\x00\x00\x00\x09EVERYONE@\x00\x00", 80, 0) = -1 EIO (Input/output error)
    stat("/tmp/easy_install-41z79x8r", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
    openat(AT_FDCWD, "/tmp/easy_install-41z79x8r", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 4
    getdents(4, /* 2 entries */, 32768) = 48
    getdents(4, /* 0 entries */, 32768) = 0
    close(4) = 0
    rmdir("/tmp/easy_install-41z79x8r") = 0
    write(2, "error: [Errno 5] Input/output er"..., 125error: [Errno 5] Input/output error: '/opt/quarantine/pydpiper/2.0/build/lib/python3.4/site-packages/pydpiper-2.0-py3.4.egg'
    ) = 125

    @albertz
    Copy link
    Mannequin

    albertz mannequin commented Oct 25, 2017

    I'm also affected by this, with Python 3.6. My home directory is on a ZFS-backed NFS share.
    See here for details:
    Homebrew/linuxbrew-core#4799

    Basically:
    Copying setuptools.egg-info to /u/zeyer/.linuxbrew/lib/python3.6/site-packages/setuptools-36.5.0-py3.6.egg-info
    error: [Errno 5] Input/output error: '/u/zeyer/.linuxbrew/lib/python3.6/site-packages/setuptools-36.5.0-py3.6.egg-info/zip-safe'

    Note that also by other tools, such as mv and cp, I get errors about setting system.nfs4_acl. But they just seem to ignore that and go on. I think this is the right thing to do here. You can print a warning about that, but then just go on. Maybe esp. just for system.nfs4_acl.

    @serhiy-storchaka serhiy-storchaka added stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error and removed type-crash A hard crash of the interpreter, possibly with a core dump labels Jul 11, 2018
    @giampaolo
    Copy link
    Contributor

    [ https://bugs.python.org/issue24564#msg246278 ]

    Adding EINVAL to the ignored errnos would fix the problem, but might hide real failures (I'm not sure about the real failures, but it seems logical).

    I think this is an acceptable compromise considering that:

    1. because of this the copy operation will ultimately fail (despite file will be copied)
    2. there is nothing we can do except emit a warning somehow
    3. EPERM and ENODATA are already silently ignored (also ENOTSUP but that is more legit)

    For completeness, other than copytree() also copy2() and move() are affected.
    I submitted a PR: #8601.

    @giampaolo giampaolo added 3.7 (EOL) end of life 3.8 only security fixes labels Aug 1, 2018
    @desbma
    Copy link
    Mannequin

    desbma mannequin commented Aug 2, 2018

    Since the copy_function is customizable to switch between copy and copy2, making copystat optional on files, perhaps the copystat should be optional on directories, as well.

    Related: https://bugs.python.org/issue32073

    @hans-meine
    Copy link
    Mannequin

    hans-meine mannequin commented May 8, 2019

    I completely agree with msg322848, both with the suggested fix and with its rationale. A future API improvement could be to have a "strict" mode where errors copying xattr are not silently ignored anymore (if there are src xattr, but they cannot be written, or if src xattr cannot be read, for instance), but the current functionality seems to be "copy xattr if possible", and that should be fixed accordingly.

    I also opened bpo-36850 arguing that in any case, EINVAL should be ignored when *reading* extended attributes, acknowledging that people might have more problems with ignoring errors on *writing* them when they were present on the source. However, as stated above, I personally vote for the fix suggested in msg322848.

    @csabella
    Copy link
    Contributor

    Nosying Hynek and Larry, as they had done the original code.

    @giampaolo
    Copy link
    Contributor

    New changeset a16387a by Giampaolo Rodola (Ying Wang) in branch 'master':
    bpo-24564: shutil.copystat(): ignore EINVAL on os.setxattr() (GH-13369)
    a16387a

    @giampaolo
    Copy link
    Contributor

    New changeset f1487b3 by Giampaolo Rodola (Miss Islington (bot)) in branch '3.7':
    bpo-24564: shutil.copystat(): ignore EINVAL on os.setxattr() (GH-13369)
    f1487b3

    @MichaelBurt
    Copy link
    Mannequin

    MichaelBurt mannequin commented Sep 9, 2019

    This is still a problem when shutil gets a errno.ENOSYS

    I hit this bug on Microsoft Azure when I mount an Azure File (managed NFS) into an AKS cluster (managed Kubernetes offering) and try to copy a file from the NFS over to the local disk on the node using shutil.copytree().

    The workaround I am using came from this StackOverflow answer: https://stackoverflow.com/a/51635427/3736286

    Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
    Labels
    3.7 (EOL) end of life 3.8 only security fixes stdlib Python modules in the Lib dir type-bug An unexpected behavior, bug, or error
    Projects
    None yet
    Development

    No branches or pull requests

    4 participants