shutil.rmtree using vagrant synched folder fails

[PeterFS]

BPO	39327
Nosy	@serhiy-storchaka, @miss-islington, @lkantola
PRs	bpo-39327: Fixed issue: Avoid "Text File Busy" OSError #31384 [3.9] bpo-39327: Close file descriptors as soon as possible in shutil.rmtree (GH-31384) #31450 [3.10] bpo-39327: Close file descriptors as soon as possible in shutil.rmtree (GH-31384) #31451
Files	Vagrantfile

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2022-02-21.09:19:35.883>
created_at = <Date 2020-01-14.09:45:52.878>
labels = ['type-bug', 'library', '3.9', '3.10', '3.11']
title = 'shutil.rmtree using vagrant synched folder fails'
updated_at = <Date 2022-02-21.09:19:35.878>
user = 'https://bugs.python.org/PeterFS'

bugs.python.org fields:

activity = <Date 2022-02-21.09:19:35.878>
actor = 'serhiy.storchaka'
assignee = 'none'
closed = True
closed_date = <Date 2022-02-21.09:19:35.883>
closer = 'serhiy.storchaka'
components = ['Library (Lib)']
creation = <Date 2020-01-14.09:45:52.878>
creator = 'PeterFS'
dependencies = []
files = ['48838']
hgrepos = []
issue_num = 39327
keywords = ['patch']
message_count = 5.0
messages = ['359961', '360028', '408174', '413595', '413636', '413637']
nosy_count = 5.0
nosy_names = ['python-dev', 'serhiy.storchaka', 'miss-islington', 'PeterFS', 'lkantola']
pr_nums = ['31384', '31450', '31451']
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue39327'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

[PeterFS]

Python 3.6.9
Ubuntu 18.04
python3 -c 'import shutil; shutil.rmtree("1a")'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.6/shutil.py", line 486, in rmtree
    _rmtree_safe_fd(fd, path, onerror)
  File "/usr/lib/python3.6/shutil.py", line 424, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/lib/python3.6/shutil.py", line 424, in _rmtree_safe_fd
    _rmtree_safe_fd(dirfd, fullname, onerror)
  File "/usr/lib/python3.6/shutil.py", line 428, in _rmtree_safe_fd
    onerror(os.rmdir, fullname, sys.exc_info())
  File "/usr/lib/python3.6/shutil.py", line 426, in _rmtree_safe_fd
    os.rmdir(name, dir_fd=topfd)
OSError: [Errno 26] Text file busy: '4a'

---

Reproduction method
mkdir synched_folder\1a\2a\3a\4a
mkdir synched_folder\1a\2b\3a\4a
mkdir synched_folder\1b\2a\3a\4a
choco install vagrant
Save Vagrantfile in empty folder
vagrant box add ubuntu/bionic64
vagrant up
vagrant ssh
cd synched_folder
python3 -c 'import shutil; shutil.rmtree("1a")'
*** Error message ***
rm -r 1a
*** Works fine ***

[PeterFS]

Problem is also reported in virtualbox
https://www.virtualbox.org/ticket/19004

From that ticket some more analysis is done;
strace reveals that Python has kept an open fd for the directory being removed.

[lkantola]

I have encountered the same problem with vboxsf mounted folders when using Debian guest on VirtualBox 6.1 hosted by Windows 10.

The general VirtualBox issue of vboxsf not supporting rmdir when there are open file descriptors is difficult to solve, and out of scope for Python.

However, it looks like the shutil.rmtree implementation can be easily modified to change the order of operations so that the rmdir is performed after the directory file descriptor is closed, which makes shutil.rmtree work under vboxsf shared folders.

Example of the change against Python 3.7 is inlined below. The changed order matches the behavior of "rm -r" and to my understanding should not change the security implications of the code, but as the code is related to the prevention of a symlink attack https://bugs.python.org/issue4489 careful review would be needed.

--- /usr/lib/python3.7/shutil.py--orig  2021-01-23 05:04:44.000000000 +0900
+++ /usr/lib/python3.7/shutil.py        2021-12-09 20:42:36.795338346 +0900
@@ -427,10 +427,6 @@
                 try:
                     if os.path.samestat(orig_st, os.fstat(dirfd)):
                         _rmtree_safe_fd(dirfd, fullname, onerror)
-                        try:
-                            os.rmdir(entry.name, dir_fd=topfd)
-                        except OSError:
-                            onerror(os.rmdir, fullname, sys.exc_info())
                     else:
                         try:
                             # This can only happen if someone replaces
@@ -442,6 +438,10 @@
                             onerror(os.path.islink, fullname, sys.exc_info())
                 finally:
                     os.close(dirfd)
+                try:
+                    os.rmdir(entry.name, dir_fd=topfd)
+                except OSError:
+                    onerror(os.rmdir, fullname, sys.exc_info())
         else:
             try:
                 os.unlink(entry.name, dir_fd=topfd)
@@ -489,10 +489,6 @@
         try:
             if os.path.samestat(orig_st, os.fstat(fd)):
                 _rmtree_safe_fd(fd, path, onerror)
-                try:
-                    os.rmdir(path)
-                except OSError:
-                    onerror(os.rmdir, path, sys.exc_info())
             else:
                 try:
                     # symlinks to directories are forbidden, see bug python/cpython#46010
@@ -501,6 +497,10 @@
                     onerror(os.path.islink, path, sys.exc_info())
         finally:
             os.close(fd)
+        try:
+            os.rmdir(path)
+        except OSError:
+            onerror(os.rmdir, path, sys.exc_info())
     else:
         try:
             if os.path.islink(path):

[serhiy-storchaka]

New changeset b77158b by Lital Natan in branch 'main':
bpo-39327: Close file descriptors as soon as possible in shutil.rmtree (GH-31384)
b77158b

[miss-islington]

New changeset 959846b by Miss Islington (bot) in branch '3.9':
bpo-39327: Close file descriptors as soon as possible in shutil.rmtree (GH-31384)
959846b

[miss-islington]

New changeset 95d6271 by Miss Islington (bot) in branch '3.10':
bpo-39327: Close file descriptors as soon as possible in shutil.rmtree (GH-31384)
95d6271