tempfile.TemporaryFile fails when dir option set to directory residing on host OS mount

[hlawrenz]

BPO	25717
Nosy	@pitrou, @vstinner, @vadmium, @serhiy-storchaka, @Vgr255, @hlawrenz
Files	tempfile-strace.txt tempfile.py fstat-failure.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2015-12-06.05:01:41.677>
created_at = <Date 2015-11-24.04:23:44.477>
labels = ['type-bug', 'library']
title = 'tempfile.TemporaryFile fails when dir option set to directory residing on host OS mount'
updated_at = <Date 2016-06-18.22:37:07.828>
user = 'https://github.com/hlawrenz'

bugs.python.org fields:

activity = <Date 2016-06-18.22:37:07.828>
actor = 'Oleg Babintsev'
assignee = 'none'
closed = True
closed_date = <Date 2015-12-06.05:01:41.677>
closer = 'martin.panter'
components = ['Library (Lib)']
creation = <Date 2015-11-24.04:23:44.477>
creator = 'Hans Lawrenz'
dependencies = []
files = ['41145', '41153', '41155']
hgrepos = []
issue_num = 25717
keywords = ['patch', '3.5regression']
message_count = 27.0
messages = ['255246', '255247', '255250', '255254', '255273', '255275', '255276', '255283', '255285', '255286', '255287', '255288', '255292', '255352', '255553', '255647', '256001', '256002', '256006', '256008', '256009', '256015', '268783', '268790', '268792', '268796', '268834']
nosy_count = 9.0
nosy_names = ['pitrou', 'vstinner', 'python-dev', 'martin.panter', 'serhiy.storchaka', 'bkabrda', 'abarry', 'Hans Lawrenz', 'Oleg Babintsev']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue25717'
versions = ['Python 3.5', 'Python 3.6']

[hlawrenz]

Inside a virtualbox vm, calling tempfile.TemporaryFile(dir=foo) where foo is a directory which resides on a volume mounted from the host OS, a FileNotFoundError exception is thrown.

In the following code sample, the second block will print "Path 2: ERROR" on Python 3.5 but not on 3.4 or 2.7 (assuming the vagrant environment provided below):

import tempfile

try:
with tempfile.TemporaryFile() as tf:
tf.write("testing testing testing\n".encode('utf-8'))
print("Path 1: worked")
except FileNotFoundError as e:
print("Path 1: ERROR")

try:
with tempfile.TemporaryFile(dir="/vagrant") as tf:
tf.write("testing testing testing\n".encode('utf-8'))
print("Path 2: worked")
except FileNotFoundError as e:
print("Path 2: ERROR")

A runnable test case can be found here:
https://github.com/hlawrenz/py35tempfiletest

[vadmium]

I am unable to produce this on native Linux so far with various kinds of mount points I normally have. What is your guest OS?

If you can provide any more details of the failure, such as a traceback, that would be helpful. What is the underlying OS call that is causing the exception?

[hlawrenz]

Host OS: Mac OS 10.11.1
Guest OS: Ubuntu Trusty 64
Virtualbox: 4.3.30
Vagrant: 1.7.4

Example with trace thrown:

vagrant@vagrant-ubuntu-trusty-64:/vagrant$ cat tt.py
import tempfile

with tempfile.TemporaryFile(dir="/vagrant") as tf:
    tf.write("testing testing testing\n".encode('utf-8'))

vagrant@vagrant-ubuntu-trusty-64:/vagrant$ python3.5 tt.py 
Traceback (most recent call last):
  File "tt.py", line 4, in <module>
    with tempfile.TemporaryFile(dir="/vagrant") as tf:
  File "/usr/lib/python3.5/tempfile.py", line 753, in TemporaryFile
    newline=newline, encoding=encoding)
FileNotFoundError: [Errno 2] No such file or directory

I think the underlying system call that's causing the error (I attached the full strace in case that's helpful):
open("/vagrant", O_RDWR|O_EXCL|O_DIRECTORY|O_NOFOLLOW|O_CLOEXEC|0x400000) = -1 EOPNOTSUPP (Operation not supported)

Finally, the mount point looks like this:
vagrant on /vagrant type vboxsf (uid=1000,gid=1000,rw)

[vadmium]

Thanks for the strace output. I think the actual error is the fstat() call a bit after that first open() call. FileNotFoundError is ENOENT:

open("/vagrant/tmpy5ioznh4", O_RDWR|O_CREAT|O_EXCL|O_NOFOLLOW|O_CLOEXEC, 0600) = 4
unlink("/vagrant/tmpy5ioznh4")          = 0
fstat(4, 0x7ffc0b326520)                = -1 ENOENT (No such file or directory)
close(4)                                = 0
write(2, "Traceback (most recent call last"..., 35Traceback (most recent call last):

My theory is that the fstat() call at <https://hg.python.org/cpython/annotate/3.5/Modules/_io/fileio.c#l441\> is failing, probably because the file entry has been removed from its directory. This call was added by revision 3b5279b5bfd1 (bpo-21679). Before this change, fstat() was called twice, but in each case an error was tolerated.

Posix does not mention fstat() returning this ENOENT error, so maybe this could be a bug with the Virtual Box or Vagrant driver for the mounted filesystem. But it might be nice to make Python more permissive if fstat() fails, like it was in 3.4.

As a workaround, you may be able to use NamedTemporaryFile, if you are happy for the file to have a name and a directory entry.

[hlawrenz]

Unfortunately changing the tempfile call isn't an easy option for me. The situation in which I'm encountering the error is running tox on our project which is mounted in the /vagrant directory in the VM (this is standard for vagrant). Tox makes its own temp directory--presumably for test isolation--in the .tox directory in the project root. The actual call to tempfile.TemporaryFile() which is triggering the error is in a third party library that is called during a test run.

I was able to work around the issue by changing the tox workdir setting to outside the mount. However, I'd like to mention that developing in vagrant in this fashion isn't uncommon and once 3.5 gains adoption I would guess this issue may affect a good number of people.

I'm happy to take a stab at writing a patch but looking at the code it's somewhat out of my comfort zone and I worry I'd make a hash of it.

[serhiy-storchaka]

What is exact version of you Python? I can't identify the line in the traceback. There are few lines "newline=newline, encoding=encoding)" in tempfile.py, but no one is closer to line 753.

[Vgr255]

I'm with Serhiy, the line number is inane. Could you put the exact contents of your /usr/lib/python3.5/tempfile.py file somewhere for us to see? Output of sys.version would be nice, too.

[hlawrenz]

Serhiy and Emanuel, I'll paste below the surrounding code and attach the exact tempfile.py. It is the version distributed with the 3.5.0 release. If you take a look at the github repo I linked in the first comment you can also try it out for yourself if you've got vagrant and virtualbox installed. I was able to recreate the error with the provisioning compiling python and with it installing from a ppa. You can see the details in the Vagrantfile. I also had a coworker test in a nearly identical environment and he had the same result.

    (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags, output_type)
    try:
        _os.unlink(name)
        return _io.open(fd, mode, buffering=buffering,
                        newline=newline, encoding=encoding)
    except:
        _os.close(fd)
        raise

One final note. I decided to try this out with a windows host and the only clarity this test adds is that the problem doesn't show up there but it fails for its own reasons on all the python versions I tried. See below for reference:

Last login: Tue Nov 24 17:16:12 2015 from 10.0.2.2
vagrant@vagrant-ubuntu-trusty-64:~$ cat /vagrant/foo.py
import tempfile

with tempfile.TemporaryFile(dir="/vagrant") as tf:
tf.write("testing testing testing\n".encode('utf-8'))
print("Path 2: worked")
vagrant@vagrant-ubuntu-trusty-64:~$ python /vagrant/foo.py
Traceback (most recent call last):
  File "/vagrant/foo.py", line 4, in <module>
    with tempfile.TemporaryFile(dir="/vagrant") as tf:
  File "/usr/lib/python2.7/tempfile.py", line 495, in TemporaryFile
    _os.unlink(name)
OSError: [Errno 26] Text file busy: '/vagrant/tmpvx7Mie'
vagrant@vagrant-ubuntu-trusty-64:~$ python2.7 /vagrant/foo.py
Traceback (most recent call last):
  File "/vagrant/foo.py", line 4, in <module>
    with tempfile.TemporaryFile(dir="/vagrant") as tf:
  File "/usr/lib/python2.7/tempfile.py", line 495, in TemporaryFile
    _os.unlink(name)
OSError: [Errno 26] Text file busy: '/vagrant/tmpNXQ6Cf'
vagrant@vagrant-ubuntu-trusty-64:~$ python3.4 /vagrant/foo.py
Traceback (most recent call last):
  File "/vagrant/foo.py", line 4, in <module>
    with tempfile.TemporaryFile(dir="/vagrant") as tf:
  File "/usr/lib/python3.4/tempfile.py", line 638, in TemporaryFile
    _os.unlink(name)
OSError: [Errno 26] Text file busy: '/vagrant/tmpfwhj7ip4'
vagrant@vagrant-ubuntu-trusty-64:~$ python3.5 /vagrant/foo.py
Traceback (most recent call last):
  File "/vagrant/foo.py", line 4, in <module>
    with tempfile.TemporaryFile(dir="/vagrant") as tf:
  File "/usr/lib/python3.5/tempfile.py", line 751, in TemporaryFile
    _os.unlink(name)
OSError: [Errno 26] Text file busy: '/vagrant/tmp02s19r_a'
vagrant@vagrant-ubuntu-trusty-64:~$ python2.7 --version
Python 2.7.6
vagrant@vagrant-ubuntu-trusty-64:~$ python3.4 --version
Python 3.4.3
vagrant@vagrant-ubuntu-trusty-64:~$ python3.5 --version
Python 3.5.0

[serhiy-storchaka]

Is your file system NFS?

[Vgr255]

Your file has a lot of shenanigans that are triggered if 'shutil' fails to be imported, adding an extra 139 lines at the top of the file, which is exactly how offset your traceback is compared to our lines. The rest of the file is virtually identical. This makes me wonder, however, why you have these differences (and that this might not be the only different file).

I would still like the output of sys.version, which, unlike 'python --version' contains the build date and time, among other things. Bonus points for the 'sys.implementation' output. It's unlikely to be of relevance, but I would rather explore every possibility.

[hlawrenz]

The file system causing the problem is of type vboxsf which is the Virtualbox shared folder file system type.

[hlawrenz]

Emanuel, sorry, I missed the request for sys.version earlier.

The tempfile.py I attached earlier is from the python 3.5 pulled from a ppa. I wouldn't be surprised if it has some patches applied by debian/ubuntu. To be clear though the problem also presents itself with a python 3.5 I compiled. The Vagrantfile in the github repository shows exactly how I compiled it.

Here it is from the version from the ppa:

Python 3.5.0 (default, Sep 17 2015, 00:00:00) 
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.version
'3.5.0 (default, Sep 17 2015, 00:00:00) \n[GCC 4.8.4]'
>>> sys.implementation
namespace(_multiarch='x86_64-linux-gnu', cache_tag='cpython-35', hexversion=50659568, name='cpython', version=sys.version_info(major=3, minor=5, micro=0, releaselevel='final', serial=0))


Here are the same for the version I compiled:
vagrant@vagrant-ubuntu-trusty-64:~$ python3.5
Python 3.5.0 (default, Nov 24 2015, 19:50:42) 
[GCC 4.8.4] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.version
'3.5.0 (default, Nov 24 2015, 19:50:42) \n[GCC 4.8.4]'
>>> sys.implementation
namespace(cache_tag='cpython-35', hexversion=50659568, name='cpython', version=sys.version_info(major=3, minor=5, micro=0, releaselevel='final', serial=0))

[vadmium]

This patch restores the previous behaviour of tolerating fstat() failures other than EBADF. Hans, if you are able to apply it to your compiled version of Python, it might prove that my fstat() theory is correct.

[hlawrenz]

Martin, I applied your patch and it proved your hypothesis. See below for how I tested. I also updated the github repo for others to reproduce if they wish.

cd
wget https://www.python.org/ftp/python/3.5.0/Python-3.5.0.tar.xz

mkdir ~/dist
cd ~/dist
tar xJf ~/Python-3.5.0.tar.xz
cd Python-3.5.0
./configure --prefix=/home/vagrant/py35/dist && \
make && make install

mkdir ~/patch
cd ~/patch
tar xJf ~/Python-3.5.0.tar.xz
cd Python-3.5.0
patch -p1 < /vagrant/fstat-failure.patch
./configure --prefix=/home/vagrant/py35/patch && \
make && make install

vagrant@vagrant-ubuntu-trusty-64:$ ./py35/dist/bin/python3.5 /vagrant/temptest.py
Path 1: worked
Path 2: ERROR
vagrant@vagrant-ubuntu-trusty-64:$ ./py35/patch/bin/python3.5 /vagrant/temptest.py
Path 1: worked
Path 2: worked

[vadmium]

Antoine or Bohuslav, since you were involved in bpo-21679, would you be able to comment on fstat-failure.patch, which reverts back to ignoring most fstat() errors? The problem here is that fstat() seems to fail if the directory entry has been unlinked, and the file is on a Virtual Box shared folder filesystem.

[bkabrda]

The patch looks good to me.

[python-dev]

New changeset 20ea12222b0e by Martin Panter in branch '3.5':
Issue bpo-25717: Tolerate fstat() failures in the FileIO constructor
https://hg.python.org/cpython/rev/20ea12222b0e

New changeset 8c978cbe057c by Martin Panter in branch 'default':
Issue bpo-25717: Merge fstat() fix from 3.5
https://hg.python.org/cpython/rev/8c978cbe057c

[vadmium]

Thanks Bohuslav, and also to Hans for helping track this down.

[vstinner]

Martin, please add a comment to explain the rationale on ignoring errors
other than EBADF. At least, mention this issue number.

[vadmium]

Okay will do

[python-dev]

New changeset e9bf5803b716 by Martin Panter in branch '3.5':
Issue bpo-25717: Add comment explaining why errors are ignored
https://hg.python.org/cpython/rev/e9bf5803b716

New changeset 8bf69413ec01 by Martin Panter in branch 'default':
Issue bpo-25717: Merge comment from 3.5
https://hg.python.org/cpython/rev/8bf69413ec01

[vstinner]

Thank you.

[OlegBabintsev]

Problem is still actual for Python 2.7
Can you provide the fix for 2.7 too?

[vadmium]

Are you sure Oleg? As far as I understand, Python 2 by default wraps C stdio file objects, and also has Python 3’s file objects in the “io” module. But I expect TemporaryFile() would use the default stdio files, and the cause of this bug, bpo-21679, should only have affected the “io” module in 3.5+.

[OlegBabintsev]

My environment:
Windows as host OS, VirtualBox VM, Linux as guest OS

When temporary directory is a directory which mounted from the host OS, a "OSError: [Errno 26] Text file busy" exception is thrown.

Test file:

import tempfile

with tempfile.TemporaryFile(dir="/tmp") as tf:
    tf.write("testing testing testing\n".encode('utf-8'))
print("Temp file: worked")

"/tmp" directory - is a shared folder (vboxsf)

Result:

root@0e87f2561643:/# python /etc/odoo/tempfile-test.py
Traceback (most recent call last):
  File "/etc/odoo/tempfile-test.py", line 3, in <module>
    with tempfile.TemporaryFile(dir="/tmp") as tf:
  File "/usr/lib/python2.7/tempfile.py", line 513, in TemporaryFile
    _os.unlink(name)
OSError: [Errno 26] Text file busy: '/tmp/tmpsl0JQI'

root@0e87f2561643:/# python --version
Python 2.7.11+

root@0e87f2561643:/# python
Python 2.7.11+ (default, Jun  2 2016, 19:34:15)
[GCC 5.3.1 20160528] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sys
>>> sys.version
'2.7.11+ (default, Jun  2 2016, 19:34:15) \n[GCC 5.3.1 20160528]'
>>>

[vadmium]

This original bug was about fstat() failing for an anonymous file (after the directory entry was removed). But in your situation the file system refuses to unlink the directory entry.

If you think there is something that can be fixed in Python, I suggest open a new bug. But it seems this is working as documented: “Under Unix, the directory entry for the file is removed immediately after the file is created.” In your case, this cannot happen, so you get the error from the OS.

[OlegBabintsev]

Thanks for explanation.