tempfile seems to treat a file as a directory when processing an exception in the onerror()

[mehwhatever0]

BPO	43153
Nosy	@serhiy-storchaka, @Fidget-Spinner, @akulakov, @mehwhatever0
PRs	bpo-43153: Don't mask PermissionError with NotADirectoryError during tempdirectory cleanup #29940

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2021-02-07.09:52:44.316>
labels = ['type-bug', 'library', '3.9', '3.10', '3.11']
title = 'tempfile seems to treat a file as a directory when processing an exception in the onerror()'
updated_at = <Date 2021-12-06.14:13:37.485>
user = 'https://github.com/mehwhatever0'

bugs.python.org fields:

activity = <Date 2021-12-06.14:13:37.485>
actor = 'kj'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2021-02-07.09:52:44.316>
creator = 'mehwhatever0'
dependencies = []
files = []
hgrepos = []
issue_num = 43153
keywords = ['patch']
message_count = 4.0
messages = ['386587', '397118', '407376', '407435']
nosy_count = 4.0
nosy_names = ['serhiy.storchaka', 'kj', 'andrei.avk', 'mehwhatever0']
pr_nums = ['29940']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue43153'
versions = ['Python 3.9', 'Python 3.10', 'Python 3.11']

Linked PRs

• gh-87319: Simplify TemporaryDirectory cleanup using os.path.isjunction() #112791
• [3.12] gh-87319: Simplify TemporaryDirectory cleanup using os.path.isjunction() (GH-112791) #112845

[mehwhatever0]

>>> import sys, tempfile, os
>>> sys.version
'3.9.1 (tags/v3.9.1:1e5d33e, Dec  7 2020, 17:08:21) [MSC v.1927 64 bit (AMD64)]'
>>> folder = tempfile.TemporaryDirectory()
>>> file = open(os.path.join(folder.name, "example.txt"), 'wb')
>>> folder.cleanup()
Traceback (most recent call last):
  File "C:\Users\user0\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 616, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 32] Процесс не может получить доступ к файлу, так как этот файл занят другим процессом: 'C:\\Users\\user0\\AppData\\Local\\Temp\\tmpin8xeb9b\\example.txt'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\user0\AppData\Local\Programs\Python\Python39\lib\tempfile.py", line 801, in onerror
    _os.unlink(path)
PermissionError: [WinError 32] Процесс не может получить доступ к файлу, так как этот файл занят другим процессом: 'C:\\Users\\user0\\AppData\\Local\\Temp\\tmpin8xeb9b\\example.txt'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\user0\AppData\Local\Programs\Python\Python39\lib\tempfile.py", line 830, in cleanup
    self._rmtree(self.name)
  File "C:\Users\user0\AppData\Local\Programs\Python\Python39\lib\tempfile.py", line 812, in _rmtree
    _shutil.rmtree(name, onerror=onerror)
  File "C:\Users\user0\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 740, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Users\user0\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 618, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\Users\user0\AppData\Local\Programs\Python\Python39\lib\tempfile.py", line 804, in onerror
    cls._rmtree(path)
  File "C:\Users\user0\AppData\Local\Programs\Python\Python39\lib\tempfile.py", line 812, in _rmtree
    _shutil.rmtree(name, onerror=onerror)
  File "C:\Users\user0\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 740, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Users\user0\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 599, in _rmtree_unsafe
    onerror(os.scandir, path, sys.exc_info())
  File "C:\Users\user0\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 596, in _rmtree_unsafe
    with os.scandir(path) as scandir_it:
NotADirectoryError: [WinError 267] Неверно задано имя папки: 'C:\\Users\\user0\\AppData\\Local\\Temp\\tmpin8xeb9b\\example.txt'

[akulakov]

This is a duplicate of https://bugs.python.org/issue43219 .

[akulakov]

My last comment was wrong, the issue I linked is unrelated.

I think the actual issue here is the code here:

cpython/Lib/tempfile.py

Line 826 in 4b97d97

except (IsADirectoryError, PermissionError):

If path is a file that causes a permission error, rmtree is called and causes a NotADirectory error.

This can be confusing when debugging and can lead to a bug if user's code tries to catch a PermissionError it expects but instead gets NotADirectory error.

A solution is probably to check if path is a file and error is PermissionError and if ignore_errors=False, re-raise it instead of calling rmtree(). If ignore_errors=True, return instead of calling rmtree().

I don't have windows so can't test the OP code with this approach.

Serhiy: adding you since you wrote def onerror() in tempfile module, do you think that is the cause of the issue and the right solution for it?

[serhiy-storchaka]

I agree with Andrei. I'll test this approach on Windows.

[bekieark]

Hi Team, Do you have any news about an eventual patch for this? Was it solved in later Python versions?

I am running Python 3.9.1 (tags/v3.9.1:1e5d33e, Dec 7 2020, 17:08:21) [MSC v.1927 64 bit (AMD64)], on a Windows 10 64-bit machine, with 8 GB of RAM and an SSD drive. No filesystem compression. I am handling large data objects, coming from a scientific package. I am simply trying to serialize them on disk, for future use.

Smaller files serialize correctly using pickle, marshal, joblib and dill libraries. However larger files (from approx. 200 MB up, in terms of their size on the disk) cause one and the same Error all the time, as listed below. I will try also HDF (hdf5) and serpy serialization, but at this point I am sure it is simply related to large file size.

The error is identical to the one observed by You 1-2 years ago here. IMHO system tries to use temporary data in local Temp folder and something goes wrong. E.g. it attempts to unlink a file, it checks folder - yet he thinks the filename is a folder name. Or something similar.

It is quite important for me, related to my day job... Is there something I could do myself, or test? I plan to dig in the <shutil.py> library, but maybe You already made some tests... $\textcolor{green}{\text{T}}\textcolor{lime}{\text{h}}\textcolor{Orange}{\text{a}}\textcolor{BurntOrange}{\text{n}}\textcolor{red}{\text{k}}\textcolor{pink}{\text{s}}$!

>>> joblib.dump(oLarge, '20230126T2243CET_o_large.jlib')
Traceback (most recent call last):
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 616, in _rmtree_unsafe
    os.unlink(fullname)
PermissionError: [WinError 32] Proces nie mo?e uzyska? dost?pu do pliku, poniewa? jest on u?ywany przez inny proces: 'C:\\Users\\myuser\\AppData\\Local\\Temp\\tmpwteculmb\\tmp.ann'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\tempfile.py", line 801, in onerror
    _os.unlink(path)
PermissionError: [WinError 32] Proces nie moze uzyskac dostepu do pliku, poniewaz jest on uzywany przez inny proces: 'C:\\Users\\myuser\\AppData\\Local\\Temp\\tmpwteculmb\\tmp.ann'

<COMMENT> The above is in Polish, which is my Windows 64-bit locale. It is a generic error, related to a "Process that cannot access file, as it is used by another process". The issue is probably the filename instead of directory name and the PermissionError / NotADirectory exception </END OF COMMENT>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<pyshell#78>", line 1, in <module>
    joblib.dump(oLarge, '20230126T2243CET_o_large.jlib')
  File "C:\Users\myuser\workspace-python\Environs\mylib\lib\site-packages\joblib\numpy_pickle.py", line 553, in dump
    NumpyPickler(f, protocol=protocol).dump(value)
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 487, in dump
    self.save(obj)
  File "C:\Users\myuser\workspace-python\Environs\mylib\lib\site-packages\joblib\numpy_pickle.py", line 355, in save
    return Pickler.save(self, obj)
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 603, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 717, in save_reduce
    save(state)
  File "C:\Users\myuser\workspace-python\Environs\mylib\lib\site-packages\joblib\numpy_pickle.py", line 355, in save
    return Pickler.save(self, obj)
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 901, in save_tuple
    save(element)
  File "C:\Users\myuser\workspace-python\Environs\mylib\lib\site-packages\joblib\numpy_pickle.py", line 355, in save
    return Pickler.save(self, obj)
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 603, in save
    self.save_reduce(obj=obj, *rv)
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 717, in save_reduce
    save(state)
  File "C:\Users\myuser\workspace-python\Environs\mylib\lib\site-packages\joblib\numpy_pickle.py", line 355, in save
    return Pickler.save(self, obj)
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 560, in save
    f(self, obj)  # Call unbound method with explicit self
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 971, in save_dict
    self._batch_setitems(obj.items())
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 997, in _batch_setitems
    save(v)
  File "C:\Users\myuser\workspace-python\Environs\mylib\lib\site-packages\joblib\numpy_pickle.py", line 355, in save
    return Pickler.save(self, obj)
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\pickle.py", line 578, in save
    rv = reduce(self.proto)
  File "C:\Users\myuser\workspace-python\Environs\mylib\lib\site-packages\openTSNE\nearest_neighbors.py", line 353, in __getstate__
    b64_index = base64.b64encode(f.read())
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\tempfile.py", line 826, in __exit__
    self.cleanup()
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\tempfile.py", line 830, in cleanup
    self._rmtree(self.name)
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\tempfile.py", line 812, in _rmtree
    _shutil.rmtree(name, onerror=onerror)
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 740, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 618, in _rmtree_unsafe
    onerror(os.unlink, fullname, sys.exc_info())
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\tempfile.py", line 804, in onerror
    cls._rmtree(path)
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\tempfile.py", line 812, in _rmtree
    _shutil.rmtree(name, onerror=onerror)
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 740, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 599, in _rmtree_unsafe
    onerror(os.scandir, path, sys.exc_info())
  File "C:\Users\myuser\AppData\Local\Programs\Python\Python39\lib\shutil.py", line 596, in _rmtree_unsafe
    with os.scandir(path) as scandir_it:
NotADirectoryError: [WinError 267] Nazwa katalogu jest nieprawid?owa: 'C:\\Users\\myuser\\AppData\\Local\\Temp\\tmpwteculmb\\tmp.ann'

<COMMENT> This one is in Polish again. It says "Folder name is not correct" </END OF COMMENT>

>>> 

[davetapley]

I just ran in to working a separate logging issue.
Windows, Python 3.11.4. MRE here.

[zenquiorra]

Is there any workaround for this issue?

[serhiy-storchaka]

Fixed in 3.11-3.13.