tempfile.NamedTemporaryFile: automatic cleanup by OS

[rphilips]

BPO	4928
Nosy	@loewis, @pitrou, @vadmium, @catherinedevlin, @eryksun, @WhyNotHugo
PRs	bpo-4928: Document NamedTemporaryFile non-deletion after SIGKILL #26198

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2009-01-13.09:22:45.685>
labels = ['type-bug', '3.8', '3.9', '3.10', 'library', 'docs']
title = 'tempfile.NamedTemporaryFile: automatic cleanup by OS'
updated_at = <Date 2021-05-17.20:18:59.095>
user = 'https://bugs.python.org/rphilips'

bugs.python.org fields:

activity = <Date 2021-05-17.20:18:59.095>
actor = 'Catherine.Devlin'
assignee = 'docs@python'
closed = False
closed_date = None
closer = None
components = ['Documentation', 'Library (Lib)']
creation = <Date 2009-01-13.09:22:45.685>
creator = 'rphilips'
dependencies = []
files = []
hgrepos = []
issue_num = 4928
keywords = ['patch']
message_count = 14.0
messages = ['79727', '79728', '83552', '109477', '109479', '109491', '109492', '109493', '109505', '109559', '260560', '260567', '374055', '389915']
nosy_count = 9.0
nosy_names = ['loewis', 'rphilips', 'pitrou', 'docs@python', 'martin.panter', 'Catherine.Devlin', 'eryksun', 'WhyNotHugo', 'portikhun']
pr_nums = ['26198']
priority = 'normal'
resolution = None
stage = 'patch review'
status = 'open'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue4928'
versions = ['Python 3.8', 'Python 3.9', 'Python 3.10']

[rphilips]

On Solaris 10 (Solaris 10 5/08 s10x_u5wos_10 X86),
with python 2.5 (Python 2.5.2 (r252:60911, Sep 8 2008, 16:53:36) [C] on
sunos5),

tempfile.NamedTemporaryFile creates - as advertised - a temporary file.

After closing this file object (e.g. by exiting the python process), the
temporary file is indeed removed.

But if you just kill the python process, the temporary file is not
destroyed.

This behavior is different on Red Hat Linux and MS Vista, on these OSes
the temporary file is removed.

A snapshot to illustrate the behavior:

[moto /]# python
Python 2.5.2 (r252:60911, Sep  8 2008, 16:53:36) [C] on sunos5
Type "help", "copyright", "credits" or "license" for more information.
>>> import os, tempfile
>>> f = tempfile.NamedTemporaryFile()
>>> f.name
'/tmp/tmpfHraUd'
>>> os.getpid()
4403
>>> x = "With root priveleges, I execute: 'kill 4403'"
>>>
>>> Terminated
[moto /]#

[rphilips]

[Replaces msg 79727]

On Solaris 10 (Solaris 10 5/08 s10x_u5wos_10 X86),
with python 2.5 (Python 2.5.2 (r252:60911, Sep 8 2008, 16:53:36) [C] on
sunos5),

tempfile.NamedTemporaryFile creates - as advertised - a temporary file.

After closing this file object (e.g. by exiting the python process), the
temporary file is indeed removed.

But if you just kill the python process, the temporary file is not
destroyed.

This behavior is different on Red Hat Linux and MS Vista, on these OSes
the temporary file is removed.

A snapshot to illustrate the behavior:

[moto /]# python
Python 2.5.2 (r252:60911, Sep  8 2008, 16:53:36) [C] on sunos5
Type "help", "copyright", "credits" or "license" for more information.
>>> import os, tempfile
>>> f = tempfile.NamedTemporaryFile()
>>> f.name
'/tmp/tmpfHraUd'
>>> os.getpid()
4403
>>> x = "With root priveleges, I execute: 'kill 4403'"
>>>
>>> Terminated
[moto /]# ls /tmp/tmpfHraUd
/tmp/tmpfHraUd
[moto /]#

[pitrou]

Martin, do you have time to confirm this problem?
Failure of (Named)TemporaryFile to remove the file on shutdown should be
considered a serious issue.

[BreamoreBoy]

Shouldn't this get looked at as it's high priority, albeit over a year old?

[skrah]

I can reproduce this on Ubuntu with Python 2.7 and 3.2.

[loewis]

Unfortunately, I typically don't have time to consider the priority of issues.

However, in the specific case, I also fail to see the bug. Where does it say that they are supposed to be deleted when the process is killed, and what mechanism specifically is supposed to actually perform the deletion?

[exarkun]

I can't think of any way that you might be able to implement the behavior being requested here.

Instead, if you don't want to leave files lying around, use TemporaryFile instead of NamedTemporaryFile.

Perhaps the documentation for NamedTemporary file could be updated to explain this limitation, though.

[loewis]

I can't think of any way that you might be able to implement the behavior being requested here.

Thanks for the confirmation; lowering the priority then.

It would, of course, be possible to improve quality by registering an atexit handler. Which, of course, would be futile in case of a SIGKILL or system crash.

[pitrou]

Where does it say that they are supposed to be deleted when the process > is killed,

Well, it is more of a "natural expectation" I guess.

and what mechanism specifically is supposed to actually perform the
deletion?

That's not the kind of questions users ask themselves.
If the process shutdown is greatful, either __del__ or a weakref callback is adequate. We probably can't do anything for brutal killings, though.

[pitrou]

Ok, my bad. The implementation already has a __del__ method.

[vadmium]

I am surprised at the report that Red Hat Linux automatically removed the file. What system calls are involved? On my Arch Linux setup with current 3.6 code it leaves the file behind (same as the Solaris report).

However the the Windows version does automatically deleting the file once all file handles are closed:

# Setting O_TEMPORARY in the flags causes the OS to delete
# the file when it is closed. This is only supported by Windows.

Perhaps this behaviour should be documented.

[eryksun]

Maybe a note could be added to suggest using a SIGTERM signal handler (e.g. the handler could raise a SIGTERM exception):

POSIX: The file will not be deleted if the process is terminated abruptly by a signal. A process may register a SIGTERM handler to ensure that the file is deleted, but SIGKILL cannot be handled. Windows: the file will be deleted when it is closed, even if the process is terminated.

Note that for Windows it is possible to make the file permanent, but it's probably not common enough to bother with documenting this. Currently the documentation states that "[w]hether the name can be used to open the file a second time, while the named temporary file is still open, varies across platforms (it can be so used on Unix; it cannot on Windows NT or later)". Actually on Windows the file can be opened again while the named temporary file is still open. You can use an opener that calls os.open with the O_TEMPORARY flag. But the opener can also call _winapi.CreateFile with read, write, and delete sharing. If DELETE access is requested this handle can be used to make the file permanent via SetFileInformationByHandle.

[WhyNotHugo]

It would seem that the main issue here lies in the documentation not being obvious enough for some of us. If you're familiar with lower level OS APIs, it might be clear, but as an app developer, some things slip by.

The key difference is that TemporaryFile will be erased properly (at least on Linux) even if the process gets a SIGKILL, while NamedTemporaryFile will linger (which fills up my disk every few weeks/months).

As an application developer, this difference is super important to me (I hope I'm not the only one). Would a PR that mentions this explicitly in the docs be an acceptable solution here?

[portikhun]

Hello

I would really appreciate and I think many of us if every time I havent have to remember everything. Im not a Sys admin, devops, or who has in his mind every low level step and just shakes out of his little finger...
Why do I have to take care at the high level where I stand with using Python? I think mustn't.

So yes, as just a little noticement would be really helpful for us to remember we should look forward if we want to make sure. That's why Im here, and not reading the docs instead...

[slateny]

Closing - the linked PR fixes this (see also docs).