Adding an atomic FS write API

[olemis]

BPO	8604
Nosy	@loewis, @warsaw, @ncoghlan, @pitrou, @vstinner, @ericvsmith, @giampaolo, @tarekziade, @merwok, @socketpair, @takluyver, @vadmium
Dependencies	bpo-8828: Atomic function to rename a file
Files	atomic_write.patch atomic_write_mkstemp.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2016-01-20.14:22:34.655>
created_at = <Date 2010-05-03.12:43:57.662>
labels = ['type-feature', 'library']
title = 'Adding an atomic FS write API'
updated_at = <Date 2016-01-20.14:22:34.654>
user = 'https://bugs.python.org/olemis'

bugs.python.org fields:

activity = <Date 2016-01-20.14:22:34.654>
actor = 'vstinner'
assignee = 'none'
closed = True
closed_date = <Date 2016-01-20.14:22:34.655>
closer = 'vstinner'
components = ['Library (Lib)']
creation = <Date 2010-05-03.12:43:57.662>
creator = 'olemis'
dependencies = ['8828']
files = ['30273', '30275']
hgrepos = []
issue_num = 8604
keywords = ['patch']
message_count = 50.0
messages = ['104833', '104834', '104835', '104836', '104837', '104855', '104856', '104857', '104859', '104860', '104862', '104863', '104866', '106585', '106586', '106588', '107427', '135850', '135852', '135912', '135917', '150002', '150036', '150084', '150087', '150145', '150156', '150157', '150158', '150160', '150163', '150164', '150174', '153451', '174104', '189324', '189325', '189329', '189330', '189332', '189333', '189334', '189342', '189382', '256849', '256850', '256851', '256961', '258681', '258684']
nosy_count = 18.0
nosy_names = ['loewis', 'barry', 'exarkun', 'ncoghlan', 'pitrou', 'vstinner', 'eric.smith', 'giampaolo.rodola', 'tarek', 'eric.araujo', 'Arfrever', 'olemis', 'meatballhat', 'milko.krachounov', 'neologix', 'socketpair', 'takluyver', 'martin.panter']
pr_nums = []
priority = 'normal'
resolution = 'out of date'
stage = 'patch review'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue8604'
versions = ['Python 3.4']

[olemis]

Often I have the contents to be written in a file at a given path that
I know as well. I recently tried to find a function in stdlib to do
that and to my surprise this is what I found :

• Such function exists
• It's distutils.file_util.write_file

IMO the last place where people'd look for such a function is inside
distutils package. Besides I reviewed modules listed under File and directory access category in Library Reference and found nothing
even similar.

The idea is to provide a a similar function in shutils module

[pitrou]

This sounds silly to me. You can write a file in two lines:

with open("foo", "wb") as f:
    f.write(contents)

If you want to do something more useful, you can add a function for atomic writing of a file.

[ericvsmith]

Does the standard library really need something so trivial? I'd put it in your own program. And I'd make the one in distutils private (and fix it to use a with statement).

[tarekziade]

@eric: remember that distutils is frozen, so it won't be removed. And historically Distutils code was meant to be 2.4 compatible (thus, no with)

@antoine: Yes, that would be the idea (provide a robust pattern by using a temporary file, then rename it)

[pitrou]

@antoine: Yes, that would be the idea (provide a robust pattern by
using a temporary file, then rename it)

Then perhaps it would be better as a context manager:

with shutil.atomic_write("foo", "wb") as f:
    f.write("mycontents")

[tarekziade]

Well, a context manager sounds overkill since we just want to write some content in a file (and nothing else during that context).

I think a simple call is straightforward:

  shutil.atomic_write("foo", "contents", mode="wb")

[tarekziade]

Notice that "contents" could be replaced here by an iterator, that would return data to send to write()

[pitrou]

Well, a context manager sounds overkill since we just want to write
some content in a file (and nothing else during that context).

Using a context manager, though, is the recommended idiom to write
files. I think there's a value in remaining consistent. We don't want to
end up like PHP which has dozens of different idioms of doing similar
things.

[tarekziade]

This idiom makes sense when you want to do some things with an open file, and replaces the usual try..finally idiom.

That's not what we want to do here. We want to write data in a file in a single step, in an atomic manner.

Giving the ability to the developers to work in a context manager means that you potentially give them the ability to break this atomicity.

So I don't think the context manager idiom prevails, and should be avoided : shutil.atomic_write is asked to write a file, given some data, and don't return until it's done.

[exarkun]

Considering that it's extremely difficult to implement this correctly and in a cross-platform way, I think it makes sense as a stdlib addition (though I'd add it as a method of a path type rather than to the "shell utilities" module ;).