Atomic function to rename a file

[vstinner]

BPO	8828
Nosy	@pitrou, @vstinner, @giampaolo, @tarekziade, @merwok, @bitdancer, @Trundle
Files	atomic_move_file-windows.py osreplace.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = <Date 2012-01-30.21:13:07.802>
created_at = <Date 2010-05-26.23:41:40.453>
labels = ['type-feature', 'library', 'OS-windows']
title = 'Atomic function to rename a file'
updated_at = <Date 2012-01-30.21:13:07.801>
user = 'https://github.com/vstinner'

bugs.python.org fields:

activity = <Date 2012-01-30.21:13:07.801>
actor = 'pitrou'
assignee = 'none'
closed = True
closed_date = <Date 2012-01-30.21:13:07.802>
closer = 'pitrou'
components = ['Library (Lib)', 'Windows']
creation = <Date 2010-05-26.23:41:40.453>
creator = 'vstinner'
dependencies = []
files = ['17603', '24328']
hgrepos = []
issue_num = 8828
keywords = ['patch']
message_count = 47.0
messages = ['106587', '106599', '106600', '106601', '106602', '106603', '106604', '106605', '106608', '106609', '107070', '107257', '107264', '107269', '107433', '107434', '107449', '143995', '144011', '146226', '146273', '146274', '146280', '146282', '146307', '150149', '150166', '150168', '150170', '150175', '150176', '150177', '150179', '150180', '150181', '150182', '150183', '150186', '150188', '150192', '150193', '150194', '150195', '150196', '152019', '152348', '152349']
nosy_count = 11.0
nosy_names = ['pitrou', 'vstinner', 'draghuram', 'techtonik', 'giampaolo.rodola', 'tarek', 'eric.araujo', 'r.david.murray', 'Trundle', 'neologix', 'python-dev']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'resolved'
status = 'closed'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue8828'
versions = ['Python 3.3']

[vstinner]

os.rename() is atomic on Linux, but on Windows it raises an error if the destination does already exist.

Not atomic pseudo-code for Windows:
if exists(b):
unlink(b)
rename(a, b)

Windows offers different functions depending on the version:

• MoveFileTransacted(): atomic! version >= (Windows Vista, Windows Server 2008)
• ReplaceFile(): version >= Windows 2000
• MoveFileEx() with MOVEFILE_REPLACE_EXISTING and MOVEFILE_WRITE_THROUGH flags: not atomic (eg. "If the file is to be moved to a different volume, the function simulates the move by using the CopyFile and DeleteFile functions."), version >= Windows 2000

I don't think that it's possible to write an atomic rename (file) function for any OS, so it's only a "best effort" atomic function. The documentation will give a list of OS on which the operation *is* atomic (eg. Linux).

Note: os.rename() uses MoveFileW() on Windows.

[vstinner]

A first implementation can be:

if os.name in ('nt', 'ce'):
   def atomic_rename(a, b):
      if os.path.exists(b):
         unlink(b)
      rename(a, b)
else:
   atomic_rename = os.rename

This implementation is atomic on POSIX, and not atomic on Windows. Tell me if I am wrong.

It can be improved later by adding the support of better Windows functions.

[vstinner]

See issue bpo-8604: proposal of a new "with atomic_write(filename) as fp: ..." context manager.

[vstinner]

Begin by removing the dest file is maybe not the safer approach :-) Here is a new try: begin by renaming the dest file to a new file.

------

# use maybe a PRNG instead of a dummy counter or tempfile
def _create_old_filename(filename):
   old = filename + '.old'
   index = 2
   while os.path.exists(old):
      old = filename + '-%s.old' % index
      index += 1
   return old

if os.name in ('nt', 'ce'):
   def atomic_rename(src, dst):
      if os.path.exists(dst):
         old = _create_old_filename(dst)
         rename(dst, old)
         rename(src, dst)
         unlink(old)
      else:
         rename(src, dst)
else:
   atomic_rename = os.rename

---

What can we do if "rename(src, dst)" fails?

[vstinner]

This implementation is atomic on POSIX, ...

Wrong :-)

"On how rename is broken in Mac OS X"
http://www.weirdnet.nl/apple/rename.html

"Update January 8, 2010: ... the original bug (5398777) was resolved in Snow Leopard but the issue I reported was not fixed. ..."

[vstinner]

We have to think about network file systems like NFS. Gnome (nautilus) had a bug on rename because NFS emitted a delete notification on a rename:
http://linux-nfs.org/pipermail/nfsv4/2009-March/010134.html
https://bugzilla.gnome.org/show_bug.cgi?id=575684

It looks like rename is atomic, it's just a bug about notification. But other virtual file systems may not implement atomic rename (eg. is rename atomic with sshfs?). Should Python detect the file system type to choose the algorithm? I would like to say no, because I consider that as a file system (or kernel) bug, not a Python bug.

--

Should we also implement a atomic version of shutil.move()? Support rename if the source and the destination are on different file systems. Or is shutil.move() already atomic?

Note: this issue is only about renaming a *file*. Atomic rename of a directory is much more complex :-)

[pitrou]

It seems you are proposing to call "atomic" something which isn't atomic:

   def atomic_rename(src, dst):
      if os.path.exists(dst):
         old = _create_old_filename(dst)
         rename(dst, old)
         rename(src, dst)
         unlink(old)

[vstinner]

@pitrou: Yes, as I wrote: it's not possible to write an atomic function for all OS. The documentation must give a list of the OS on which the function is atomic. Would you prefer to not define the function instead of writing a pseudo-atomic function?

--

Java bug opened in 1996, closed with "Will Not Fix", last comment ("i am facing similar issue...") in 2008:
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4017593

"How to do atomic writes in a file" in a MSDN blog:
http://blogs.msdn.com/b/adioltean/archive/2005/12/28/507866.aspx

Extract: "Sometimes shell operations like Delete, Rename can fail for various reasons. For example, it might just happen that an antivirus or content indexing application randomly scans the whole file system once in a while. So, potentially, the file Foo.Tmp.txt will be opened for a short period which will cause ... failed delete. And, not only that, but also Rename can fail if the old file already exists, and someone has an open handle on it."

To avoid that "antivirus or content indexing application" open the file, we may need to use a lock on the files.

[pitrou]

Would you prefer to not define the function instead of writing a pseudo-
atomic function?

Your current implementation is useless, since it doesn't achieve anything new.
Besides, if the function isn't atomic, don't name it atomic_XXX.

[vstinner]

Antoine Pitrou <pitrou@free.fr> added the comment:
> Would you prefer to not define the function instead of writing a pseudo-
> atomic function?

Your current implementation is useless, since it doesn't achieve anything
new. Besides, if the function isn't atomic, don't name it atomic_XXX.

Someone may reimplement it with unlink+rename which is worse :-)

But ok, you prefer to not define the function if no real atomic implementation
can be written.

[bitdancer]

Victor: you could always name it best_effort_at_atomic_rename :)

[techtonik]

Atomic file renames are reimplemented by:

1. Mercurial
2. Trac
3. Dulwich (aka HgGit)

and essential for many other projects out there, so this should be in standard library. Atomic renames depend on filesystem more that on the OS.

[vstinner]

Implementation of "atomic" rename() for Windows in Mercurial:
http://selenic.com/repo/hg/file/tip/mercurial/windows.py
(search for "def rename")

[techtonik]

Dulwich bug discussion (closed) - https://bugs.edge.launchpad.net/dulwich/+bug/557585

Trac implementation - http://trac.edgewall.org/browser/trunk/trac/util/__init__.py?#L82

Stackoverflow - http://stackoverflow.com/questions/167414/is-an-atomic-file-rename-with-overwrite-possible-on-windows

[vstinner]

About the function names:

• shutil.atomic_move_file(): only available on some OS
• shutil.move_file(): use shutil.atomic_move_file() if available, or fall back to a best effort implementation

Implement an atomic function to rename a directory is more complex and should be done in another issue. That's why I added "_file" suffix.

[vstinner]

[atomic_move_file-windows.py]: implementation of atomic_move_file() for Windows, depends on CreateTransaction() and MoveFileTransacted(), only available on Windows Vista, Windows Server 2008, or more recent version. This function *is* atomic.

This function is also able to rename a directory!

[techtonik]

Does it work with FAT32 or network filesystem?

[vstinner]

According to the following article, a fsync is also needed on the directory after a rename. I don't understand if is it always needed for an atomic rename, or if we only need it for the "atomic write" pattern.

http://lwn.net/Articles/457667/

"The more subtle usages deal with newly created files, or overwriting existing files. A newly created file may require an fsync() of not just the file itself, but also of the directory in which it was created (since this is where the file system looks to find your file). This behavior is actually file system (and mount option) dependent. You can either code specifically for each file system and mount option combination, or just perform fsync() calls on the directories to ensure that your code is portable."

[neologix]

According to the following article, a fsync is also needed on the
directory after a rename. I don't understand if is it always needed for
an atomic rename, or if we only need it for the "atomic write" pattern.

It's not needed if you just want atomicity, i.e. the file is visible either under its old name or its new name, but not neither or both.
If is however needed if you want durability, i.e. you want to guarantee that the file is visible under its new name after your atomic_rename returns.

[neologix]

The recent issue bpo-13146 renewed my interest, so I'd like to make this move forward, since I think an atomic rename/write API could be quite useful.
Issue bpo-8604 (Adding an atomic FS write API) can be achieved relatively easily with the typical (fsync() left aside)

• create temporary file
• write to the temp file
• atomically rename the temp file to the target path

But the problem is that rename is only atomic on POSIX, and not on Windows.
So I'd suggest to:

• rename this issue to target specifically Windows ;-)
• add MoveFileTransacted to the standard library (PC/msvcrtmodule.c, posixmodule?)

I'm -1 on exposing a "best effort" atomic rename/file API: either the OS offers the primitives necessary to achieve atomicity, or it doesn't. It's better to have a working implementation on some OSes than a flaky implementation on every OS.

Note that I'll happily take over the atomic file API part (issue bpo-8604), but since my Windows kung-fu is so poor, it'd be nice if someone with some Windows experience could tackle this MoveFileTransacted

[pitrou]

MoveFileTransacted is only available under Vista or later. You should be able to use MoveFileEx for the same effect.

[pitrou]

"The solution? Let's remember that metadata changes are atomic. Rename is such a case."

This is from a MSDN blog, I would hope he knows what he's talking about.
http://blogs.msdn.com/b/adioltean/archive/2005/12/28/507866.aspx

(MoveFileEx appears in Windows XP, which is fine: in Python 3.3, "Windows 2000 and Windows platforms which set COMSPEC to command.com are no longer supported due to maintenance burden")

[neologix]

MoveFileTransacted is only available under Vista or later. You should be able to use MoveFileEx for the same effect.

Nice.

"The solution? Let's remember that metadata changes are atomic. Rename is such a case."

Hmmm.
Is he referring to the "standard" rename? The blog doesn't evoke a
specific function, but if it was the case, then why bother at all?

By the way:
"""

• MoveFileEx() with MOVEFILE_REPLACE_EXISTING and
  MOVEFILE_WRITE_THROUGH flags: not atomic (eg. "If the file is to be
  moved to a different volume, the function simulates the move by using
  the CopyFile and DeleteFile functions."), version >= Windows 2000
  """

There's exactly the same limitation with the POSIX version (except
that it'll fail with EXDEV instead of silently doing the copy+unlink).

[pitrou]

> "The solution? Let's remember that metadata changes are atomic. Rename is such a case."
>

Hmmm.
Is he referring to the "standard" rename? The blog doesn't evoke a
specific function, but if it was the case, then why bother at all?

Standard rename (MoveFile) fails when the target exists, and that's
AFAICT the whole problem with it. MoveFileEx allows to overwrite the
target with MOVEFILE_REPLACE_EXISTING.

By the way:
"""

• MoveFileEx() with MOVEFILE_REPLACE_EXISTING and
  MOVEFILE_WRITE_THROUGH flags: not atomic (eg. "If the file is to be
  moved to a different volume, the function simulates the move by using
  the CopyFile and DeleteFile functions."), version >= Windows 2000
  """

There's exactly the same limitation with the POSIX version (except
that it'll fail with EXDEV instead of silently doing the copy+unlink).

If you don't specify the MOVEFILE_COPY_ALLOWED flag, MoveFileEx also
fails. I don't understand what Victor was trying to say.

[pitrou]

I see that Sun/Oracle Java trusts MoveFileEx to do atomic renames:

290 // atomic case
291 if (atomicMove) {
292 try {
293 MoveFileEx(sourcePath, targetPath, MOVEFILE_REPLACE_EXISTING);
294 } catch (WindowsException x) {
295 if (x.lastError() == ERROR_NOT_SAME_DEVICE) {
296 throw new AtomicMoveNotSupportedException(
297 source.getPathForExceptionMessage(),
298 target.getPathForExceptionMessage(),
299 x.errorString());
300 }
301 x.rethrowAsIOException(source, target);
302 }
303 return;
304 }

(from http://www.docjar.com/html/api/sun/nio/fs/WindowsFileCopy.java.html )

[pitrou]

So how about providing a new public os module function doing a rename-with-overwrite on all platforms?
We could name it e.g. os.replace().
os.rename_overwrite() is another possibility, more explicit but also longer.

[bitdancer]

What is the motivation for providing a new function?

[pitrou]

What is the motivation for providing a new function?

Because changing os.rename would break compatibility.

[bitdancer]

Ah, I see, people may be depending on rename on Windows not overwriting.

I suppose a new function (and eventually deprecating the old?) would be the most straightforward way forward, though I dislike the necessity :)

An alternative might be a flag on rename: overwrite=['always', 'os_default'], with a warning and a switch of the default in a subsequent release. That's ugly too, of course. The existing per-platform variation in behavior of rename gives us a mess to deal with.

[neologix]

I'd prefer an optional flag to rename() too.
I really don't like having different functions that achieve the same thing.
It's not obvious to infer from 'replace' its real intent, since it
doesn't match any standard syscall/library.
Ideally, this should be made an option to rename(), especially since
on Unix this will just perform a standard rename.
Another advantage of options over new functions is that it reduces
boilerplate code (i.e. argument parsing, addition to posix_methods,
repeating OS idiosyncrasies/conditional compilation blocks, docstring,
documentation block...).
But I remember Martin thinks that the os module should just be a thin wrapper around underlying syscalls/libraries (but we already have listdir() and friends).

[techtonik]

os.rename(overwrite=True) to produce consistent cross-platform behavior.

[pitrou]

An alternative might be a flag on rename: overwrite=['always', 'os_default']

How about overwrite=[None, True] with None meaning "OS default"?

with a warning and a switch of the default in a subsequent release.

I think we should be conservative with warnings and
compatibility-breaking changes. In this case there's no pressing need to
change behaviour: the default isn't less secure or less efficient.

[techtonik]

One of the Python advantages is providing predictable cross-platform behavior. If we can't introduce nice API without BC break, it is not a reason to introduce ulgy API.

[bitdancer]

I'm good with None/True, but that would imply that for posix rename we'll need to implement the overwrite=False option...which would be a nice thing (the shell mv command has -i for that).

I think a warning would be good, because a unix programmer will assume rename will work the same on windows as it does on posix, and vice versa for a windows programmer. I suppose the fact that we haven't gotten many (any?) complaints about it means it isn't that common a problem, though, so I don't feel strongly about it.

[pitrou]

I'm good with None/True, but that would imply that for posix rename
we'll need to implement the overwrite=False option...which would be a
nice thing (the shell mv command has -i for that).

My point was rather to forbid False as a value (on all OSes) :)

I think a warning would be good, because a unix programmer will assume
rename will work the same on windows as it does on posix, and vice
versa for a windows programmer.

It is already documented. I don't think we want to add a warning for
every documented peculiar piece of behaviour.

[neologix]

How about overwrite=[None, True] with None meaning "OS default"?

+1.

One of the Python advantages is providing predictable cross-platform
behavior. If we can't introduce nice API without BC break, it is not
a reason to introduce ulgy API.

We cannot make rename() overwrite existing files by default (on Windows).
It's out of question, too much code might rely on this, and this may very well introduce security flaws.
If you're concerned with the semantics difference between platforms, well, there's not much we can do about it now. Java behaves in the same way, for example.

I'm good with None/True, but that would imply that for posix rename
we'll need to implement the overwrite=False option...which would be a
nice thing (the shell mv command has -i for that).

Why?
The problem is that it's simply impossible to implement reliably.
I didn't check mv source code, but it must be using something like:

if '-i' and os.path.exists(target_path):
    continue
os.rename(src_path, target_path).

But there's a TOCTTOU race.

[bitdancer]

Ah, you are right about the race of course. So yes, I agree with your proposal. It's a weird API, but probably the best we can do.

[techtonik]

2011/12/23 Charles-François Natali <report@bugs.python.org>

> One of the Python advantages is providing predictable cross-platform
> behavior. If we can't introduce nice API without BC break, it is not
> a reason to introduce ulgy API.

We cannot make rename() overwrite existing files by default (on Windows).
It's out of question, too much code might rely on this, and this may very
well introduce security flaws.
If you're concerned with the semantics difference between platforms, well,
there's not much we can do about it now. Java behaves in the same way, for
example.

I propose quite the opposite. rename() should not overwrite existing files
by default.

[neologix]

I propose quite the opposite. rename() should not overwrite existing
files by default.

1. That's not what I understood from:

os.rename(overwrite=True) to produce consistent cross-platform
behavior.

2. The above argument on backward incompatible change applies in exactly the same way (just exchange Windows for POSIX).

3. As explained above, it can not be done reliably on POSIX (TOCTTOU race).

[techtonik]

2011/12/23 Charles-François Natali <report@bugs.python.org>

Charles-François Natali <neologix@free.fr> added the comment:

> I propose quite the opposite. rename() should not overwrite existing
> files by default.

1. That's not what I understood from:
   > os.rename(overwrite=True) to produce consistent cross-platform
   > behavior.

2. The above argument on backward incompatible change applies in exactly
   the same way (just exchange Windows for POSIX).

3. As explained above, it can not be done reliably on POSIX (TOCTTOU race).

os.rename(overwrite=False) by default will do less harm than the opposite,
so I'd say it is a way to go even if it can not be reliably done on POSIX.
But I believe correct file system that supports transactions will make it
possible to do reliably on POSIX too.

Then I believe that having a small chance of overwriting file just created
at exactly the same moment on a POSIX is a small price to pay for function
that does safety check before rename on a platform that doesn't have such
functionality at all. If you need this functionality - you will still do
'if exists ... rename' and the TOCTTOU race will affect you. So, the only
exit is to place a little sign "in some extreme cases Linux suffers from
TOCTTOU problem when renaming files without overwriting, so keep that in
mind".

BUT let me remind you that this bug is about "Atomic rename" which should
rollback in case of errors.

[pitrou]

os.rename(overwrite=False) by default will do less harm than the opposite,

Disagreed.

Then I believe that having a small chance of overwriting file just created
at exactly the same moment on a POSIX is a small price to pay for function
that does safety check before rename on a platform that doesn't have such
functionality at all.

Disagreed.
If you need the functionality, it's *one* additional line of code. Not
every trivial function deserves to be in the stdlib.

[techtonik]

On Fri, Dec 23, 2011 at 10:35 PM, Antoine Pitrou <report@bugs.python.org>wrote:

> os.rename(overwrite=False) by default will do less harm than the
opposite,

Disagreed.

Fine. No arguments == no consensus.

> Then I believe that having a small chance of overwriting file just
created
> at exactly the same moment on a POSIX is a small price to pay for
function
> that does safety check before rename on a platform that doesn't have such
> functionality at all.

Disagreed.
If you need the functionality, it's *one* additional line of code. Not
every trivial function deserves to be in the stdlib.

As a Windows programmer I am quite surprised to read this thread with
information that on Linux os.rename() overwrites files without questions,
so as I Windows programmer I want os.rename() to stop that. I always guard
my code against accidental rewrite by catching the exception. EAFP and all
that stuff. But there is no way I can ask forgiveness when files are
already overwritten.

[bitdancer]

So maybe my warning idea isn't such a bad idea :)

As a unix programmer, I was very surprised to read in this thread that Windows doesn't overwrite the file on rename. As a unix programmer, I don't check for errors on a rename, because I expect it to just work. I'd like the windows rename call to stop throwing errors if the file exists, it breaks my programs if they run on windows.

(Actually, very few of my programs ever get run on Windows, but you get the idea.)

Thus, the only possible course is to maintain backward compatibility, and allow the programmers who care to specify the desired behavior. Since one of the important aspects of 'rename' in unix programmers' minds is that it is atomic, a race condition is not acceptable, therefore overwrite=False is not an acceptable option for os.rename.

[pitrou]

As a Windows programmer I am quite surprised to read this thread with
information that on Linux os.rename() overwrites files without questions,
so as I Windows programmer I want os.rename() to stop that.

As a Windows programmer, you are not really qualified to criticize the
behaviour of Unix systems.
As a Linux programmer, I don't want to see the default behaviour
changed.

No arguments == no consensus.

We don't strive to achieve consensus. Agreement among the majority of
core developers is enough, and you aren't part of the population anyway.

[pitrou]

Here is a patch adding os.replace().

[python-dev]

New changeset 80ddbd822227 by Antoine Pitrou in branch 'default':
Issue bpo-8828: Add new function os.replace(), for cross-platform renaming with overwriting.
http://hg.python.org/cpython/rev/80ddbd822227

[pitrou]

os.replace() committed in 3.3!