Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
Use atomic file writing and updating for configuration and key files #1060
Had forgot about it, but here's a draft I had laying around from a month ago or so. Feel free to recycle :)
class SaveFile: """ Update file contents atomically. Must be used as a context manager. On a journaling file system the file contents are always updated atomically and won't become corrupted, even on pure failures or crashes (for caveats see SyncFile). """ SUFFIX = '.tmp' def __init__(self, path): self.path = path self.tmppath = os.path.join(self.path, self.SUFFIX) def __enter__(self): # XXX use platform-specific SyncFile self.fd = open(self.tmppath, 'xb') return self.fd def __exit__(self, exc_type, exc_val, exc_tb): self.fd.close() if exc_type is not None: os.unlink(self.tmppath) return os.rename(self.tmppath, self.path) # XXX use platform-specific sync_dir sync_dir(os.path.dirname(self.path)) def write(self, data): self.fd.write(data)
(this would go into borg.platform.base)
While doing "corruption testing" (results below) with various file systems (NTFS-3g, ext4, XFS)  I noticed that given just the right sequence of operations  one can manage to put some data in a freshly created repository on XFS and then lose the repo config rendering the repo unusable until an advanced power-user re-creates the config file.
I don't think this is a critical/big problem in normal usage patterns since the first create after an init will almost certainly take longer than the dirty-writeout timeout of the kernel (one to a couple minutes).
 borg init, borg create small archive (<1 MB in my case), borg create big archive, pull drive during create.
added a commit
Jul 9, 2016
referenced this issue
Jul 26, 2016
I really don't know. That live/visible metadata refers to non-existing MFT entries suggests (in my mind) that some basic precondition of the journaling must've been broken. I ATM don't have any mag drives I'd be willing to use for testing this, but when I get some I'll test more hardware wrt. this bug. (And take it upstream, if they have a proper list/bugtracker)