Add support for "snapshot"

[gordonmessmer]

https://bitbucket.org/gordonmessmer/dragonsdawn-snapshot

Similar to the existing LVM snapshot support with some notable improvements. This works both locally and for ssh backups. This supports syncing data to disk (such as making SQL database files consistent) before backup. This supports filesystems other than ext4 on LVM (such as btrfs, Gluster, and NetApp NFS). This is also aware of and will avoid snapshots that can potentially destroy a filesystem (such as making a snapshot of an ext filesystem with an external journal).

[bebehei]

The coding-style of your pull-request is very good! I like it.

There are few objections, what let me reject this PR.

Your code follows a quite simple approach: Snapshotting every backup, rsync every backup, unmount every backup. On the one hand side, his will give the ability to ensure that the backup is actually made (almost) directly at the execution time. On the other hand, there is a snapshot, which will lay around during the whole backup process. Also if a backup stops, rsnapshot may be forced to remove snapshots, which were never used and the snapshot had been unneccessary.

I've got another objection: By merging this, I don't want to draw a dependency to your tool with the cmd_snapshot configuration-option. There needs to be information added to added, that others can write their own snapshot-programs without using your tool.

I like the idea to add a cmd_snapshot-option in general. I imagine to define an API, which applies to rsnapshot and you stick with your tool to the API, too. So other users don't depend on your tool and are able to write their own snapshot-commands.

[gordonmessmer]

I think you and I have similar concerns about dependencies. I actually wrote "snapshot" to establish an API and provide a proof of concept implementation with the hope that distributions and non-Linux POSIX systems would develop their own preferred implementations.

snapshot has its own documentation to describe the implementation, but in brief it looks like this: The snapshot itself should be handled by a process external to the backup tool. Obviously, that allows better code reuse, but it also provides the important function of cleaning up the snapshots if the backup process crashes or exits for any reason. The snapshot tool is invoked, optionally with the path to a mount point for the snapshots, and with the paths to data that will be backed up. Any daemons which have scripts installed will be instructed to quiesce data. Snapshots will be made and mounted. Daemons will then be instructed to resume operation. At this point snapshot closes its stdout fd to signal that the backup can be made. When its stdin fd closes, it will unmount and remove the snapshots.

So, to your points:

On the other hand, there is a snapshot, which will lay around during the whole backup process

Well, yes. If it's not around for the whole backup process, it couldn't be backed up.

It's actually important that all of the snapshots are made at once for the same reason that it's important to make snapshots at all: if you back up different parts of the system individually, then they may not be consistent. A daemon might write data to one of the filesystems that you back up later that makes it inconsistent with the filesystem or path that was backed up earlier.

What you've described is an intentional feature, necessary for consistent backups. It is not a flaw.

Also if a backup stops, rsnapshot may be forced to remove snapshots, which were never used and the snapshot had been unneccessary.

I think this is addressed in the same way. All of the paths need to be snapped at once for consistency's sake. The possibility of early termination does not eliminate that requirement.

There needs to be information added to added, that others can write their own snapshot-programs without using your tool.

Do you mean documentation in rsnapshot? I could readily contribute some of snapshot's documentation, or a pointer to documentation, so that alternate implementations can be written. The snapshot tool already has documentation that I believe is suitable for developing alternate implementations. And again, that is something that I really want to encourage. Requirements vary from system to system, and I would like to see a standard interface for making snapshots for backups. POSIX systems have been without one for far too long.

Please consider merging this pull request. Doing so helps establish an API for making consistent system backups on POSIX systems. Adding this feature does not remove rsnapshot's existing lvm snapshot feature, but offers a variety of improvements that I outlined initially. It supports remote systems, where rsnapshot can only currently snapshot local filesystems. It supports non-LVM filesystem types, where rsnapshot currently does not. Finally, it is aware of conditions in which making snapshots might corrupt the system, which rsnapshot currently is not.

-- edit at 2015-04-10 putting the quotes into markdown-quotes //bebehei

[bebehei]

After reading through your answer, I see there much consensus. Right now, I'm just unable to answer fully and will do this later these days. Sorry. If I forgot, drop a "ping"-line here and I will respond.

[gordonmessmer]

Thanks. I'll ping you next Friday if I don't hear anything.

[sgpinkus]

Providing an an abstract API / CLI for taking a snapshots in a technology independent way may be a valid goal. Is it valid to make rsnapshot dependent on this API though? Wouldn't it be more generally useful to just support arbitrary pre and post backup actions better?

Requirements vary from system to system, and I would like to see a standard interface for making snapshots for backups. POSIX systems have been without one for far too long.

Sure.

snapshot will first run each of the scripts in its "writers.d" directory, signaling to applications that they should make their data consistent and stop writing new data.

"stop writing new data" is never going to work as a general solution.

[gordonmessmer]

Wouldn't it be more generally useful to just support arbitrary pre and post backup actions better?

The API offered by "snapshot" keeps a pipe open between the backup application and the snapshot application, so that if the backup application crashes or otherwise exits, the snapshots can be closed. You cannot offer such a guarantee if you rely on a post-execution script or command. Moreover, rsnapshot runs pre/post commands per filesystem rather than per host, which is inadequate to making consistent snapshots.

For instance, if your SQL server has data files on one set of disks (filesystem) and write-ahead logs on a separate set of disks (and hence, a separate filesystem), you really want both of those filesystems to be snapped at the same time for consistency.

"stop writing new data" is never going to work as a general solution.

Every production snapshot system that I'm aware of relies on the same execution pattern that "snapshot" uses. Instruct daemons to make their data consistent, snapshot the filesystems, and then instruct the daemon to continue. Daemons are quiescent for a few seconds.

rsnapshot's existing lvm snapshot doesn't do that, which means that such backups will have consistent filesystems, but not necessarily consistent data. An active SQL database, for instance, would probably have inconsistent data in a backup made by rsnapshot with its current lvm snapshot support.

[sgpinkus]

The API offered by "snapshot" keeps a pipe open between the backup application and the snapshot application, so that if the backup application crashes or otherwise exits, the snapshots can be closed.

What your saying is we need to add specific support this stdin/stdout open file handle signalling protocol to external snapshot like applications? It solves some problem that cannot be solved otherwise. That problem is rsnapshot crashing with open snapshots. What if the snapshot program crashes?

I still think pre post actions are a better more generally useful solution, an should be used if possible.

Moreover, rsnapshot runs pre/post commands per filesystem rather than per host, which is inadequate to making consistent snapshots.

I don't get this. How is it specifically per filesystem?

Every production snapshot system that I'm aware of relies on the same execution pattern that "snapshot" uses. Instruct daemons to make their data consistent, snapshot the filesystems, and then instruct the daemon to continue. Daemons are quiescent for a few seconds.

I don't doubt you. Could you provide some references to the system you are thinking of for my personal benefit.

[bebehei]

Moreover, rsnapshot runs pre/post commands per filesystem rather than per host, which is inadequate to making consistent snapshots.

I don't get this. How is it specifically per filesystem?

The pre/post hooks are running before and after the whole backup-process. There is nothing per-filesystem and nothing per host.

[gordonmessmer]

The pre/post hooks are running before and after the whole backup-process

My recollection is flawed. I had thought I'd seen evidence otherwise.

[gordonmessmer]

What your saying is we need to add specific support this stdin/stdout open file handle signalling protocol to external snapshot like applications?

I'm saying that rsnapshot currently has a very naive lvm snapshot facility and that rather than massively bloat rsnapshot with a better implementation, a small patch allows it to use an external implementation that can be reused by other backup systems as well. The interface to "snapshot" uses only the most basic POSIX features, and is simple enough to implement in bash.

It solves some problem that cannot be solved otherwise. That problem is rsnapshot crashing with open snapshots. What if the snapshot program crashes?

The snapshot application reads from stdin until that file closes. It is relatively unlikely that a process would crash during a simple read() on stdin.

The backup application, on the other hand, has to handle memory allocation errors, filesystem IO errors, network protocol errors, etc.

I still think pre post actions are a better more generally useful solution, an should be used if possible.

Sure, and this patch doesn't remove that option for users who agree with you. Anyone who wants to do that can. This patch isn't an alternative or replacement for pre/post exec. Instead, it's an alternative to the LVM snapshot implementation in rsnapshot. Unlike that code, "snapshot" handles remote systems, it handles non-LVM filesystems, it makes data consistent on disk, and it cleans up snapshots if the backup process crashes or is interrupted/disconnected.

Essentially, I would ask: if pre/post exec were good enough as a general solution, why was LVM snapshot support added to rsnapshot? Everything it does could have been done just as well in pre/post exec scripts.

Could you provide some references to the system you are thinking of for my personal benefit.

Windows VSS would be the first to come to mind.

Snapshot scripts for postgresql:
https://code.google.com/p/pglvmbackup/

Snapshot scripts for MySQL:
http://www.lenzg.net/mylvmbackup/

[bebehei]

The pre/post hooks are running before and after the whole backup-process

My recollection is flawed. I had thought I'd seen evidence otherwise.

No, look at #74. A seperate configuration-option is neccessary.

I think you and I have similar concerns about dependencies. I actually wrote "snapshot" to establish an API and provide a proof of concept implementation with the hope that distributions and non-Linux POSIX systems would develop their own preferred implementations.

ACK, full ack on the bold

Providing an an abstract API / CLI for taking a snapshots in a technology independent way may be a valid goal.

Full ACK

Requirements vary from system to system, and I would like to see a standard interface for making snapshots for backups. POSIX systems have been without one for far too long.

Full ACK.

snapshot has its own documentation to describe the implementation, but in brief it looks like this:

The only information I find, is your README-file. It's multiple pages long, but it's far behind an API specification. Many lines are about snapshots in general only a few about how your implemenation works, and none about an API/standard to follow. TBH I was too lazy to read the whole document. Who else will do this to have a snapshot-feature?

I believe you both agree with me, that rsnapshot should support an API to make data consistent. "snapshot" could be an implementation of this. But it's not an API, neither there are standards or documents describing such an API.

I really welcome a PR, which gives support to such an API. providing, an option like @gordonmessmer referenced a cmd_snapshot option and talks to a program, which snapshots the filesystem.

But why do I not want to merge it? It's not an API, it's a possible implementation of a not-yet-existing API. So if I would merge, rsnapshot has got a direct dependency to your program (mind: not API). Who can guarantee, that your program will get right maintainance? I don't see the relevance to add your program.

So: Come up with an API document and we can discuss further. There is no need to have a big document covering everything, but it has to outline the call of the snapshotting program.