Destructive rotations

[jkufner]

Hello,
rsnapshot backup rotations does not take into account, that backup may fail. There should be check how old is the oldest daily backup and rotate the weekly only if the oldest daily is at least week old. When a lot of backups fail, for example when trying to backup a missing laptop, rotations will eventualy delete all old backups, which is definitely wrong (at when cron is used to rotate as recommended in man page).

I use following script to invoke rsnapshot:

echo "$host: /"

if ping -c 20 -i 0.2 -W 10 -n "$host" >/dev/null
then
        # do backup
        rsnapshot -c "$conf" sync \
        && rsnapshot -c "$conf" daily

        # rotate only older than 6.5 day
        last_daily="`date -r daily.6 +%s || echo 0`"
        first_weekly="`date -r weekly.0 +%s || echo 0`"
        #echo "last_daily - first_weekly = $last_daily - $first_weekly = "$(($last_daily - $first_weekly))
        [ $(( $last_daily - $first_weekly )) -gt 561600 ] \
        && rsnapshot -c "$conf" weekly

        # update symlinks
        rm [0-9][0-9][0-9][0-9]-* 2>/dev/null
        find -mindepth 1 -maxdepth 1 -type d \
        | while read d
        do
                ln -s "$d" "`date -r "$d" +'%Y-%m-%d %H:%M:%S'`"
        done

else
        echo "     ... offline. (No ping response.)"
fi

I use only daily and weekly backups. When backup fails, daily rotation is not performed. When oldest daily backup is older than 6.5 day (to avoid rounding errors when one day is backup faster than other), weekly rotation is performed.

Goal is to have weekly backups at least 6.5 days from each other.

When there is more daily backups than expected, extra daily backups are rotated away. And next weekly backup is created when the rotated-away backup is old enough.

When daily backups are missing, weekly backup is created earlier than after 7 daily rotations. So if laptop is connected in LAN only once a week, daily backups will be in week interval and each daily backup will become a weekly backup. This will remove too old backups.

Final part of the script creates symlinks with date of backup. Very simple and very practical. But a little bit off topic here.

What do you think about real-time based rotations?

[sgpinkus]

I think the sync_first option solves the backup failing problem. Example:

rsnapshot -c backups-rsnapshot.conf sync && \
rsnapshot -c backups-rsnapshot.conf level0

Weekly/higher level rotations will also fail to rotate because there is nothing to rotate if lower levels fails

[jkufner]

It is not that simple :(

Example: 7 daily backups, 4 weekly backups. Backup is done every day (just after midnigt) using && daily rotation, then cron is set to do weekly rotation every monday.

(Notation: backup symbols represent calendar date of their creation = mtime of backup directory. The first is newest. Number is calendar number of the week. The first group are daily backups, the second are weekly backups.)

Day 1 - Friday, 6th week, after successful backup:
  { Fri6, Thu6, Wen6, Tue6, Mon6, Sun5, Sat5 }, { Sun4, Sun3, Sun2, Sun1 }
Day 2 - Saturday, after successful backup, Sat rotated away:
  { Sat6, Fri6, Thu6, Wen6, Tue6, Mon6, Sun5 }, { Sun4, Sun3, Sun2, Sun1 }
Day 3 - Sunday, after weekly rotation, Sun1 rotated away:
  { Sat6, Fri6, Thu6, Wen6, Tue6, Mon6       }, { Sun5, Sun4, Sun3, Sun2 }
Day 3 - Sunday, after successful backup:
  { Sun6, Sat6, Fri6, Thu6, Wen6, Tue6, Mon6 }, { Sun5, Sun4, Sun3, Sun2 }
Day 4 - Monday, 7th week, after successful backup, Mon6 rotated away:
  { Mon7, Sun6, Sat6, Fri6, Thu6, Wen6, Tue6 }, { Sun5, Sun4, Sun3, Sun2 }

So far everything as expected. Lets get unlucky next week, using current implementation (with sync && daily):

Day  8 - Friday, 7th week, after failed backup, no rotation:
  { Thu7, Wen7, Tue7, Mon7, Sun6, Sat6, Fri6 }, { Sun5, Sun4, Sun3, Sun2 }
Day  9 - Saturday, after successful backup, Fri6 rotated away:
  { Sat7, Thu7, Wen7, Tue7, Mon7, Sun6, Sat6 }, { Sun5, Sun4, Sun3, Sun2 }
Day 10 - Sunday, after weekly rotation, Sun2 rotated away:
  { Sat7, Thu7, Wen7, Tue7, Mon7, Sun6,      }, { Sat6, Sun5, Sun4, Sun3 }
Day 10 - Sunday, after successful backup:
  { Sun7, Sat7, Thu7, Wen7, Tue7, Mon7, Sun6 }, { Sat6, Sun5, Sun4, Sun3 }
Day 11 - Monday, 8th week, after successful backup, Sun6 rotated away:
  { Mon8, Sun7, Sat7, Thu7, Wen7, Tue7, Mon7 }, { Sat6, Sun5, Sun4, Sun3 }

Now there is only 6 days between first and second weekly backup.

Let's go back to day 8 and use real-time based implementation instead (weekly rotation is done after the daily rotation):

Day  8 - Friday, 7th week, after failed backup, no rotation:
  { Thu7, Wen7, Tue7, Mon7, Sun6, Sat6, Fri6 }, { Sun5, Sun4, Sun3, Sun2 }
Day  9 - Saturday, after successful backup, Fri6 rotated away:
  { Sat7, Thu7, Wen7, Tue7, Mon7, Sun6, Sat6 }, { Sun5, Sun4, Sun3, Sun2 }
Day 10 - Sunday, after successful backup, Sat6 rotated away:
  { Sun7, Sat7, Thu7, Wen7, Tue7, Mon7, Sun6 }, { Sun5, Sun4, Sun3, Sun2 }
Day 10 - Sunday, after weekly rotation:
  { Sun7, Sat7, Thu7, Wen7, Tue7, Mon7       }, { Sun6, Sun5, Sun4, Sun3 }
Day 11 - Monday, 8th week, after successful backup:
  { Mon8, Sun7, Sat7, Thu7, Wen7, Tue7, Mon7 }, { Sun6, Sun5, Sun4, Sun3 }

Now all weekly backups are nicely in 7 day interval.

Maybe I should write a script to calculate some more interesting examples.

[sgpinkus]

Hey, revisiting this since it was referenced from a new issue. Just wanted to note the sync && rotate I proposed does not solve your issue. If your goal was simply

... to have weekly backups at least 6.5 days from each other.

You would do the opposite. I.e not use sync && rotate, but just always rotate. This gives rise to other issues of course. Specifically you'll have places in your chain where say alpha.N is exactly the same image as alpha.N+1.

What do you think about real-time based rotations?

Think its a big change. My preference right now is KISS.

[jkufner]

Here is little example, the ls of my current backups.

$ LANG=C ls -l
lrwxrwxrwx 1 root root   10 Aug 23 22:47 2015-06-30 22:36:22 -> ./weekly.4
lrwxrwxrwx 1 root root   10 Aug 23 22:47 2015-07-18 02:02:25 -> ./weekly.3
lrwxrwxrwx 1 root root   10 Aug 23 22:47 2015-07-24 22:33:56 -> ./weekly.2
lrwxrwxrwx 1 root root   10 Aug 23 22:47 2015-08-01 22:37:26 -> ./weekly.1
lrwxrwxrwx 1 root root   10 Aug 23 22:47 2015-08-09 02:42:15 -> ./weekly.0
lrwxrwxrwx 1 root root    9 Aug 23 22:47 2015-08-11 23:02:25 -> ./daily.5
lrwxrwxrwx 1 root root    9 Aug 23 22:47 2015-08-12 22:37:47 -> ./daily.4
lrwxrwxrwx 1 root root    9 Aug 23 22:47 2015-08-13 22:41:28 -> ./daily.3
lrwxrwxrwx 1 root root    9 Aug 23 22:47 2015-08-14 22:32:49 -> ./daily.2
lrwxrwxrwx 1 root root    9 Aug 23 22:47 2015-08-15 22:36:31 -> ./daily.1
lrwxrwxrwx 1 root root    9 Aug 23 22:47 2015-08-23 22:40:24 -> ./daily.0
drwxr-xr-x 6 root root 4096 Aug 23 22:40 daily.0
drwxr-xr-x 6 root root 4096 Aug 15 22:36 daily.1
drwxr-xr-x 6 root root 4096 Aug 14 22:32 daily.2
drwxr-xr-x 6 root root 4096 Aug 13 22:41 daily.3
drwxr-xr-x 6 root root 4096 Aug 12 22:37 daily.4
drwxr-xr-x 6 root root 4096 Aug 11 23:02 daily.5
drwxr-xr-x 6 root root 4096 Aug  9 02:42 weekly.0
drwxr-xr-x 6 root root 4096 Aug  1 22:37 weekly.1
drwxr-xr-x 6 root root 4096 Jul 24 22:33 weekly.2
drwxr-xr-x 6 root root 4096 Jul 18 02:02 weekly.3
drwxr-xr-x 6 root root 4096 Jun 30 22:36 weekly.4

I'm using the script from the issue description, symlinks are updated automatically. Note that I was away for a few days and the latest backup was yesterday when I returned, but no rotations were made when I was away.

The time-based rotations are just different algorithm to decide what to rotate. Backward compatibility must be preserved for current users (so option to enable them is required), but otherwise, it should not complicate stuff too much.

[rfjakob]

New rsnapshot user here.

I'm glad I looked through the bug reports, because this is pretty bad. If the backup fails, snapshots must not be deleted. In other words, rsnapshot should by default sync to a temporary directory first, and rotate afterwards.

[rfjakob]

@sam-at-github If you agree I will prepare a pull request to make that the default. Should be a small change as sync_first already has the functionality implemented. So the new default would be like sync_first but without splitting up the "sync" and "rotation" stage.