The --daily option inexplicably(?) deletes yesterday's backup unless -p is also given

[ernstki]

I read through the other issues (again), but I'm seeing daily backups deleted right before my eyes with the --daily= switch, and I'm still not sure if this is the intended behavior.

The gist of the problem is --daily=n (n > 1) without --prefer-recent seems to delete backups that occurred within the last 24 hours, which never allows enough of them to stick around for weekly, yearly, or monthly backups to be retained.

Here's the setup:

• CentOS 6, Python 2.7.something (from the RedHat SCL), rotate-backups version 4.4

• a nightly cron job which invokes rotate-backups (at 12:39 am, if that matters) with these options

    --quiet \
    --prefer-recent \
    --daily=$KEEP_DAYS \
    --weekly=$KEEP_WEEKS \
    --monthly=$KEEP_MONTHS \
    --yearly=$KEEP_YEARS \
  

  (where those variables have values of 7, 4, 6, and 2, respectively)

• the cron job runs rotate-backup before creating a tar.bz2 backup of some files, and a .sql.gz database dump, both of which get a timestamp that looks like

    sqldump=owncloud-sqldump_$(date +%Y-%m-%d).sql
  

  (i.e., ISO 8601 format)

• the tar.bz2 files seem to be properly rotated, but the sql.gz one is just deleted every night, so when I go check on rotate-backups's handiwork, there's just the one that finished at around 1 am, and no others.

I'm not sore about the files being deleted (there are ample warnings in the documentation about using --dry-run), but I'm puzzled by this behavior.

Here are two invocations of rotate-backups with the same options, except the second one adds --prefer-recent:

[owncloud_backups]$ rotate-backups --dry-run --daily=7 --weekly=4 --monthly=6 --yearly=2 .
<snipped> rotate_backups.cli[19231] INFO Performing a dry run (because of --dry-run option) ..
<snipped> rotate_backups[19231] INFO Scanning /var/adm/owncloud_backups for backups ..
<snipped> rotate_backups[19231] INFO Found 12 timestamped backups in /var/adm/owncloud_backups.
<snipped> rotate_backups[19231] INFO Preserving /var/adm/owncloud_backups/owncloud-data_2017-06-06.tar.bz2 (matches 'monthly' and 'yearly' retention periods) ..
<snipped> rotate_backups[19231] INFO Preserving /var/adm/owncloud_backups/owncloud-data_2017-09-22.tar.bz2 (matches 'weekly' and 'monthly' retention periods) ..
<snipped> rotate_backups[19231] INFO Preserving /var/adm/owncloud_backups/owncloud-data_2017-09-25.tar.bz2 (matches 'weekly' retention period) ..
<snipped> rotate_backups[19231] INFO Deleting /var/adm/owncloud_backups/owncloud-data_2017-09-28.tar.bz2 ..
<snipped> rotate_backups[19231] INFO Preserving /var/adm/owncloud_backups/owncloud-data_2017-09-29.tar.bz2 (matches 'daily' retention period) ..
<snipped> rotate_backups[19231] INFO Preserving /var/adm/owncloud_backups/owncloud-data_2017-09-30.tar.bz2 (matches 'daily' retention period) ..
<snipped> rotate_backups[19231] INFO Preserving /var/adm/owncloud_backups/owncloud-data_2017-10-01.tar.bz2 (matches 'daily' and 'monthly' retention periods) ..
<snipped> rotate_backups[19231] INFO Preserving /var/adm/owncloud_backups/owncloud-data_2017-10-02.tar.bz2 (matches 'daily' and 'weekly' retention periods) ..
<snipped> rotate_backups[19231] INFO Preserving /var/adm/owncloud_backups/owncloud-data_2017-10-03.tar.bz2 (matches 'daily' retention period) ..
<snipped> rotate_backups[19231] INFO Preserving /var/adm/owncloud_backups/owncloud-data_2017-10-04.tar.bz2 (matches 'daily' retention period) ..
<snipped> rotate_backups[19231] INFO Preserving /var/adm/owncloud_backups/owncloud-data_2017-10-05.tar.bz2 (matches 'daily' retention period) ..
<snipped> rotate_backups[19231] INFO Deleting /var/adm/owncloud_backups/owncloud-sqldump_2017-10-05.sql.gz ..

versus

[owncloud_backups]$ rotate-backups --dry-run --prefer-recent --daily=7 --weekly=4 --monthly=6 --yearly=2 .
<snipped> Performing a dry run (because of --dry-run option) ..
<snipped> Scanning /path/to/owncloud_backups for backups ..
<snipped> Found 12 timestamped backups in /path/to.
<snipped> Preserving /path/to/owncloud-data_2017-06-06.tar.bz2 (matches 'monthly' retention period) ..
<snipped> Preserving /path/to/owncloud-data_2017-09-22.tar.bz2 (matches 'weekly' retention period) ..
<snipped> Deleting /path/to/owncloud-data_2017-09-25.tar.bz2 ..
<snipped> Deleting /path/to/owncloud-data_2017-09-28.tar.bz2 ..
<snipped> Preserving /path/to/owncloud-data_2017-09-29.tar.bz2 (matches 'daily' retention period) ..
<snipped> Preserving /path/to/owncloud-data_2017-09-30.tar.bz2 (matches 'daily' and 'monthly' retention periods) ..
<snipped> Preserving /path/to/owncloud-data_2017-10-01.tar.bz2 (matches 'daily' and 'weekly' retention periods) ..
<snipped> Preserving /path/to/owncloud-data_2017-10-02.tar.bz2 (matches 'daily' retention period) ..
<snipped> Preserving /path/to/owncloud-data_2017-10-03.tar.bz2 (matches 'daily' retention period) ..
<snipped> Preserving /path/to/owncloud-data_2017-10-04.tar.bz2 (matches 'daily' retention period) ..
<snipped> Deleting /path/to/owncloud-data_2017-10-05.tar.bz2 ..
<snipped> Preserving /path/to/owncloud-sqldump_2017-10-05.sql.gz (matches 'daily', 'weekly', 'monthly' and 'yearly' retention periods) ..

Why does it seem to be working fine for the .tar.bz files?

Is this totally sensible behavior and my brain just can't grasp how the default algorithm works? Or is this errant behavior introduced recently, by a recent PR or something? I feel like the current behavior is contrary to the usual connotation of "daily backups" or "keep 7 days of daily backups."

[Klabuser]

I think this behavior is actually normal. As I understand it, rotate-backup expects one folder to be filled with only one type of backup. It will evaluate all the files present in this folder and run the algorithm on all of them, no matter the filename.

I see 2 solutions for your problem :

First one would be to put your backups in 2 differents folders :

/path/to/owncloud/data/
/path/to/owncloud/sqldump/

And put 2 differents call to rotate-backups in your crontab :

rotate-backups [arguments] /path/to/owncloud/data/
rotate-backups [arguments] /path/to/owncloud/sqldump/

If you want to keep both backup types in the same folder, the other solution would be to use something like :

 rotate-backups --include '*.tar.bz2' [arguments] /path/to/owncloud/
 rotate-backups --include '*.sql.gz' [arguments] /path/to/owncloud/

But this solution seems to be buggy, as you may have seen in the issue I opened : #7

[ernstki]

I thought about that, but it's not obvious from the documentation. I'll fix that in a PR if I could get confirmation from @xolox that this is, indeed, the intended behavior.

But anyway, why would --include or --exclude options be necessary if that were the case (that only one type of backup were expected to be present in the target directory)?

[xolox]

Hi Kevin and thanks for the feedback. Reading through #7 and #8 made me realize that the available documentation was insufficient, causing backups to be rotated when users aren't expecting it, without a good explanation why. I've tried to improve this now:

• In 47c61a4 I documented the backup collection strategy, including the following paragraph: All of the matched directory entries are considered to be backups of the same data source, i.e. there's no filename similarity logic to distinguish unrelated backups that are located in the same directory. If this presents a problem consider using the --include and/or --exclude options.

• In b150a23 I documented all of the supported configuration file options and their expected values. This should fix issues like Deletion not expected #7 where the option name include was being used where rotate-backups was expecting the option name include-list (the actual include/exclude list processing is working fine, it was just the documentation that was insufficient).

I'm going to close this issue now because everything was working as intended, but that wasn't clear from the documentation, which should be much better now.

[ernstki]

@xolox Super, thanks for being receptive to feedback from your users. Extra stars in my book. :)