Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

default expiration logic not work with hourly cron job #140

Open
smasterfree opened this issue Dec 25, 2018 · 5 comments

Comments

Projects
None yet
4 participants
@smasterfree
Copy link

commented Dec 25, 2018

I use cron job to hourly back up my folder

# m h  dom mon dow   command
11 *  * * *  rsync_tmbackup.sh -p 22 ~  root@remote-backup:~

and the remote back folder looks like

drwxr-x--- 34 root root 4.0K Dec 24 01:25 2018-12-24-101101/
drwxr-x--- 34 root root 4.0K Dec 24 02:46 2018-12-24-111101/
drwxr-x--- 34 root root 4.0K Dec 24 03:25 2018-12-24-121102/
drwxr-x--- 34 root root 4.0K Dec 24 03:25 2018-12-24-131101/
drwxr-x--- 34 root root 4.0K Dec 24 05:51 2018-12-24-141101/
drwxr-x--- 34 root root 4.0K Dec 24 07:02 2018-12-24-151102/
drwxr-x--- 34 root root 4.0K Dec 24 07:02 2018-12-24-161101/
drwxr-x--- 34 root root 4.0K Dec 24 07:02 2018-12-24-171101/
drwxr-x--- 34 root root 4.0K Dec 24 07:02 2018-12-24-181101/
drwxr-x--- 34 root root 4.0K Dec 24 07:02 2018-12-24-191101/
drwxr-x--- 34 root root 4.0K Dec 24 07:02 2018-12-24-201101/
drwxr-x--- 34 root root 4.0K Dec 24 07:02 2018-12-24-211101/
drwxr-x--- 34 root root 4.0K Dec 24 07:02 2018-12-24-221101/
drwxr-x--- 34 root root 4.0K Dec 24 07:02 2018-12-24-231102/
drwxr-x--- 34 root root 4.0K Dec 24 07:02 2018-12-25-011102/
drwxr-x--- 34 root root 4.0K Dec 24 07:02 2018-12-25-021101/
drwxr-x--- 34 root root 4.0K Dec 24 07:02 2018-12-25-031102/
drwxr-x--- 34 root root 4.0K Dec 24 07:02 2018-12-25-041101/
drwxr-x--- 34 root root 4.0K Dec 24 07:02 2018-12-25-051101/
drwxr-x--- 34 root root 4.0K Dec 24 07:02 2018-12-25-061101/
drwxr-x--- 34 root root 4.0K Dec 24 07:02 2018-12-25-071101/
drwxr-x--- 34 root root 4.0K Dec 24 07:02 2018-12-25-081102/
drwxr-x--- 34 root root 4.0K Dec 24 07:02 2018-12-25-091101/
drwxr-x--- 34 root root 4.0K Dec 24 07:02 2018-12-25-101101/

according to readme :

The default strategy is 1:1 30:7 365:30, which means:

After 1 day, keep one backup every 1 day (1:1).
After 30 days, keep one backup every 7 days (30:7).
After 365 days, keep one backup every 30 days (365:30).

but if all the backups(more than one day) were deleted, how to keep the backups older than 30days?

I think the default backup strategy is similar as Grandfather-father-son(https://en.wikipedia.org/wiki/Backup_rotation_scheme), so if an error occur, but I do not noticed in a week, I still have chance to restore from monthly backup. How could I combined GFS with hourly backup ?

@matthewbednarski

This comment has been minimized.

Copy link

commented Dec 28, 2018

I have the same problem. It's only keeping the backups from the last 24h.

@radarjd

This comment has been minimized.

Copy link

commented Jan 5, 2019

I have the same issue. I run the script on multiple directories with different schedules in cron (some 4 hours apart, some 6, some 12), but in all cases, only the past 24 hours of backups are kept.

I have attempted to leave --strategy unset, and to explicitly set it. The behavior does not seem to change.

I do make use of following rsync flags: --numeric-ids --links --hard-links --one-file-system --archive --no-perms --no-group --itemize-changes

My base OS is Ubuntu 18.04.01 LTS (Bionic Beaver). I'm using the git clone version of this software as of today's date.

I'm wondering if it's a log file issue. As far as I can tell, there are no files in the directory specified by the --log-file directory, which is the default.

Please let me know if I can provide any additional diagnostic information. I don't think I'm good enough with bash to puzzle this one out myself.

@gostelna

This comment has been minimized.

Copy link

commented Jan 11, 2019

Same thing here, on OSX 10.14.2.
Using the default config and a cron every hour, the script is only keeping the 24 latest backups.
rsync_tmbackup.sh --log-dir /logdir xxx@xxx.xxx.xxx.xxx:/docPath /backupPath excludeFile.txt

@radarjd

This comment has been minimized.

Copy link

commented Jan 12, 2019

I think this is actually the same issue as that reported in #121

Albert444 suggested a solution in that issue, and I have copied it into my local version to see if that will fix the issues.

Here's the diff of the version provided by Albert444:

60c60
<               linux*) date -d "${1:0:10} ${1:11:2}:${1:13:2}:${1:15:2}" +%s ;;
---
>               linux*) date -d "${1:0:10}" +%s ;;
129c129,133
<                               if [ "$interval_since_last_kept" -lt "$cut_off_interval" ]; then
---
> 
>                               dayssincestart=$(((backup_timestamp-start_timestamp)/86400))
>                               modulo=$((dayssincestart % (cut_off_interval/86400)))
> 
>                               if [ "$modulo" -ne "0" ]; then
133,134c137,142
<                                       # No: Keep it
<                                       last_kept_timestamp=$backup_timestamp
---
>                                       if [ "$interval_since_last_kept" -lt "86400" ]; then #if diffenrence is less than one day
>                                               fn_expire_backup "$backup_dir"
>                                       else
>                                               # No: Keep it
>                                               last_kept_timestamp=$backup_timestamp
>                                       fi
@smasterfree

This comment has been minimized.

Copy link
Author

commented Jan 13, 2019

now I use borg backup instead.

it's prune example is more clear.

borg prune visualized
=====================

Assume it is 2016-01-01, today's backup has not yet been made and you have
created at least one backup on each day in 2015 except on 2015-12-19 (no
backup made on that day).

This is what borg prune --keep-daily 14 --keep-monthly 6 would keep.

Backups kept by the --keep-daily rule are marked by a "d" to the right,
backups kept by the --keep-monthly rule are marked by a "m" to the right.

Calendar view
-------------

                            2015
      January               February               March
Mo Tu We Th Fr Sa Su  Mo Tu We Th Fr Sa Su  Mo Tu We Th Fr Sa Su
          1  2  3  4                     1                     1
 5  6  7  8  9 10 11   2  3  4  5  6  7  8   2  3  4  5  6  7  8
12 13 14 15 16 17 18   9 10 11 12 13 14 15   9 10 11 12 13 14 15
19 20 21 22 23 24 25  16 17 18 19 20 21 22  16 17 18 19 20 21 22
26 27 28 29 30 31     23 24 25 26 27 28     23 24 25 26 27 28 29
                                            30 31

       April                  May                   June
Mo Tu We Th Fr Sa Su  Mo Tu We Th Fr Sa Su  Mo Tu We Th Fr Sa Su
       1  2  3  4  5               1  2  3   1  2  3  4  5  6  7
 6  7  8  9 10 11 12   4  5  6  7  8  9 10   8  9 10 11 12 13 14
13 14 15 16 17 18 19  11 12 13 14 15 16 17  15 16 17 18 19 20 21
20 21 22 23 24 25 26  18 19 20 21 22 23 24  22 23 24 25 26 27 28
27 28 29 30           25 26 27 28 29 30 31  29 30m


        July                 August              September
Mo Tu We Th Fr Sa Su  Mo Tu We Th Fr Sa Su  Mo Tu We Th Fr Sa Su
       1  2  3  4  5                  1  2      1  2  3  4  5  6
 6  7  8  9 10 11 12   3  4  5  6  7  8  9   7  8  9 10 11 12 13
13 14 15 16 17 18 19  10 11 12 13 14 15 16  14 15 16 17 18 19 20
20 21 22 23 24 25 26  17 18 19 20 21 22 23  21 22 23 24 25 26 27
27 28 29 30 31m       24 25 26 27 28 29 30  28 29 30m
                      31m

      October               November              December
Mo Tu We Th Fr Sa Su  Mo Tu We Th Fr Sa Su  Mo Tu We Th Fr Sa Su
          1  2  3  4                     1      1  2  3  4  5  6
 5  6  7  8  9 10 11   2  3  4  5  6  7  8   7  8  9 10 11 12 13
12 13 14 15 16 17 18   9 10 11 12 13 14 15  14 15 16 17d18d19 20d
19 20 21 22 23 24 25  16 17 18 19 20 21 22  21d22d23d24d25d26d27d
26 27 28 29 30 31m    23 24 25 26 27 28 29  28d29d30d31d
                      30m

List view
---------

--keep-daily 14     --keep-monthly 6
-------------------------------------------------
 1. 2015-12-31          (2015-12-31 kept by daily rule)
 2. 2015-12-30       1. 2015-11-30
 3. 2015-12-29       2. 2015-10-31
 4. 2015-12-28       3. 2015-09-30
 5. 2015-12-27       4. 2015-08-31
 6. 2015-12-26       5. 2015-07-31
 7. 2015-12-25       6. 2015-06-30
 8. 2015-12-24
 9. 2015-12-23
10. 2015-12-22
11. 2015-12-21
12. 2015-12-20
    (no backup made on 2015-12-19)
13. 2015-12-18
14. 2015-12-17


Notes
-----

2015-12-31 is kept due to the --keep-daily 14 rule (because it is applied
first), not due to the --keep-monthly rule.

Because of that, the --keep-monthly 6 rule keeps Nov, Oct, Sep, Aug, Jul and
Jun. December is not considered for this rule, because that backup was already
kept because of the daily rule.

2015-12-17 is kept to satisfy the --keep-daily 14 rule - because no backup was
made on 2015-12-19. If a backup had been made on that day, it would not keep
the one from 2015-12-17.

We did not include yearly, weekly, hourly, minutely or secondly rules to keep
this example simple. They all work in basically the same way.

The weekly rule is easy to understand roughly, but hard to understand in all
details. If interested, read "ISO 8601:2000 standard week-based year".

and from my test, it works as expected:

remote-2019-01-07T23:05:02 Mon, 2019-01-07 23:05:03 [b9e32f94e9a2a37c41df42548a42423b9d292b4fd5d923a176cedad511fc0bdd]
remote-2019-01-08T23:05:02 Tue, 2019-01-08 23:05:03 [7ee1988d17177c24d7d151deae500138d772ba2502f212a2644890d670f72525]
remote-2019-01-09T23:05:02 Wed, 2019-01-09 23:05:03 [e91d6c2453522af0d10f8cbe0eb1a8bb0116e4e8426a0597892480fc871a68c6]
remote-2019-01-10T23:05:02 Thu, 2019-01-10 23:05:03 [7f7f884715abd62bc91719b98ef074b89f57ff098a776d6c83d6d15961de86aa]
remote-2019-01-11T23:05:02 Fri, 2019-01-11 23:05:03 [c672daebf23fa8239a252797c23c74bf49f651eafc915dc4759df7bd0b15919a]
remote-2019-01-12T14:05:02 Sat, 2019-01-12 14:05:03 [73547871b921e80d9b920b8f0cf418e4d774327a8d975dacfa15bbbbe8473c2f]
remote-2019-01-12T15:05:01 Sat, 2019-01-12 15:05:03 [523fa148c60f44d97dee9963e6a5f08dd94f132c51e5d5752dc439785d2e3a26]
remote-2019-01-12T16:05:02 Sat, 2019-01-12 16:05:03 [719f16e12fbf10effa27eb62a3d7de4864bc400cb690967212fc56b11a774ec4]
remote-2019-01-12T17:05:02 Sat, 2019-01-12 17:05:03 [8cb564a20ec9b07de8ff10b776c83adead62421a9fbf40564cbd0e6881777f84]
remote-2019-01-12T18:05:01 Sat, 2019-01-12 18:05:03 [9e75f2f61a4f4a064ddb7be9a5135d0611e4e371fdab723a66d44ca47a7b9e5e]
remote-2019-01-12T19:05:02 Sat, 2019-01-12 19:05:03 [8668038c460f0f1d7e7c5ebfbccb15b903403715b519feea357b554523fe0e80]
remote-2019-01-12T20:05:02 Sat, 2019-01-12 20:05:03 [21c321b53c8e81dfef2d500e7ddfe15fd7358e1b46354f39e6685044991e5ce8]
remote-2019-01-12T21:05:02 Sat, 2019-01-12 21:05:03 [e33b2c34b6e7f51b6850d74bb7ef4e67271844e59cc3ae2065de775c8522c92e]
remote-2019-01-12T22:05:02 Sat, 2019-01-12 22:05:03 [e8a2771c62c5c418464ac44311fc1d196a3d2bb863ed08fb13bef9b02bd06179]
remote-2019-01-12T23:05:01 Sat, 2019-01-12 23:05:03 [82797bcb4d77b56fbfb404938682c34cf79242293bb54c8dbe0d994a339be9b2]
remote-2019-01-13T00:05:02 Sun, 2019-01-13 00:05:03 [3f2108ed7d176b0c35efbb23a4b4d118e38910fb6178a9a1c521082e986f53ff]
remote-2019-01-13T01:05:02 Sun, 2019-01-13 01:05:03 [e1fd3117529fa0f5a9f58dea151286394cf1bf6774ef5c0bff8f0ec2697d905e]
remote-2019-01-13T02:05:02 Sun, 2019-01-13 02:05:03 [31aca3bf46d5678920a3b1bb979c7e8672e00cf2a3f51228337018680dc06f8d]
remote-2019-01-13T03:05:02 Sun, 2019-01-13 03:05:03 [2b55a79a3b8b8c4b46f4c79d429e628c637b3839bc8d1aedb844b292c40b7149]
remote-2019-01-13T04:05:02 Sun, 2019-01-13 04:05:03 [5352485332c36de4249aaab02b0eaf292aa62af5cc4cc83ab8a90083aae747a6]
remote-2019-01-13T05:05:01 Sun, 2019-01-13 05:05:02 [4486d27a9b8b6eb915ad00ad700fe5e96e2d175cc31dbd7128015c4b7dd39098]
remote-2019-01-13T06:05:02 Sun, 2019-01-13 06:05:03 [a278295343d40a75e7aad8a16c46418ef146681929eb9748def5e44f42eabe00]
remote-2019-01-13T07:05:01 Sun, 2019-01-13 07:05:03 [b5bd9f8c7f371608ec1e45b115f500ae38b9c716ed64b710fb14892b4625ca0d]
remote-2019-01-13T08:05:01 Sun, 2019-01-13 08:05:03 [2698f1690967f40451f0c7c840b03e178ab1a2c8591461e0e9e2241b7e2f9f9d]
remote-2019-01-13T09:05:01 Sun, 2019-01-13 09:05:02 [70e20a5167deb7a0b8b6d64d97edac40b1e57598b78e20f5ff5907cbd483dcaf]
remote-2019-01-13T10:05:01 Sun, 2019-01-13 10:05:02 [6d69cea793fbb486b1417a3431d4d8b4c1f2fd9a2004dd95ab6582ea64d5de16]
remote-2019-01-13T11:05:02 Sun, 2019-01-13 11:05:03 [0ca8b8c500ee72bd23962959a55719fedbebbd33f84d4ba219b310dd6770abc4]
remote-2019-01-13T12:05:01 Sun, 2019-01-13 12:05:03 [38c50ec0b313fa1cc399eb4d343a7f99fc6422bfbe440e36d95cb3fa87bbb66c]
remote-2019-01-13T13:05:02 Sun, 2019-01-13 13:05:03 [18e18a874649a653d1a7d1a961ae87e897574fa0cae0411f766bc3cda468f31f]
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.