Backup s3 keep not working #751
Comments
|
Hi store_with Local do |local|
local.path = "~/backups/#{ENV['RAILS_ENV']}_db_backup"
local.keep = 10
end |
|
Hi @heldopslippers and @nahidnascenia, that doesn't seem right. Can you try version 4.2.0? That's before I reworked the keep option a bit. See if that works? If so, I can narrow the issue down |
|
Ok this is weird, just tested it with 4.2.0. |
|
I'm having this same problem on s3 with a daily/weekly/monthly backup. Daily backups are duplicating and weekly backups are only keeping 1. I haven't run it long enough to see what happens on month 2. And this runs daily from a Backup version: I can make a new issue if that's appropriate, but it seems to be a similar issue. I haven't tried 4.2.0, is that really the solution to this problem? |
|
@benortiz yes, it might be caused by a recent update. Please try the 4.2.0 version. It it still happens I know if both problems are caused by it |
|
@tombruijn thanks! I fixed the duplication problem, that was my mistake (I had it deployed to two separate servers). I'll try 4.2.0 to see if that fixes the weekly backup. |
|
@tombruijn Using 4.2.0 I get an error for the reply_to method: My config for mail looks like this |
|
@tombruijn Ah okay, so should I be using 4.2.1 then? or 4.2.0 and remove the |
|
either option works (sorry for the delayed response) |
|
i'm Using gem 4.3 still not working keep on s3 |
|
That's because it's not made to work on syncers, only storage. |
|
I have this same issue with S3 - Backup::Model.new(:pr, 'Backup the database.') do
require 'yaml'
db = YAML.load_file(File.expand_path('../config/database.yml', File.dirname(__FILE__)))
s3 = YAML.load_file(File.expand_path('../config/backup-secrets.yml', File.dirname(__FILE__)))
database PostgreSQL do |d|
d.name = db['production']['database']
d.username = db['production']['username']
d.password = db['production']['password']
d.host = db['production']['host']
end
compress_with Bzip2
encrypt_with OpenSSL do |encryption|
encryption.password = 'iC2cat4sfdsfgat6uK1Oy9Hai0asasdfwas9a'
encryption.base64 = true
encryption.salt = true
end
time = Time.now
if time.day == 1
storage_id = :monthly
keep = 12
elsif time.sunday?
storage_id = :weekly
keep = 4
else
storage_id = :daily
keep = 7
end
store_with S3 do |s|
s.access_key_id = s3['access_key_id']
s.secret_access_key = s3['secret_access_key']
s.bucket = 'backups.example.com'
s.path = storage_id.to_s
s.fog_options = { path_style: true }
s.region = 'us-east-1'
s.keep = keep
s.encryption = :aes256
end
notify_by Mail do |mail|
mail.delivery_method = :sendmail
mail.on_success = true
mail.on_failure = true
mail.on_warning = true
mail.from = 'backups@example.com'
mail.to = 'brady@example.com'
end
end
# * * * * * * * * * * * * * * * * * * * *
# Do Not Edit Below Here.
# All Configuration Should Be Made Above.
##
# Load all models from the models directory.
Dir[File.join(File.dirname(Config.config_file), 'models', '*.rb')].each do |model|
instance_eval(File.read(model))
end |
|
Can you try version 4.2.0? Does it still not work properly? |
|
Log below. It looks like 4.2.0 may work correctly (it removes the 6th oldest backup), but it does not trim older backups (a.k.a. if I have 42 backups in I think expected behaviour would be to truncate the folder to (in this case, 5) backups total for |
|
Is this still an expected issue with 4.3? It seems quite a fundamental problem. I am still seeing problems with 4.3. I have 25 backups and I'm expecting only 11. If I use v 4.2.1 as suggested, I get the same problem (Actually 4.2.1 is missing from gem, 4.2.3 is present |
|
@max-allan-surevine Please can you post the IAM policy permissions that the user has on the S3 bucket? By coincidence, I have seen an issue using keep with 4.3 on S3 this week. I haven't had time to fully investigate, but in that case the issue at least appears to be the effective permissions. |
|
Hey @stuartellis, as I posted previously above there seems to be a difference in function between 4.2 and 4.3. The bug was introduced between these two versions, and (this bug at least) occurs regardless of permissions. |
|
Thanks, @bouchard. I should be able to take a look this weekend. |
|
I was assuming there was a general issue with keep. Not limited to S3. But now I see it seems to be working fine on S3 but not on local files. @nahidnascenia mentioned the same issue above. |
|
still a problem with 4.4.0 |
|
Hello @benben - This ticket is getting a little confused. Would you mind opening a new ticket for your specific issue with 4.4.0 ? The ticket template specifies the information that we will need to reproduce your issue. |
|
This needs to be fixed, falling back to 4.2.0 is not an option for everyone because of this: #820 which is that installation of older versions fail on ruby 2.4+ |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
|
bump, needs fix |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
|
bump |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
|
still not working need fix |
|
Feel free to send a merge request |
|
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
|
lol this is still open, I was wondering why we have 90 TB of backups which we have to pay for, just to find out that instead of 24 hourly backups we have 6000. lol. |
|
/reopen |
|
I guess the reason is that the S3 Storage class doesn't implement the cycle method? backup/lib/backup/storage/base.rb Line 44 in 02165ec We should at least throw an exception if keep is set, but keep not supported? This error has probalby cost us around 60k EUR (also fault of us due to not noticing it over the last years). |
|
@h0jeZvgoxFepBQ2C I don't think cycler gets called directly, it runs something else which then uses "remove" in the S3 code: backup/lib/backup/storage/s3.rb Line 112 in 02165ec But it's a long time since I looked at this project! IIRC I found a clash between some other storage class (local files) that was resetting the keep flag for no apparent reason. And if you used local files and S3, only local files would cycle properly. |
Hi Guys,
Trying it with 4.2.2 and 4.2.3.
keep doesn't seem to work with S3 (didn't try it with other adapters).
Anybody else have the same problems?
The text was updated successfully, but these errors were encountered: