Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

3.0.24 fog dependency error #306

Closed
sreid99 opened this Issue · 22 comments

9 participants

@sreid99

With backup 3.0.24, I recently started getting a backup error :

[2012/04/27 19:51:44][warning] ConfigurationError: [DEPRECATION WARNING]
[2012/04/27 19:51:44][warning] Backup::Compressor::Gzip.best has been deprecated as of backup v.3.0.24
[2012/04/27 19:51:44][warning] This setting has been replaced with:
[2012/04/27 19:51:44][warning] Backup::Compressor::Gzip.level
[2012/04/27 19:51:44][warning] Backup::Compressor::Gzip.level is being set to '9'
[2012/04/27 19:51:44][warning] ConfigurationError: [DEPRECATION WARNING]
[2012/04/27 19:51:44][warning] Backup::Compressor::Gzip.fast has been deprecated as of backup v.3.0.24
[2012/04/27 19:51:44][warning] This setting has been replaced with:
[2012/04/27 19:51:44][warning] Backup::Compressor::Gzip.level
[2012/04/27 19:51:44][error] Dependency::LoadError: Dependency missing
[2012/04/27 19:51:44][error] Dependency required for:
[2012/04/27 19:51:44][error] Amazon S3, Rackspace Cloud Files (S3, CloudFiles Storages)
[2012/04/27 19:51:44][error] To install the gem, issue the following command:
[2012/04/27 19:51:44][error] > gem install fog -v '~> 1.1.0'
[2012/04/27 19:51:44][error] Please try again after installing the missing dependency.

When I revert to 3.0.23, everything works fine. In both cases fog 1.3.1 in installed.

@meskyanichi
Owner

The ~> 1.1.0 means it has to be version 1.1.x so 1.3.x is too high. Does it still not work when installing Fog 1.1.x?

@sreid99

Guess I must have another gem which uses fog, and requires the latest version.

Why does backup 3.0.24 need an older version of fog than 3.0.23, and shouldn't bundler either force 1.1.x, or flag a fog version conflict ?

Not sure I want to revert to fog 1.1.x, as if might affect the other gems which use fog. I'll stick with backup 3.0.23 for now.

@ghost

You would be better off keeping the gems for backup with a project.
For example, if you stick with the default location of ~/Backup, you can do the following:

$ cd ~/Backup
$ bundle init
$ echo "gem 'backup'" >> Gemfile
$ bundle install --path vendor/bundle --binstubs

You would want to add whatever gem dependencies you need for backup in your Gemfile.
So, to add fog, you would:

$ echo "gem 'fog', '~> 1.1.0'" >> Gemfile
$ bundle install

You can then keep your ~/Backup folder under SCM.

$ git init
$ cat > .gitignore <<EOF
> vendor/bundle
> bin/
> .tmp/
> data/
> log/
> EOF
$ git add .
$ git commit -m 'initial commit'

I usually ignore .bundle/, but since this would not be a shared project, you may want to keep this.
You'll want to keep .cache/ if you use Dropbox.

To generate a model:
$ ~/Backup/bin/backup generate:model --trigger my_backup (...etc)
To run the job:
$ ~/Backup/bin/backup perform --trigger my_backup

If you choose a different folder than ~/Backup, then you'll need to use the --root-path option.
In which case you would run:
/other/folder/bin/backup perform --trigger my_backup --root-path /other/folder
Or, say from a cron job:
bash -l -c 'cd /other/folder; bin/backup perform --trigger my_backup --root-path .'

@ledermann

+1 for updating Fog dependency to 1.3.1

@john

+2

@JDutil

+1 for Fog 1.3.1 I ran into the same issue, and it's due to airbrake requiring a higher version of multi_json than Fog 1.1.x does.

  In Gemfile:
    fog (~> 1.1.0) ruby depends on
      multi_json (~> 1.0.3) ruby

    airbrake (>= 0) ruby depends on
      multi_json (1.3.5)

In Fog 1.3.1 the multi_json dependency is less strict becoming ~> 1.0 rather than the ~> 1.0.3 that causes issues with other gems depending on a greater version of multi_json than 1.0.x

@JDutil JDutil referenced this issue from a commit in JDutil/backup
@JDutil JDutil Update fog dependency. [fixes #306] d13cdef
@kjohnston

+1 for @jdutil's bumping of the fog dependency.

@ischuster

As far as I understood "jdutil" has found the problem and made the corrections. So the need is to do the pull and merge to master branch.
The question is, will it be done in master branch or should we go by ourselves?

Tks.

@babi4

+100500 @ischuster

@sjain

@burns your comment seem to suggest that keeping backup dependencies in the same Gemfile as your rails project is a bad idea. backup should really exist as separate project, with separate repository. If so, performing backup using this gem just got a lot harder than before. Now, do we have a separate capistrano deploy to push this backup config to production environment?!

@sjain

Instead of trying to manage dependency outside of bundler (lib/backup/dependency.rb), could we split the optional gems related functinoality (s3 backup using fog, for example) as a separate gem - call it backup-s3. This is similar to how omniauth manages individual plugins. They have add-on gems called omniauth-facebook, omniauth-openid and such. This will allow us to upgrade individual optional plugins without having to roll forward for each minor upgrade to these gems.

@meskyanichi
Owner

Can someone explain to me why they associate "Ruby on Rails" (or something similar), a web framework, with Backup, a command line utility? I don't really understand why people would add Backup to their applications' Gemfile at all.

ssh user@ip -p<port>
gem install backup
# .. generate config/models .. configure config/models
backup perform -t <trigger>

That's it. I don't see how people relate Backup to a Ruby application. Rather, it's related to datastores and file systems which don't have gem dependencies. So, the Backup dependencies should be completely decoupled from your application dependencies and cause zero conflict.

We also do not want to over complicate the gem. I have not thought of releasing a gem for every potential dependency that comes around. Seeing as we have a fairly large list of components, making a gem for everything seems tedious to me. I can see some benefits, but I don't know if they outweigh the cons that come with them. Please understand that Backup is a fairly large project with a ton of dependencies and it's not very easy to maintain. It's hard to make everyone happy.

Feel free however to convince @burns and myself why it should be added to a Gemfile of a Ruby application, rather than just using it in the stand-alone way it was meant to be. There are a lot of non-Ruby programmers that only use Ruby to utilize tools such as Backup, Whenever, Capistrano, Chef, Puppet, etc. and don't use anything like Rails, Sinatra, Merb, Padrino etc.

@sjain

Fair enough. I don't think this fact of keeping backup outside of rails project has been clear in communication enough. I have been using backup for over 1.5 years and always worked inside of rails even with Gemfile. I believe, people just assume that it all works until it doesn't. Lately though, dependency graph of backup gem has expanded and it causes such conflicts to occur often.

About extracting optional functionality as plugin, I believe, it is a matter of runtime vs. build time convenience. Sure, it will be more work to maintain 30 separate gems as opposed to one. But most end users use only 2-3 of those optional features and will be just happy to install those and go their way.

Some warning to rails folks to keep backup separate from Gemfile will help.

@meskyanichi
Owner

Fair enough. I don't think this fact of keeping backup outside of rails project has been clear in communication enough. I have been using backup for over 1.5 years and always worked inside of rails even with Gemfile. I believe, people just assume that it all works until it doesn't. Lately though, dependency graph of backup gem has expanded and it causes such conflicts to occur often.

Backup in fact did start out as a Rails plugin (back in version 1, not sure if it was in version 2). In any case, once I released version 2 or 3 I came to realize what I just stated, about a backup utility having nothing to do with a web framework and only overcomplicates things due to compatibility issues, and that it would be much better suited as it's own thing.

About extracting optional functionality as plugin, I believe, it is a matter of runtime vs. build time convenience. Sure, it will be more work to maintain 30 separate gems as opposed to one. But most end users use only 2-3 of those optional features and will be just happy to install those and go their way.

We would have to evaluate this some time.

Some warning to rails folks to keep backup separate from Gemfile will help.

Will probably add something like that to the README sometime.

Thanks for the feedback!

@JDutil

Personally I believe you should be able to bundle backup with your Rails app. I set require false within the Gemfile so that it isn't being loaded with the app.

The reason I do this is so that I can keep all my application dependencies within the project. That includes services you may not consider part of the application such as Whenever, Backup, Capistrano, God, and anything else I am relying on for my application.

While it may make sense for you to use Backup outside the context of your application it simply causes more work, more documentation, more organization, and more processes to manage. I'm just working on some small apps, and it works much better to keep everything I need within the application repository. It helps new developers bootstrap the application, and have everything they need in one place. It also allows me to have my Capistrano deployment scripts manage updating my Backup configuration when necessary.

While I understand you not wanting to always support dependency updates I think it's foolish not to update this Fog dependency to please the dozen or so people in favor of it. More time is being spent debating whether or not to update Backup to better support cloud storage than it would take to simply update it when I've already submitted and tested a pull request for it.

It is going to benefit everyone. Yourselves and the other developers like myself that enjoy the simplicity of keeping all their dependencies in one place. Besides simply leaving Backup out of your Gemfile does not prevent this error from happening to people. Bundler was created to help alleviate these issues.

It is very plausible that someone trying to use Backup even separately from their application could still run into this dependency error if they have multiple versions of different Gems installed. So to me the argument that this should be kept separate from the Rails app doesn't hold much weight personally.

@meskyanichi
Owner

Personally I believe you should be able to bundle backup with your Rails app. I set require false within the Gemfile so that it isn't being loaded with the app.

But doesn't think still cause dependency issues when running bundle install?

The reason I do this is so that I can keep all my application dependencies within the project. That includes services you may not consider part of the application such as Whenever, Backup, Capistrano, God, and anything else I am relying on for my application.

Understandable.

While it may make sense for you to use Backup outside the context of your application it simply causes more work, more documentation, more organization, and more processes to manage. I'm just working on some small apps, and it works much better to keep everything I need within the application repository. It helps new developers bootstrap the application, and have everything they need in one place. It also allows me to have my Capistrano deployment scripts manage updating my Backup configuration when necessary.

While I understand you not wanting to always support dependency updates I think it's foolish not to update this Fog dependency to please the dozen or so people in favor of it. More time is being spent debating whether or not to update Backup to better support cloud storage than it would take to simply update it when I've already submitted and tested a pull request for it.

The problem is that Rails (and typically the gems people typically use around Rails) evolve at a rapid pace. If these tools constantly update then people will constantly be coming back to Backup asking for dependency updates because otherwise Bundler is unable to bundle their app due to Backup using an older version of fog compared to what another dependency in the app requires.

It is very plausible that someone trying to use Backup even separately from their application could still run into this dependency error if they have multiple versions of different Gems installed. So to me the argument that this should be kept separate from the Rails app doesn't hold much weight personally.

Backup always requires gems through the gem function before require-ing it, meaning it always loads in the correct gem version and does not conflict even if newer versions are installed. Of course, if you're bundling it along with say Carrierwave in a Rails app which also uses Fog (but a later version) then it will conflict because of Bundler.


Now, despite all of this, it might be worth trying to just losen the dependencies to a >= rather than ~> where any newer version would work, rather than needing a certain minor version of a gem. I think most things are backwards compatible so there would less likely be need for dependency updates when other gem authors update their gems and dependencies.

What do other people think of this?

@ischuster
@ghost

This is all just a misunderstanding of how to use Bundler.

Bundler is not a tool to package all the gems for every application related to your project.
It's purpose is to create an isolated environment for your application, which includes the specific versions
of all the gems your app requires. It's purpose is to solve these dependency issues.

Backup is not a dependency of your Rails application, and therefore should not be in your Gemfile.
The only thing you should really keep with your Rails app (and under SCM)
is the Backup config.rb and Models for backup jobs related to your Rails app.

If you wish to keep a specific version of Backup with your Rails project,
then you can place it in a subdirectory of your project; like vendor/backup.
But it should have it's own Gemfile, which should only contain those gems which Backup requires.
(To be clear, don't put gem 'rails' in Backup's Gemfile.)

Your cron jobs should run Backup within it's own environment.

bash -l -c 'cd /rails_app/vendor/backup; bundle exec backup perform ...

If you want a rake task within your Rails application to run a backup job,
then you need to create a subshell outside of your app's environment
so you can run Backup within it's environment.

# /rails_app/Rakefile
task :backup do
  dir = File.expand_path('../', __FILE__)
  Bundler.with_clean_env do
    exec "cd #{ dir }/vendor/backup/; bundle exec backup perform --config-file #{ dir }/Backup/config.rb ... "
  end
end
@meskyanichi
Owner

Agreed 100% to Jeff "More time is being spent debating whether or not to update Backup to better support cloud storage than it would take to simply update ..." Gemfile, bundle, ... were made to keep things simple and easy. Why do not
use them? Maybe it brings some extra work. But that's the strength of social coding. You don't have to do all by yourself. Is this case in particular we've seen this. The solution was given by a user and published in a branch ready to be merged.

I understand that social coding in a lot of cases reduces the amount of work there is to be done by the maintainers. But consider that any changes might also have a negative impact on the project. Backup 2 is a good example of that, I pulled in too much unnecessary things and the gem became unmaintainable.

Now, this might look like a minor/insignificant change. However, I still feel it's an unnecessary one. So let me explain how I'm dealing with my own projects when I deploy to EC2/VPS or dedicated servers.

1) Mina/Capistrano for application deployment
2) Chef for provisioning

When I deploy my application, I scope it out to only doing what is essential for getting the application deployed and running. Backup plays no role here. However, when I provision my server with Chef (installing/configuring everything like nginx, mongodb, redis, ruby, deployment user, etc, etc) Backup is no exception here. I tell Chef to install Backup and Whenever. I tell it to write my Backup main config to /root/Backup/config.rb, my model to /root/Backup/models/*.rb and Whenever's schedule.rb to /root/Backup/schedule.rb. Once in place, I tell it to update the crontab with whenever --update-crontab -i myapp -f /root/Backup/schedule.rb. Done. Backup is now in place and is performing scheduled backups.

In my opinion, this is what you should be doing. Zero gem conflicts should occur when you do it this way (which I consider the appropriate way). Keep a separate Chef repository aside of your App repository. I understand people will always have mixed feelings but considering everything (should) work when you do not directly tie it in to a Ruby app means it's not broken, and Rails isn't an excuse to update any Backup dependencies. A bug/issue in Backup because of a dependency of course is a valid reason for updating a dependency, assuming the update fixes that particular bug.

Also, Chef is very easy to learn if you haven't already, and if you're not using it you're missing out. Go check out RailsCasts.com for Chef-Solo, then check out the Knife-Solo and Librarian gems which make managing cookbooks easy. It makes a world of a difference compared to regular shell scripts or using Capistrano to as a provisioning tool.

I'm not sure whether I'm going to want to change ~> to >= dependencies. Because this has a lot higher potential for causing issues than a simple Rails dependency conflict. If Fog does something to it's API, causing backwards incompatibility, then Backup is forced to push out a new gem just because Fog decided to change it's API in a newer version, which caused people that installed Backup afterwards to break and start posting issues regarding that again here. It'll become an endless loop which in my opinion is unnecessary and avoidable by just having users use Backup as it should be used, which is "stand-alone".

I hope people can understand this decision.

One more thing to note, seeing as most people will be using Capistrano + Bundler + Backup. If you really don't want to use a separate provisioning tool like Chef/Puppet, why don't you just make another Capistrano task instead?

task :manage_backup do
  run "gem install backup"
end

Then you won't have any dependency conflicts with your Rails dependencies in Bundler.

@JDutil

I understand your reservations but shouldn't it also be a goal of backup to continue progressing forward? There's reasons that backups dependencies release new versions like bug fixes, performance improvements or new features. I think it's in the best interest of the project to periodically update it's dependencies when there are not upgrade issues. Otherwise it's very short sighted to just say hey guys my setup works so there's no reason to update it. That's how projects slowly die as everyone else continues moving forward, but the project waits so long to try getting up to date that people move on elsewhere or it becomes more of a hassle to upgrade than it's worth.

@meskyanichi
Owner

I understand your reservations but shouldn't it also be a goal of backup to continue progressing forward? There's reasons that backups dependencies release new versions like bug fixes, performance improvements or new features. I think it's in the best interest of the project to periodically update it's dependencies when there are not upgrade issues.

Periodically updating dependencies to keep up should be fine. Preferably iterate through all of Backups dependencies on a monthly basis to check for non-breaking updates might be worthwhile. Then release a gem based on these updates. I just don't feel much for doing one commit to update dependencies and then releasing a whole new gem just for that.

Otherwise it's very short sighted to just say hey guys my setup works so there's no reason to update it. That's how projects slowly die as everyone else continues moving forward, but the project waits so long to try getting up to date that people move on elsewhere or it becomes more of a hassle to upgrade than it's worth.

Well, why fix something that isn't broken really. It works perfectly fine (I assume). Assuming everyone wouldn't associate Backup as a dependency for their application. That's the main issue. Backup isn't a dependency of a project. Everyone's project will still run even if Backup isn't present. This seems, for some odd reason, very hard to understand for many people. Lots of non-Rubyists use Backup as their backup tool for Python, Java, Scala, PHP, etc, projects. It even got ported to the FreeBSD ports system so even sysadmins that don't even deploy applications use it. It is a tool on it's own and is in no way related to pretty much anything.

Now, as I stated I do not see a problem periodically updating dependencies. But these are not meant to "fix" dependency conflicts people have in their Rails apps. Note that if someone uses an older version of CarrierWave and I update Fog but CarrierWave isn't compatible, then even though I updated it, CarrierWave has not and the dependency issue is back again. So while I agree that periodically updating all the dependencies you see in backup dependencies --list, it wouldn't be to help fix Bundler dependency conflicts. Like you said, the benefits would be potential bugfixes/performance improvements/features.

I will have to make this clear somewhere in the README because people keep opening tickets thinking that backup is supposed to go in to a project's Gemfile.

@JDutil

I'm glad we agree on the need to gradually update dependencies. I just think it's simpler to update them gradually over time like this on a case by case basis. It's much more likely to cause issues when you attempt to update several gems at once than it is to update each gem on a case by case basis.

@sjain sjain referenced this issue from a commit
Commit has since been removed from the repository and is no longer available.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.