Plugin dependency management revamp #2769

Merged
merged 54 commits into from Jan 7, 2014

Conversation

Projects
None yet
@mitchellh
Member

mitchellh commented Jan 6, 2014

This is a huge revamp of how plugin dependency management is done. To understand the changes here, a brief history lesson is in order:

Since Vagrant 1.1, plugins have been loaded as RubyGems. Once Vagrant was loaded, it would iterate through a list of installed plugins, and require that plugin. This mostly worked okay. But the devil is in the details, and the edge cases were really bad. In addition to the edge cases (mentioned below), building things like updaters, version constraints (">= 1.0", "< 1.1"), etc. all had to be done manually. This seemed silly, since RubyGems itself (and Bundler) do these sort of things for you. Why reinvent the wheel?

As for edge cases: the primary edge case is that since the dependencies of Vagrant and its respective plugins weren't resolved as a whole, you can run into cases where plugin installation succeeded, but plugin loading failed because Vagrant already loaded a common dependency with the wrong version. An example explains this best:

  • Vagrant depends on "A >= 1.0, < 1.2"
  • vagrant-plugin depends on "A = 1.1"
  • When you run Vagrant, it loads the latest possible matching dependencies, so it would load A 1.2
  • When Vagrant loads vagrant-plugin, it can't load, because A 1.2 is active, so A 1.1 can't be loaded.

The error above should never happen: the versions available for A should satisfy both Vagrant and vagrant-plugin (by loading v1.1 for both).

With this new branch, all plugin installation, dependency resolution, updating, etc. is managed by Bundler. This has yielded numerous benefits:

  • Vagrant now resolves dependencies before Vagrant is even loaded. This ensures that all plugins will be able to load. No more conflicts at run-time.
  • Conflicts are detected at vagrant plugin install time. This means that if there would be a crash if that plugin were to load, the plugin won't even install and a human-friendly error is shown to the end user.
  • vagrant plugin install now accepts complex version constraints such as "~> 1.0.0" or ">= 1.0, < 1.1". Vagrant stores these constraints for updating, which leads to the next point.
  • vagrant plugin update without arguments now updates all installed plugins, respecting the constraints specified by vagrant plugin install.
  • vagrant plugin update NAME will only update that gem (still respecting constraints).
  • Internally, there are a lot more unit tests. /cc @phinze :)

The goal of this branch was to replace the existing system and functionality with Bundler-ized management. It did not introduce any new features except where they naturally fell into place (version constraints). However, with this new system, many new possibilities are also available:

  • Vagrant environment local plugins (i.e. a Gemfile but for a specific Vagrant environment).
  • Plugin installation from git

I'm sure those will be pursued at some point in the future.

This fixes: #2612, #2406, #2428

@raphaelcosta

This comment has been minimized.

Show comment
Hide comment

👎

This comment has been minimized.

Show comment
Hide comment
@mitchellh

mitchellh Jan 5, 2014

Member

@raphaelcosta To our change or to log4r's? I can't upgrade to 1.1.11 yet because there are actually quite a few Vagrant plugins that still use the 1.1.10 API... so I think at this point it would break so much more.

Member

mitchellh replied Jan 5, 2014

@raphaelcosta To our change or to log4r's? I can't upgrade to 1.1.11 yet because there are actually quite a few Vagrant plugins that still use the 1.1.10 API... so I think at this point it would break so much more.

This comment has been minimized.

Show comment
Hide comment
@daker

daker Jan 5, 2014

👎 for log4r's API changes :(

daker replied Jan 5, 2014

👎 for log4r's API changes :(

This comment has been minimized.

Show comment
Hide comment
@raphaelcosta

raphaelcosta Jan 5, 2014

@mitchellh No, to log4r! They can't do that!

@mitchellh No, to log4r! They can't do that!

This comment has been minimized.

Show comment
Hide comment

👏

@mitchellh

This comment has been minimized.

Show comment
Hide comment
@mitchellh

mitchellh Jan 7, 2014

Member

I want to get some acceptance tests around plugin management into vagrant-spec before merging this...

Member

mitchellh commented Jan 7, 2014

I want to get some acceptance tests around plugin management into vagrant-spec before merging this...

lib/vagrant.rb
+# This file is load before RubyGems are loaded, and allow us to actually
+# resolve plugin dependencies and load the proper versions of everything.
+
+ENV["VAGRANT_INSTALLER_EMBEDDED_DIR"] = "/Applications/Vagrant/embedded"

This comment has been minimized.

@tmatilai

tmatilai Jan 7, 2014

Collaborator

@mitchellh this doesn't seem very portable =)

@tmatilai

tmatilai Jan 7, 2014

Collaborator

@mitchellh this doesn't seem very portable =)

This comment has been minimized.

@mitchellh

mitchellh Jan 7, 2014

Member

HAHA! Nice catch. Removing that.

@mitchellh

mitchellh Jan 7, 2014

Member

HAHA! Nice catch. Removing that.

@mitchellh

This comment has been minimized.

Show comment
Hide comment
@mitchellh

mitchellh Jan 7, 2014

Member

The acceptance tests pass: https://github.com/mitchellh/vagrant-spec/blob/master/acceptance/cli/plugin_spec.rb

Going to go ahead and merge it in. Oof.

Member

mitchellh commented Jan 7, 2014

The acceptance tests pass: https://github.com/mitchellh/vagrant-spec/blob/master/acceptance/cli/plugin_spec.rb

Going to go ahead and merge it in. Oof.

mitchellh added a commit that referenced this pull request Jan 7, 2014

Merge pull request #2769 from mitchellh/f-bundlerize
Plugin dependency management revamp

This is a huge revamp of how plugin dependency management is done. To understand the changes here, a brief history lesson is in order:

Since Vagrant 1.1, plugins have been loaded as RubyGems. Once Vagrant was loaded, it would iterate through a list of installed plugins, and `require` that plugin. This mostly worked okay. But the devil is in the details, and the edge cases were _really_ bad. In addition to the edge cases (mentioned below), building things like updaters, version constraints (">= 1.0", "< 1.1"), etc. all had to be done manually. This seemed silly, since RubyGems itself (and Bundler) do these sort of things for you. Why reinvent the wheel?

As for edge cases: the primary edge case is that since the dependencies of Vagrant and its respective plugins weren't resolved as a whole, you can run into cases where plugin installation succeeded, but plugin loading failed because Vagrant already loaded a common dependency with the wrong version. An example explains this best:

* Vagrant depends on "A >= 1.0, < 1.2"
* vagrant-plugin depends on "A = 1.1"
* When you run Vagrant, it loads the latest possible matching dependencies, so it would load A 1.2
* When Vagrant loads vagrant-plugin, it can't load, because A 1.2 is active, so A 1.1 can't be loaded.

The error above should never happen: the versions available for A should satisfy both Vagrant and vagrant-plugin (by loading v1.1 for both). 

With this new branch, all plugin installation, dependency resolution, updating, etc. is managed by [Bundler](http://gembundler.com). This has yielded numerous benefits:

* Vagrant now resolves dependencies before Vagrant is even loaded. This ensures that all plugins will be able to load. No more conflicts at run-time.

* Conflicts are detected at `vagrant plugin install` time. This means that if there would be a crash if that plugin were to load, the plugin won't even install and a human-friendly error is shown to the end user.

* `vagrant plugin install` now accepts complex version constraints such as "~> 1.0.0" or ">= 1.0, < 1.1". Vagrant stores these constraints for updating, which leads to the next point.

* `vagrant plugin update` without arguments now updates all installed plugins, respecting the constraints specified by `vagrant plugin install`.

* `vagrant plugin update NAME` will only update that gem (still respecting constraints). 

* Internally, there are a lot more unit tests. /cc @phinze :)

The goal of this branch was to replace the _existing_ system and functionality with Bundler-ized management. It did not introduce any new features except where they naturally fell into place (version constraints). However, with this new system, many new possibilities are also available:

* Vagrant environment local plugins (i.e. a Gemfile but for a specific Vagrant environment). 

* Plugin installation from git

I'm sure those will be pursued at some point in the future.

This fixes: #2612, #2406, #2428

@mitchellh mitchellh merged commit ba85627 into master Jan 7, 2014

@mitchellh mitchellh deleted the f-bundlerize branch Jan 7, 2014

@phinze

This comment has been minimized.

Show comment
Hide comment
@phinze

phinze Jan 7, 2014

Member

This is a FEAT @mitchellh. Nicely done! 🍻

Member

phinze commented Jan 7, 2014

This is a FEAT @mitchellh. Nicely done! 🍻

@patcon

This comment has been minimized.

Show comment
Hide comment
@patcon

patcon Jan 8, 2014

Contributor

So awesome. Thanks dude

Contributor

patcon commented Jan 8, 2014

So awesome. Thanks dude

@p0deje

This comment has been minimized.

Show comment
Hide comment
@p0deje

p0deje Jan 13, 2014

Contributor

so how does one develop plugins now?

Contributor

p0deje commented on d98868d Jan 13, 2014

so how does one develop plugins now?

This comment has been minimized.

Show comment
Hide comment
@tmatilai

tmatilai Jan 13, 2014

Collaborator

I had to add the plugins's library to $LOAD_PATH and use normal Kernel.require instead of Vagrant.require_plugin. For example:

# In case this Vagrantfile is in the root directory, otherwise adjust the path accordingly
$:.unshift File.expand_path('../lib', __FILE__)

def require_plugin(plugin)
  if Gem::Version.new(Vagrant::VERSION) < Gem::Version.new('1.5.0.dev')
    Vagrant.require_plugin(plugin)
  else
    require plugin
  end
end

require_plugin 'vagrant-foo'

# ....
Collaborator

tmatilai replied Jan 13, 2014

I had to add the plugins's library to $LOAD_PATH and use normal Kernel.require instead of Vagrant.require_plugin. For example:

# In case this Vagrantfile is in the root directory, otherwise adjust the path accordingly
$:.unshift File.expand_path('../lib', __FILE__)

def require_plugin(plugin)
  if Gem::Version.new(Vagrant::VERSION) < Gem::Version.new('1.5.0.dev')
    Vagrant.require_plugin(plugin)
  else
    require plugin
  end
end

require_plugin 'vagrant-foo'

# ....

This comment has been minimized.

Show comment
Hide comment
@p0deje

p0deje Jan 13, 2014

Contributor
Contributor

p0deje replied Jan 13, 2014

@fgrehm

This comment has been minimized.

Show comment
Hide comment
@fgrehm

fgrehm Jan 27, 2014

Collaborator

👏 Great work! 🍻

Collaborator

fgrehm commented Jan 27, 2014

👏 Great work! 🍻

@kowal

This comment has been minimized.

Show comment
Hide comment
@kowal

kowal Apr 21, 2014

@mitchellh I'm just curious - could vagrant possibly introduce something like auto-installing plugins, provided that Bundler can install gems via API? I mean - isn't typing vagrant plugin install a bit redundant if those plugins are already listed in Gemfile? And instead of using vagrant plugin update why not `bundle update``?

kowal commented Apr 21, 2014

@mitchellh I'm just curious - could vagrant possibly introduce something like auto-installing plugins, provided that Bundler can install gems via API? I mean - isn't typing vagrant plugin install a bit redundant if those plugins are already listed in Gemfile? And instead of using vagrant plugin update why not `bundle update``?

@mitchellh

This comment has been minimized.

Show comment
Hide comment
@mitchellh

mitchellh Apr 21, 2014

Member

@kowal Yes, but it introduces some difficult technical and user experience questions that have to be answered before this is ever possible.

Member

mitchellh commented Apr 21, 2014

@kowal Yes, but it introduces some difficult technical and user experience questions that have to be answered before this is ever possible.

@tknerr

This comment has been minimized.

Show comment
Hide comment
@tknerr

tknerr May 28, 2014

Contributor

Concerning the user experience it would probably be best to not mix Vagrant plugins with Gemfile. For example, if we keep it separate:

  • VagrantPluginsfile akin to Gemfile
  • vagrant bundle install akin to bundle install
  • etc..

@mitchellh Would this solve the concerns about the user experience?

About the technical difficulties: are there some major roadblocks on the way? (or just not on the roadmap?)

/cc @mhahn @fgrehm

Contributor

tknerr commented May 28, 2014

Concerning the user experience it would probably be best to not mix Vagrant plugins with Gemfile. For example, if we keep it separate:

  • VagrantPluginsfile akin to Gemfile
  • vagrant bundle install akin to bundle install
  • etc..

@mitchellh Would this solve the concerns about the user experience?

About the technical difficulties: are there some major roadblocks on the way? (or just not on the roadmap?)

/cc @mhahn @fgrehm

@mitchellh

This comment has been minimized.

Show comment
Hide comment
@mitchellh

mitchellh May 28, 2014

Member

Yeah I was thinking of doing that already. Not sure what the file would be called but probably something in that direction. As for technical difficulties, the primary one in this thread is it.

Member

mitchellh commented May 28, 2014

Yeah I was thinking of doing that already. Not sure what the file would be called but probably something in that direction. As for technical difficulties, the primary one in this thread is it.

@ivan-kolmychek

This comment has been minimized.

Show comment
Hide comment
@ivan-kolmychek

ivan-kolmychek Sep 14, 2014

I would like to have Bundler-like/librarian-puppet-like module dependencies for Vagrant too. =)

I would like to have Bundler-like/librarian-puppet-like module dependencies for Vagrant too. =)

@mikeschinkel

This comment has been minimized.

Show comment
Hide comment
@mikeschinkel

mikeschinkel Apr 30, 2016

Hi @mitchellh:

Since it's almost 2 years later, I am wondering if you guys have looked into a way to handle this?

I am currently working on a Vagrant box for local WordPress development and we want to minimize the steps people need to talk to get our box working. Currently we need to tell them to install VirtualBox then Vagrant then a Host Updater plugin then a Triggers plugin.

We would love if we could at least minimize to just VirtualBox then Vagrant and not have to specify plugins but that we could instead specify within the Vagrantfile which plugins need to be installed, but having to install a plugin in order to avoid installing plugins is ironically at cross purposes.

Here is what I'd love to see, commands something like this for the Vagrantfile:

Vagrant.configure(2) do |config|
    config.plugins.auto_install = true;
    config.plugins.add "cogitatio/vagrant-hostsupdater";
    config.plugins.add "emyl/vagrant-triggers";
    ...
end

I am hoping that 2 years later it is finally time for Hashicorp to tackle this...

Hi @mitchellh:

Since it's almost 2 years later, I am wondering if you guys have looked into a way to handle this?

I am currently working on a Vagrant box for local WordPress development and we want to minimize the steps people need to talk to get our box working. Currently we need to tell them to install VirtualBox then Vagrant then a Host Updater plugin then a Triggers plugin.

We would love if we could at least minimize to just VirtualBox then Vagrant and not have to specify plugins but that we could instead specify within the Vagrantfile which plugins need to be installed, but having to install a plugin in order to avoid installing plugins is ironically at cross purposes.

Here is what I'd love to see, commands something like this for the Vagrantfile:

Vagrant.configure(2) do |config|
    config.plugins.auto_install = true;
    config.plugins.add "cogitatio/vagrant-hostsupdater";
    config.plugins.add "emyl/vagrant-triggers";
    ...
end

I am hoping that 2 years later it is finally time for Hashicorp to tackle this...

karlkfi added a commit to karlkfi/vagrant that referenced this pull request May 31, 2016

Revert 3f9fb2e from GH-2769
- For compatibility with ba77d4b

Fixes GH-7073

gitebra pushed a commit to gitebra/vagrant that referenced this pull request Jun 1, 2016

Jonas Johansson
Merge commit 'a7ee56459b2c990550c4fc33883c55cf4145f463'
* commit 'a7ee56459b2c990550c4fc33883c55cf4145f463':
  provisioners/ansible(both): fix ansible config files presence checks
  provisioner/salt: fix orchestrations documentation
  Update CHANGELOG
  provider/docker: Add -u flag to exec
  provider/docker: Separate -i and -t flags for exec
  provider/docker: Add docker-exec command
  Update CHANGELOG
  Revert 3f9fb2e from GH-2769
  Up bundler dep
  Added basic unit test
  Adding static IPv6 address support for Fedora
@ciastek

This comment has been minimized.

Show comment
Hide comment
@ciastek

ciastek Jun 12, 2018

NOTE: As of version 1.9, Vagrant doesn't use Bundler anymore: #7793 [core] Remove bundler usage for plugin management

ciastek commented Jun 12, 2018

NOTE: As of version 1.9, Vagrant doesn't use Bundler anymore: #7793 [core] Remove bundler usage for plugin management

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment