Bundler should have a plugin system #1945

indirect opened this Issue May 22, 2012 · 17 comments


None yet

9 participants


Based on previous discussions, I think there are two main Bundler extension points right now: First, the ability to add arbitrary commands to bundler itself (like your bundle license command). The second obvious place for extensions would be callback hooks around the built-in bundler commands. For example, a bundler plugin that installs ri and rdoc for the gems in your Bundle after the install or update command has completed.

The trickiest part of a plugin system for Bundler is how to distribute and activate such plugins. If they are distributed as Rubygems, there are potential issues around when the plugins get activated, and how fragile gem activation can be already. I can attest to many, many hours spent debugging unpredictable issues with Rubygems plugins, and I would like to avoid a repeat of that situation if possible.

Additionally, loading Bundler plugins could then potentially be an issue anytime you run a Bundled command from inside another Bundled command. The easiest way to reproduce a situation like that is to run bundle exec bundle exec rake. The "inner" Bundler would never even have access to the system-level Rubygems to try to find plugins and load them. On the other hand, if Bundler plugins have to be included in the Gemfile, then they only apply to a single project, and that seems pretty lame, too.

Bundler plugins that are system-level rubygems might be more feasible once with_original_env restores the original GEM_HOME, which is in progress in #1920.

@indirect indirect was assigned May 22, 2012

Perhaps this should be similar to the way Heroku handles plugins? With the Heroku gem, plugins are stored in $HOME/.heroku/plugins so they're always available regardless of RVM gemsets or Bundler environments.


Seems like it might be a reasonable strategy. Bundler already has the concept of a .bundle directory, both inside each project and inside a user's home.

How are Heroku plugins installed? Where are they fetched from? Does the Heroku command contain its own plugin manager?


The Heroku CLI contains its own plugin manager, and fetches them (generally) from git. Installing a plugin goes something like this:

heroku plugins:install git://github.com/ddollar/heroku-accounts.git

List installed plugins like this:

heroku plugins

Unistall like this:

heroku plugins:uninstall heroku-accounts


Just to add to that, each plugin has an init.rb in its root directory (like an old-style Rails plugin) that the Heroku client looks for, that file loads the plugin and hooks everything up where it's needed.


Seems like that strategy could potentially work for Bundler. It's really important that Bundler be more robust than most Ruby tools, so hopefully we can come up with a strategy that allows the main functions of Bundler to keep working even in the face of possibly dodgy plugins.


Yeah, there may be a better way, but handling plugins the way the Heroku client handles them was the first thing that came to mind when I saw this. 😁

@hone would probably have more insight than I do about how that works in their client (I just know how I've used it lol).. but I might be able to take a look at their code this weekend and get a better idea of how everything fits together.


Ok, I see how Heroku does it but I'm not too sure if the same technique would work for Bundler (I'm not really too familiar with Thor).


I've been thinking about the plugin system some more, and just posted Towards a Bundler plugin system on my blog. Feel free to chime in here with questions or comments about that post as well as the plugin system in general.


@indirect Thanks again for sharing your thoughts. This proposal looks great. I've spent a few weekend nights hacking on what I hoped could lay the foundation for this, but every time got sidetracked and deterred by complexities that inevitably arose. :)

I'd still like to help with this, and seeing your initial ideas fleshed out in code could really help get this moving. Very exciting.


Just read the blog post about this. A new feature I've wanted into Bundler is "subgroups." I have a pull request that's been open for a while doing some exploratory work on this: #1863. I'm wondering what kind of hooks would be needed in the plugin system to allow a feature like this to exist via a plugin.


Having supported a plugin-API in Vagrant for over a year now, I'd like to add my two cents here so you can hopefully avoid the problems I ran into.

First, I think plugins for bundler, or for any project really, are a fantastic way to vastly expand the original feature set of a project and also to have a test bed for future core features. This is the right direction to take.


RubyGems will work great. Also, great job on not "auto activating." Vagrant did this for some time and it was a terrible mistake from a performance perspective.

One issue you'll run into pretty early on is conflicting dependencies. One gem will require fog ~> 1.2.0 and another will require fog ~> 1.3.0 and they'll conflict. Since Bundler is a Ruby-based tool, this actually probably won't be any problem because your users are Ruby devs and they'll understand the error. For Vagrant, this is more complicated because a majority of my devs actually aren't Ruby devs.

But all in all 👍 here.


Now, let's talk about API. API is where things get tricky. When I first introduced Vagrant plugins, I thought "my classes are so elegantly designed, I'll just expose the core internals!" This appealed to me because then plugins would basically dog-food the internals (or vice versa, depending on how you want to look at it). I thought that if I exposed the internals I had a bigger incentive to keep them stable because plugins depended on them and so on.

This was a terrible decision. Even if internals are perfect at some point, future features and changes eventually lead to refactors and reorganizations and changing any internal classes suddenly break plugins.

Plugins need a layer between the internal classes and themselves, so that they are buffered from internal change.

How Vagrant (1.x+) Does It

So how do I buffer plugins from internal change? The plugin system in Vagrant now separates the plugin API into two parts, which is so far working great:

  1. Plugin definition.
  2. Plugin functionality

I guarantee backwards compatibility for all time for plugin definitions, where as plugin functionality will cease to work at certain versions of Vagrant (major versions specifically). The idea is that _loading plugins should never crash_ Vagrant, even if they are incompatible with the current version.

Plugin Definitions

Plugin definitions in Vagrant look something like this:

# Create a class to define a plugin. The class name is insignificant.
# The superclass specifies the version of the plugin interface to load
# and returns the proper superclass.
class MyPlugin < Vagrant.plugin("1")
  name "my plugin"

  command("foo") { MyCommandClass }

There are a few main parts that need to be explained here:

  • Vagrant.plugin("1"). I knew from the beginning that future versions of Vagrant will have new pluggable components. To support this, I knew that plugins would then adhere to different "versions" of the plugin interface, so I would need to support different superclass versions. In addition to this, I decided to make the superclass a method Vagrant.plugin instead of a class like Vagrant::Plugin::V1, because I wanted the flexibility to change my namespacing later, whereas I can feel comfortable committing to never changing Vagrant.plugin. The choice to use a class instead of some fancy block-based DSL has no real reasoning behind it.
  • Defining components (i.e. the command method above). This part tells Vagrant what this plugin does. In the case of the above example, it is telling Vagrant that it has a command "foo." The class that implements this command is given in a block so that it can be lazy-loaded. This is critical to the idea that _loading a definition never crashes, while _activating a plugin can crash (i.e. the MyCommandClass may use parts of Vagrant internals that will change in the future).

Plugin Functionality

As an example for the functionality, let's see what the MyCommandClass from above would look like:

class MyCommandClass < Vagrant.plugin("1", :command)
  # ...

This is pretty standard. Some class implements some API which is specific to each component. I decided to use a second argument to Vagrant.plugin method in order to return a different class so I can continue to change namespacing of my classes without worrying about breaking plugins. That's about it there.

Pros/Cons of this Approach

There are pros to this approach:

  • As long as you commit to never changing the definition API, old plugins will continue to load without breaking forever. That means that plugins written for Vagrant 1.1 will continue to load properly still for Vagrant 9.0 theoretically. This is not even painful to support, because remember the plugins only need to be able to load, not actually function.
  • When a new version of Vagrant comes out, I can detect old versions of plugins and properly warn the user that they're out of date and can't be loaded. For example, it would be pretty trivial to output "Plugin "foo" was written for Vagrant 1.0 and will not work with this version of Vagrant. Please look for an update if you want to use this plugin."
  • It is easy for users of plugins to quickly dive into a plugin and see what the plugin does without actually seeing the implementation. A user only needs to look at a definition to know "oh this plugin does X, Y, and Z, and works for version W."
  • Clear separation of plugin API and internal API. As I mentioned earlier, this is important.
  • Maintenance is easy, since you only need to maintain the definition API, which is "dumb" anyways (the brains are in the implementation, the definition just points you to the implementation).

There are also some cons, actually one that I've thought of (though there certainly can be/are more):

When you upgrade your interface version and decide to outdate the previous version, all previous plugins now fail to work, and each developer has to upgrade their plugin to support your latest version. This is true even if the interface for some component didn't change. For example, imagine in addition to commands, you add support for plugins to hook into other parts of the system, and you decide to add this to the V2 plugin interface, even though the command interface didn't change. By supporting only the V2 interface, all command-based plugins for V1 fail to work even though they technically could.

With Vagrant, I've decided this is okay, because I promise plugin compatibility for a major version (1.0, 2.0, etc.) but I make no promises for plugins between versions. I think this is a fair choice, because major versions are very far apart (years).

In Closing...

The message I want to leave with is: think about the API and how your backwards compatibility policy will be going forward BEFORE it is too late. I also see this as a good opportunity to get feedback on my plugin design approach, which I feel has been working well.

As I said before, plugins for Bundler are a good idea and I wish you the best of luck going forward!


@indirect, thank you for starting this discussion.

In your blog post you asked what hooks people would need. If there were a plugin system today, I would write a "bundle profile" plugin that tells me which gems in my bundle take the longest to load. To do this I would need these hooks:

  • before_require - passes in the name of the file that is about to be required. I suppose it wouldn't hurt to also pass in the dependency object.
  • after_require - passes the file name and dependency again. Also passes the LoadError instance that was raised, if any.
cowboyd commented Mar 21, 2013

@mitchellh Perhaps one way to mitigate your one "con" would be to allow plugins to specify flexible dependency ranges with semver a la Gem::Requirement This would mean that plugin developers would not need to release separate versions of their plugin merely to satisfy version compatibility, especially when the underlying code did not change at all. They might need to install a new version, but that version could be compatible with older versions of the host system.

Another alternative might be a configuration option to force loading of outdated plugins anyway in the event that they are actually compatible despite indicating an incompatible version.

Also, I'm curious about the choice to version at the extension point level and not the plugin level (e.g. passing a version to the command definition). i.e. does it make senes to have a command that is compatible with v2, but its plugin requires v3?

This looks like good work, and I'm thinking about modeling the sputnik plugin system on it. How are things looking now 8 months out? Any more pros/cons to speak of?


@cowboyd Thing are working great. To address the con:

It is only slightly a con. The underlying code can still remain the same, the plugin just needs to update to add a new version definition, which is a really minor change. So I still stick by the original approach rather than making it more complicated with SemVer.

So far, it is working great though, I've received many compliments on the structure of the Vagrant plugin system! :)


@cowboyd Also, developing Vagrant plugins is fully documented here, so you can check that out if you want to base another plugin system on it: http://docs.vagrantup.com/v2/plugins/development-basics.html

sheerun commented Apr 2, 2013


@xaviershay xaviershay referenced this issue in bundler/bundler-features Aug 12, 2013

Plugin system #8


Great discussion! I moved it to bundler/bundler-features#8, summarising the existing points.

@xaviershay xaviershay closed this Aug 12, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment