Proposal: Docker Plugins

[dmcgowan]

Proposal for plugins to provide hooks into the Docker engine and extend engine behavior.

Notes as discussed with @BrianBland @shykes @crosbymichael

[nathanleclaire]

Very glad to see momentum in this direction.

It seems to be the case that this model for plugins can just react to things in the event stream (e.g. I'm running a container, so mount a volume), but not actually add brand new functionality to docker. The reason I ask is because I'd rather have stuff like host management (#8681) available as a plugin. Would this model allow the ability to add new subcommands to the Docker CLI?

If we're going to run plugins inside containers, do they show up in the output of docker ps?

How are plugins produced? Presumably they will be written in Go and compiled using the libchan client for starters? Do we require that every plugin be cross-compilable/cross-platform as we enter a multi-OS Docker world?

So do I understand correctly that "built-in" plugins include code compiled as part of Docker itself, and subprocess plugins cause the daemon to spawn a new subprocess on the host?

An administrative client which connects to the daemon through a unix socket using libchan can be used to configure daemon plugins at runtime without restarting the daemon process.

So... is this a separate binary, or part of docker?

How are plugins distributed?

More sample code (what does a simple plugin look like?) and command line examples would be helpful. I'm having a hard time visualizing the actual UI of this from the implementation details.

What are some real-world examples where this model will be useful?

[shykes]

Nathan all good feedback, just a general note to not worry, this is not the
final proposal, far from it! We askedDerek to dump raw notes early and we
will iterate in the open.

@dmcg can you give me write access to the pr so we can iterate side by side?

On Tuesday, November 4, 2014, Nathan LeClaire notifications@github.com
wrote:

Very glad to see momentum in this direction.

I find several bits of this proposal confusing. It seems to be the case
that this model for plugins can just react to things in the event stream
(e.g. I'm running a container, so mount a volume), but not actually add
brand new functionality to docker. The reason I ask is because I'd rather
have stuff like host management (#8681
#8681) available as a plugin.
Would this model allow the ability to add new subcommands to the Docker CLI?

Plugins may either be in process, a sub process, or a container (future
version) managed by the container.

What's the second container here?

If we're going to run plugins inside containers, do they show up in the
output of docker ps?

How are plugins produced? Presumably they will be written in Go and
compiled using the libchan client for starters? Do we require that every
plugin be cross-compilable/cross-platform as we enter a multi-OS Docker
world?

So do I understand correctly that "built-in" plugins include code compiled
as part of Docker itself, and subprocess plugins cause the daemon to spawn
a new subprocess on the host?

An administrative client which connects to the daemon through a unix
socket using libchan can be used to configure daemon plugins at runtime
without restarting the daemon process.

So... is this a separate binary, or part of docker?

How are plugins distributed?

More sample code (what does a simple plugin look like?) and command line
examples would be helpful. I'm having a hard time visualizing the actual UI
of this from the implementation details.

What are some real-world examples where this model will be useful?

—
Reply to this email directly or view it on GitHub
#8968 (comment).

[dmcgowan]

I think we should figure out how to extend the plugin model to not just reactive, but to add features. We want to come up with a proposal that will establish a new method for adding features such as host management in way that can easily broken out into external plugins. The engine/job model was a good first step toward this sort of model, with libchan we can extend it farther and get a fully pluggable architecture, at least that is the goal of the design.

@shykes you should have permission now

[errordeveloper]

Can someone add "Proposal" tag to this?

[erikh]

@errordeveloper done

[cpuguy83]

I think the pipeline model is probably the more powerful of the two since it allows for most of the benefits of a hub model, since a plugin could just take the data and pass it on w/o modification, while also having the power to actually be able to change behavior (ie, modify the data that is sent to the next thing).

[dqminh]

The reason I ask is because I'd rather have stuff like host management (#8681) available as a plugin. Would this model allow the ability to add new subcommands to the Docker CLI?

I think the plugin system comes in two form, daemon plugins and client plugins. IIRC, this is about daemon plugins while #8681 is implemented purely in client side. I think git subcommand plugin architecture can be a good fit for client plugin ( internally git's main binary reexec git subcommand as git-subcommand if subcommand is not implemented in the main binary )

Regarding the model, i think the pipeline looks better.

• Will the pipeline be executed synchronously or asynchronously. I prefer the execution to be sync because then it can be used to implement hooks Hooks for more customization #6982 and/or overriding the core behavior.
• since the plugin can be sub-processes executed by docker, how should docker handle the lifecycle of the plugins ( when they misbehaved / terminated )

[tonistiigi]

What are in-process/built-in/default plugins?
Why do we need in-process/sub-process plugins?

IMHO, plugins should all run in their own container and be distributed the same way as docker images are. We already have a top of the line way to run and distribute software, why not reuse it. Starting to use a plugin should be as simple as it is to run a container from the hub. Whether they appear in docker ps or have "label" that makes them hidden by default is not something very critical. I'd suggest to remove them from there and add something as simple as docker plugins use hosts and docker plugins ps.

This does not mean in any way that plugins are limited to listening event stream from the main daemon socket. Neither should they be limited to running Engine jobs. Where the hooks or exposed features are needed should be decided per use case. Anywhere hosts or cluster need it, they should be added. From current community "plugins" I'd suggest ensuring that weave and cadvisor could be implemented. We do not need to worry about getting the plugin API 100% right the first time or backwards compatibility. Plugin itself should define what versions of docker it supports(probably smth like META proposal #8955). When user updates docker they should update the plugins as well.

Even if a plugin needs to break out of its container to modify some host property it should be distributed like the rest. You can already break into a host with just access to the daemon socket, but I guess we can provide a nicer way to do it. As a plugin author I'd rather take this extra step and get a nicer distribution in return.

@dqminh There is no need for client plugins. Something like docker hosts lets the daemon know what kind of actions it supports. Then the daemon exposes these commands to any client connecting to it(smth like #7631). The libchan connection can be used both ways.

[shykes]

Totally agree with using containers as the delivery mechanism for plugins. At the same time I think some sysadmins might appreciate an "escape hatch" to configure their host to execute an arbitrary binary on their host, a-la git hooks. At least, I think that will be necessary until we have a convenient way to build very small images. See for example #7115 .

[erikh]

shouldn't this be --plugin-exec <name> <command> instead of just <command>?

[huslage]

I fear that this is going to be a UX problem. While it's technically appropriate to do a git-like plugin system or even to run containers for plugins it will be nearly impossible to use for normal humans. Git, IMO, should never be used as a marker for good UX.

[erikh]

A couple of comments:

1. Shell scripts do not care about libchan. I don't think requiring it, or even requiring a bridge, is the best idea. They aren't going to be able to consume streaming content trivially anyways, so that limits them to a few cases. However, we should be very comfortable supporting them. I'm offering an alternative to the protocol below (but only slightly).
2. The way the options are represented will give our option parser hives. perhaps --plugin name=value and --plugin-exec name=value instead.

Protocol

Similar to the proposal we have static fd's that are used for communication.

Each frame has an envelope that contains a single line string declaration of the format of content: e.g., json, or map (k/v one=two like env) or chan. This allows the tool to select the transport it can handle the best. libchan can encapsulate or convert the data from the fd's if it is to be used to replace the job system (which ftr I am 100% behind).

Conclusion

While it is very ideal to have libchan everywhere, I just don't think that'll be realistic nor will a lot of people care if their plugin is hard to use/develop.

[cpuguy83]

This was my thought as well... docker run --plugin <some image>, which would inject a libchan socket from the daemon, the process started would register itself and wait.

A lot of this feels like swarmd (just not in-mem) to me, with the added that the plugins would exist in a container.

@erikh I just think we'd have a generic (libchan) plugin that is for execing and proxying the returns back.

[erikh]

I really don’t see how that solves anything. What would be the input/output format? How do we handle streaming content, if at all? What about languages that need to fork/exec this process and pipes are hard to use within them (e.g., python or ruby)?

Distributing another binary could be problematic as well.

On Nov 5, 2014, at 11:22 AM, Brian Goff notifications@github.com wrote:

This was my thought as well... docker run --plugin , which would inject a libchan socket from the daemon, the process started would register itself and wait.

A lot of this feels like swarmd (just not in-mem) to me, with the added that the plugins would exist in a container.

@erikh https://github.com/erikh I just think we'd have a generic (libchan) plugin that is for execing and proxying the returns back.

—
Reply to this email directly or view it on GitHub #8968 (comment).

[cpuguy83]

@erikh I don't see how input/output format is any more relevant (or why it should be any different) for exec'd stuff vs stuff going over the socket.
The interchange happens over stdin/stdout/stderr.

[erikh]

No, it happens over fd’s 3 and 4.

On Nov 5, 2014, at 11:39 AM, Brian Goff notifications@github.com wrote:

@erikh https://github.com/erikh I don't see how input/output format is any more relevant (or why it should be any different) for exec'd stuff vs stuff going over the socket.
The interchange happens over stdin/stdout/stderr.

—
Reply to this email directly or view it on GitHub #8968 (comment).

[cpuguy83]

@erikh I'm speaking specifically of a plugin that is for execing, not the --plugin-exec (though, I suppose these should be basically the same).

[erikh]

I’m still not clear how a shell script would interact with libchan, and I believe this is confusing matters.

—plugin-exec link=/path/to/mylinktool.sh

This references the ‘link’ job in the bridge driver. How does this work?

On Nov 5, 2014, at 11:49 AM, Brian Goff notifications@github.com wrote:

@erikh https://github.com/erikh I'm speaking specifically of a plugin that is for execing, not the --plugin-exec (though, I suppose these should be basically the same).

—
Reply to this email directly or view it on GitHub #8968 (comment).

[nathanleclaire]

I had sort of envisioned plugins as simply apps running inside containers that get the docker socket and binary injected by default, perhaps with some additional handling around permissions e.g. a plugin.json that defines something like

{
    "name": "docker-volumes",
    "permissions": {
        "RunContainers": ["ReadVolumes", "WriteVolumes"]
    }
}

(terrible example, but hopefully you get the idea)

As @tonistiigi mentioned, they would be containers which are invisible to docker ps but show up in docker plugins or docker plugins ps. Nothing particularly special about them, they're just containers. With the right implementation, people could start writing plugins immediately in shell, Python, Go or any language they prefer / has decent bindings and distribute them via the Hub.

So the added libchan complexity really confuses me. What are the benefits?

[cpuguy83]

@nathanleclaire This is what plugins are today (minus the restricted actions you'd mentioned). The problem is access to the remote API isn't enough for many use-cases.

I don't know what the actual implementation will look like, but in 60 lines of Go I was able to make a libchan client that can call (almost) any engine job ("containers", "create", "logs", "allocate_interface") on the fly, and handle the return properly (be it a stream of data like "logs" or a simple json response).
A small subset of jobs are available via the REST API (as it should be).

Plugins could also potentially replace existing jobs... For instance maybe (and this is completely hypothetical) a plugin would replace the allocate_interface job so that instead of allocating a new docker bridge interface, it would do something with ovs.

[shykes]

Nathan the limitation of using regular remote api to communicate with
plugins is that it's not suitable for passing logs or lifecycle events to a
plugin, or a worker model where the plugin must first act as a client
(to register itself) then as a server (listening for events and commands
to process).

In exchange for those limited capabilities we would not get much benefit ,
since the remote api is pretty complicated to understand and implement . If
you ask each plugin to implement a working remote api implementation... We
won't get many plugins.

Libchan by contrast is appropriately powerful AND 100x simpler to use than
the current remote api.

To address @erikh's comment, we would need to provide helpers for your dev
environment of choice (including shell scripts) but as soon as you're
talking about sending structured data back and forth on a socket, you're
going to need that regardless - bash can't parse http or json anymore than
it can msgpack and libchan. So it boils down to the quality of the tooling.

On Wednesday, November 5, 2014, Brian Goff notifications@github.com wrote:

@nathanleclaire https://github.com/nathanleclaire This is what plugins
are today (minus the restricted actions you'd mentioned). The problem is
access to the remote API isn't enough for many use-cases.

I don't know what the actual implementation will look like, but in 60
lines of Go I was able to make a libchan client that can call (almost) any
engine job ("containers", "create", "logs", "allocate_interface") on the
fly, and handle the return properly (be it a stream of data like "logs" or
a simple json response).
A small subset of jobs are available via the REST API (as it should be).

Plugins could also potentially replace existing jobs... For instance maybe
(and this is completely hypothetical) a plugin would replace the
allocate_interface job so that instead of allocating a new docker bridge
interface, it would do something with ovs.

—
Reply to this email directly or view it on GitHub
#8968 (comment).

[nathanleclaire]

OK, that helps clear stuff up a bit, thanks.

[thaJeztah]

At the same time I think some sysadmins might appreciate an "escape hatch" to configure their host to execute an arbitrary binary on their host

Maybe I'm naive, but couldn't this still be achieved with the "containers" approach by bind-mounting that binary inside a container? If adapters are required (e.g. a shell-script adapter), distribute that as an image, and bind-mount the host binaries inside that container to "convert them" to a plugin.

[ibuildthecloud]

I don't think I fully understand the description of the options, so I'll put my random comments here and we'll see how they align with what is being proposed.

Implicit vs. Explicit

There are two basic approaches for plugins that I see. Either plugins are implicitly called as part of the regular execution of code or explicit changes are made to the code to invoke plugins at specific places.

I believe the implicit approach is inline with the pipelining approach described. In this approach we can essentially make one code change and then most everything in Docker is theoretically pluggable. The implicit approach looks almost like aspect oriented programming or a message mediation layer seen in something like an ESB. Basically we provide a framework such that calls can be intercepted and input/output data can be modified. This means the caller is completely unaware of the plugin.

The explicit approach is where the caller is aware of the notion of a plugin and explicitly looks up and invokes a plugin with a defined input, expecting some known response format. In the explicit approach we must make a code change for every extension point we want to provide.

I strongly discourage the implicit approach. While it is very tempting to make one generic change I think it has some real dangers. The first dangers is that it can effectively impact the quality of Docker. If anything and everything is pluggable, knowing the cleverness of the Docker community, people will plugin wherever they can and do totally unexpected things. The problem we will create is basically the same as what Firefox went through. With Firefox extension you can change just about anything in Firefox. This was great until people started installing a lot of extensions that were poorly written. The overall speed and stability of Firefox declined and people got a bad impression of Firefox when a lot of it had to do with the extensions installed.

I think Docker should be careful in what hooks are provided, how it is done, and what are the possible impacts.

The second downside of the implicit approach is that it rarely works as magically as one would hope. The implicit approaches makes different calling patterns difficult, limits the contextual information to the input message only, and makes coordination of operations across calls difficult.

In the proposal for networking (#8997) I just created, I would like it to be plugin based. The calling mechanism of the plugin, as I've described it, requires the caller to dynamically choose the correct plug-in to be used by iterating the list of plugins and choosing the best one. This calling pattern is difficult in the implicit approach. First, since the caller is unaware of the multiple plugins that exist, the plugin selection must be done in a sort of meta plugin. So the meta plugin intercepts the message, then finds the best plugin. In the specific example in my proposal, the information needed to select the right plugin would not exist in the message being sent. The message being send would essentially have just container ID, and NetworkSettings, but the data needed to select the right plugin is from the -net parameter. You could just say well put the value of the -net parameter into the message, but now you've broken the magicalness of this approach because the caller is adding new data to make the implicit operation easier. So you can imagine with other existing messages in Docker that plugins might not have enough context data to operate properly.

Plugin Lifecycle/Statefulness

Docker itself maintains a good amount of state in memory. Additionally Docker relies on go's defer capabilities to do things like proper cleanup. For example, because of the awesomeness of go you can exec a process, defer a cleanup routine and then to exec.Wait. Both of these things I just described would not be possible if the plugins are forked on every execution.

We need to explicitly call out what is the intended lifecycle of a plugin. I'm guessing, based on what is implied in this proposal, is that a plugin would be launched once for the life of the Docker daemon process. This is fine and allows for plugins to keep stateful in memory information. But what happens if a plugin crashes? Do we restart it (probably). If we restart it, the plugin should know its being restarted/start so that it could possibly rebuild the state it needs.

How to obtain additional information

Inevitably plugins will need to obtain information outside the scope of the data being passed in. There's two approaches I see here. One is that the plugin could just use the Docker Remote API to obtain additional context. A second approach is that we actually start creating additional internal libchan based APIs to get data not publicly available.

The second approach may eventually be needed, but I would defer until I see a real use case. I think the first approach will initially be needed. I would propose that we pass on fd 5 a connection that effectively is the same as opening the Docker socket. The only difference is that connection should be a pre authenticated channel. Docker auth for the API doesn't exist yet, but it's going to be needed very shortly. By passing in the connection to the socket, we allow ourselves down the line to do more explicit security policies around what a plugin can access and do.

Multiple plugins per extension

Based on the nature of the extension point multiple plugins may or may not be needed. I think the plugin registration should allow multiple plugins to be registered for an extension point and a priority can be assigned to the plugin. This will effectively create an ordered list of plugins in memory. Callers to the plugin (assuming an explicit model) can the call GetPlugin(...) to get the first plugin (highest priority) or GetPlugins(...) to get a complete list ordered by priority.

[dmcgowan]

@ibuildthecloud

Implicit vs. Explicit

I am currently leaning in the middle toward a hybrid approach which would basically allow for pipeline plugins as a callable endpoint on the channel hub (should have linked docker/libchan#61 for how I envision the base channel hub implementation). I understand the arguments for the hub being kind of a messy model for piping lifecycle events, but I think having the explicit registration will be needed to replace much of the existing job framework. I will be convinced of only pipelining (implicit) when it I see a viable model for replacing the existing jobs, which I am going to try and prototype. I am curious what information do you think would be missing and could not simply be part of the pipelined object passed into the plugins. Understood if there was just a basic event description that may not be enough information, but I was thinking of the pipelining model as able to send any rich object and have the data type specific to the message type.

Plugin Lifecycle/Statefulness

The doc should probably be modified to mention more about state. The thought is that anything that may need to be persisted across calls to a plugin should store state by calling a plugin. The state plugin would need to worry about persistence and expose an interface (through plugins) for simple object storage. Then when a plugin comes online (whether after a crash or clean restart shouldn't matter) it would be able to rebuild any internal state needed.

How to obtain additional information

Perhaps we should explore how we can do such calls using the pipelining model, it should be possible since it is already libchan based. For example it will be possible to invoke a message to the plugin system with a return channel for data. The hub case is more clear cut on how this is possible since it is essentially as you described, an internal libchan based API. Either way this will be enabled by using libchan requiring no tie in with the existing Remote API. In fact I think that the Remote API and the plugins API should be separated as much as possible to not create any sort of undesirable dependency.

Multiple plugins per extension

In case where multiple plugins are registered at the same point we should come up with a way to order them. I intentionally left this out of the doc for now since I didn't have a solution when writing. I think we should assume that ordering will be needed but until we decide on what the architecture looks like we can't start incorporating into the Plugin API.

[SvenDowideit]

@dmcgowan if you can line wrap to 80 chars, that would be great (its a current docs requirement) - obviously, not urgent tho :)

[squaremo]

MBOI to people looking at the plugin proposal: #9013 describes "Container annotations", the main motivation for which is to be able to attach arbitrary values to containers for plugins to interpret.

[goehmen]

@dmcgowan @shykes Looks like you are off to a good start. Ironically, I just cut v6.7.0 of the Cloud Foundry CLI yesterday and introduced a plugin MVP to CF's community. There is a great github issue where a conversation occurred for the CF CLI similar to your's here [0].

I'm happy to compare notes, share experience, have a coffee, whatever. Especially since Docker joined the Cloud Foundry Foundation [1]. :-) Cheers & good luck!

[0] cloudfoundry/cli#123.
[1] http://blog.docker.com/2014/05/docker-joins-cloud-foundry-foundation/

[danehans]

@dmcgowan what is the next step for moving plugins forward? Is their a G+ meeting to finalize the design details? Has the design been finalized? Is code available for any of the design options? Any feedback you can provide would be appreciated.

[cpuguy83]

@danehans I'm working on an implementation of exec plugins and applying that to graphdrivers atm: https://github.com/cpuguy83/docker/tree/poc_storage_plugin

[danehans]

@cpuguy83 I am interested in creating a network plugin POC. Are their any other references I should be aware of? Do you have any recommendations before I start the POC effort?

[cpuguy83]

@danehans I would say to hold off atm, I know @erikh has been working on something in that area, not ready to show off yet.

[danehans]

@erikh I have been reviewing the network related commits in your vxlan and extensions branches. Are their any other branches I should reference to bring myself up to speed?

[icecrime]

Sorry, I'm gonna close this as we have ongoing initiatives and this PR seems to be stalled (cc @lukemarsden @aluzzardi @shykes).