Create extpoint for graphdrivers

[cpuguy83]

Allows people to create out-of-process graphdrivers that can be used
with Docker.

Extensions must be started before Docker otherwise Docker will fail to
start.

For simplicity, uses in-process NaiveDiffDriver for diffing.

[duglin]

testcases?

[cpuguy83]

Yeah, tests are coming.

One issue is it seems that if I use the vfs driver for the "remote" driver, it gets frozen when it hits chrootarchive.CopyWithTar.
Changing to archive.CopyWithTar instead, it works fine.
And of course, overlay and aufs work just fine.

[cpuguy83]

Pushed tests with just using the overlay driver as the external driver for now.
Added exported function to the vfs driver which allows injecting a copying function to use.
I'd prefer to:

1. make sure the test is able to create a real container
2. not implement a working graphdriver just for tests.

Thoughts?

[icecrime]

@cpuguy83 What are the use cases for this? I don't feel we have a need for this right now (it's not like we get dozens of PR about adding graphdriver that we refuse because we don't want them compiled in), so I'm a weak no.

[cpuguy83]

we've already had requests for gluster and ceph... Ceph being one that can't be linked statically either.
It also allows experimenting with new drivers without having to build in.

Also consider drivers for enterprise storage devices like NetApp, EMC, and I could name a few others that would prefer not to be named.
These can all implement their own layering, CoW, etc.

[unclejack]

@icecrime Please allow me to explain why this would be useful and why my vote would be an aye.

You've said that we don't get many PRs for adding new graph drivers and that this might not be needed now. While I do agree that we don't receive many PRs for new graph drivers, we also need to think about the reasons.

Companies like EMC, NetApp and others will most likely not be sending pull requests to add their product specific graph driver to the Docker repository. That can be because of a multitude of reasons: they want to keep it closed source, they want to put some proprietary stuff in there, they'd want to be able to change it, update it separately from Docker releases and so on.

The Ceph driver has been requested before and I've also rejected that idea because we'd have to tie it into Docker by linking to Ceph libraries which are version specific. The last thing we need is to link against more foreign C/C++ libraries and produce multiple Docker binaries for all those systems with all the different Ceph versions. It's obvious that we won't get a Ceph PR because we've turned it down for these reasons, so Ceph support can't happen without plugin support for graph drivers.

These graph driver plugins could live in a separate repository and they could be packaged separately from Docker. The Ceph graph driver would probably be the first one.

@cpuguy83 Perhaps making the necessary changes to allow custom Diff implementations would be a good idea. I also anticipate some changes in the graph driver API and having some kind of versions for this API would certainly help with that.

[icecrime]

Sorry, I missed the comment here.

I see, makes sense then.

[hustcat]

@cpuguy83 I write ceph rbd storage driver plugins refer to your interface, see here. I wish your PR be merged ASAP.

[cpuguy83]

@hustcat Really cool! Could I recommend to change your default graph store dir to not be /var/lib/docker? This should be pretty much private to the docker daemon and you should not rely on its existence.

[hustcat]

Yea,Thanks for your suggestion. I will continue to improve it:)

[thaJeztah]

@hustcat are you still working on this?

[thaJeztah]

Sorry, wrong ping, GitHub autocomplete... meant to type @cpuguy83 😊

[cpuguy83]

This is in design review.
The implementation is done.

[thaJeztah]

Doh! Looks like I'm not having my day, LOL, should not be doing late-night browsing through issues 😉

[mpatlasov]

Brian, I'd like to use your implementation of graphdriver extpoint to resolve issues I attempted to solve by #15594, but this would require a minor improvement of what you have done -- it would be great not to loose the knowledge of "home" and "options" that GetDriver() has as arguments. For example, we could implement mux.HandleFunc("/Plugin.Init", ...) and call it from LookupExtPoint() after plugins.Get(). What do you think?

[cpuguy83]

@mpatlasov This would make sense if Docker is going to initialize the plugin, but if the plugin is already running, it really shouldn't be doing this, and also is a bit weird in cases where maybe this is a multi-host solution.

[cpuguy83]

Ok, this is now fully functional, figured out the issue with chrootarchive in test.

In writing this, I noticed that there is no code path that actually calls graphdriver.Diff, and only 1 code path (which seems to never be called) which calls graphdriver.DiffSize.

[unclejack]

Does anyone think we should have a versioning system for this API? This graphdriver plugins API will most likely get improved once people start building graph drivers and that means we'll have to version it.

[cpuguy83]

@unclejack Right now the overall plugin API is bumped when there are changes to anything, and we don't stay backwards compatible.

[calavera]

We can change the content type to be specific about the API endpoint. We already have a version number there. We don't need another versioning system.

[cpuguy83]

Is there something that should change in this PR?

[kolyshkin]

Sorry for the noise, but is this going to be merged or not? Is there anything I can help with?

[LK4D4]

@kolyshkin Yup, thanks for ping.

[LK4D4]

LGTM
ping @thaJeztah @moxiegirl for docs too

[thaJeztah]

docs LGTM

[moxiegirl]

@cpuguy83 LGTM -- just an FYI I generally don't review experimental until we move it into the docs dir.

[LK4D4]

ping @docker/core-maintainers

[jessfraz]

ok LGTM, shall we fire ze missiles on this one

[mx2323]

@cpuguy83 is there any ability to pass options to the external graph driver? with the volume driver, you can pass in volume specific options that can be used by the driver to customize behavior.. i think that would be awesome to have here as well.

[cpuguy83]

@mx2323 No, the graphdriver is not user-facing.
We may add something like --storage-opt to docker run, in which case that will be possible, but it is not how graphdrivers work right now.

[mx2323]

@cpuguy83 thanks for the quick reply. sounds like it probably wouldn't be that hard to add. i can look into doing it and sending a diff out... thnx

[cpuguy83]

@mx2323 There's an open PR already: #19661