New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

A new upstream project to break up Docker into independent components #32691

Merged
merged 2 commits into from Apr 20, 2017

Conversation

Projects
None yet
@shykes
Collaborator

shykes commented Apr 18, 2017

Work has been ongoing to break Docker into modular components for some time, with runc and containerd as examples. Containerd for example has been donated to the CNCF. We are now completing this work with the goal being that the monolithic docker repo eventually ceases to exist, instead being assembled from a set of components.

Docker is, and will remain, a open source product that lets you build, ship and run containers. It is staying exactly the same from a user’s perspective. Users can download Docker from the docker.com website.

Moby is a project which provides a “Lego set” of dozens of components, the framework for assembling them into custom container-based systems, and a place for all container enthusiasts to experiment and exchange ideas. You can learn more at http://mobyproject.org

Docker the product will be assembled from components that are packaged by the Moby project.

The Moby project provides a command-line tool called moby which assembles components. Currently it assembles bootable OS images, but soon it will also be used by Docker for assembling Docker out of components, many of which will be independent projects.

As the Docker Engine continues to be split up into more components the Moby project will also be the home for those components until a more appropriate location is found.

Docker is transitioning all of its open source collaborations to the Moby project going forward.
During the transition, all open source activity should continue as usual.

This pull request changes the README and kicks off the process of breaking up the engine under Moby. Again: if you are a Docker user, nothing changes.

README.md Outdated
@@ -276,29 +84,5 @@ For more information, please see https://www.bis.doc.gov
Licensing
=========
Docker is licensed under the Apache License, Version 2.0. See

This comment has been minimized.

@AkihiroSuda

AkihiroSuda Apr 18, 2017

Member

Docker -> Moby?

@fsouza

This comment has been minimized.

Contributor

fsouza commented Apr 18, 2017

Is updating the import paths within the moby project part of the scope of this change?

Right now if I do go get -d github.com/moby/moby/... this repo will get cloned twice because moby's source is still using the import path github.com/docker/docker. And since go get doesn't do shallow clones, cloning this repo twice can take a lot of time, specially if you're using some conference's wifi :)

Thanks!

@netroby

This comment has been minimized.

netroby commented Apr 19, 2017

Did that means docker will all to moby

docker run   =>  moby run 
docker build => moby build

Am i right?

@mwarkentin

This comment has been minimized.

mwarkentin commented Apr 19, 2017

@netroby I think docker CE / EE will continue to exist, and the CLI will remain the same - however they'll be built out of these moby components.

@omeid

This comment has been minimized.

omeid commented Apr 19, 2017

This is not gonna work nice for all the projects that depend on github.com/docker/docker. No notice, nothing, and just breaking everything. Not cool.

@netroby

This comment has been minimized.

netroby commented Apr 19, 2017

The docker/docker repository does not exists. only moby/moby..

Where is the docker ce code? inside the moby project?

@shykes

This comment has been minimized.

Collaborator

shykes commented Apr 19, 2017

@omeid nothing should break. We have moved the repo before (from the dotcloud org) and your go get should handle the redirect gracefully.

In parallel we are setting up a facade URL for go packages (which we should have done long ago) to separate package names from git hosting.

@shykes

This comment has been minimized.

Collaborator

shykes commented Apr 19, 2017

@netroby the Docker CE code is spread out in dozens of repos: containerd, libnetwork, swarmkit etc. Some of it is in moby/moby, and is being spun out into new components until moby/moby only includes the tooling to assemble Docker CE from the components (or your custom alternative to Docker CE).

@kunalkushwaha

This comment has been minimized.

Contributor

kunalkushwaha commented Apr 19, 2017

What are the plans for Infrakit and Swarmkit? Will they be part of docker.org or may come to Moby project, considering they are vital part of docker container platform.

@frekele

This comment has been minimized.

frekele commented Apr 19, 2017

It no makes sense to change the name of the project already widespread in the community, many projects will be broken because of this, change without any warning.
I think there is a lack of common sense with a community.

I'm very confused by this change.
Moby = Open Source Development?
Docker CE = Open Source Release to Production? Or commercial now? or Dead?
Docker EE = Commercial Release to Production? or Dead?

Where is the Docker CE now?

Sorry, but it's not clear to me. Could you explain the structure of the project better and how will the licenses stay?

@shykes

This comment has been minimized.

Collaborator

shykes commented Apr 19, 2017

@frekele

Moby = open source development
Docker CE = free product release based on Moby
Docker EE = commercial product release based on Docker CE.

Nothing is dead; and everything that was open-source remains open-source. In fact we are open-sourcing new things.

@frekele

This comment has been minimized.

frekele commented Apr 19, 2017

Thank you @shykes for the feedback.

It would be interesting to create a migration guide from docker to moby, so we can better understand what it affects, before the next release.

@duffn

This comment has been minimized.

duffn commented Apr 19, 2017

Moby = open source development
Docker CE = free product release based on Moby
Docker EE = commercial product release based on Docker EE.

Nothing is dead; and everything that was open-source remains open-source. In fact we are open- sourcing new things.

@shykes This should be made markedly clearer then. A few line GitHub issue without additional reference is not enough for your many, many users.

@shykes

This comment has been minimized.

Collaborator

shykes commented Apr 19, 2017

@duffn understood we will refine the explanation. We wanted to get things started in the open as early as possible, so that we can prepare the transition in the open woth all the contributors and maintainers.

Note: for the huge majority of Docker users nothing changes at all: the binary packages remain exactly the same. The github repo is used by thousands of people - the binary packages by millions.

@shykes

This comment has been minimized.

Collaborator

shykes commented Apr 19, 2017

@frekele good idea. the short version is that Docker CE and EE users are not affected at all. The binary releases remain exactly the same.

Moby will get its own release process - probably source releases instead of binary. We need to discuss with the maintainers what it will be. We have a summit thursday to get things started.

@shykes

This comment has been minimized.

Collaborator

shykes commented Apr 19, 2017

@kunalkishwaha SwarmKit and InfraKit will remain standalone like today and will not be affected. We would like to move them into separate standalone github orgs to clarify that we want them to be open and not dependent on Docker. We will discuss that move with the maintainers first. We will use Moby to develop and integrate SwarmKit and InfraKit, but they will not be "owned" by Moby.

@panuhorsmalahti

This comment has been minimized.

panuhorsmalahti commented Apr 19, 2017

Will Docker CE be in part proprietary in the future? (Maybe it already is and I missed the news).

@Vanuan

This comment has been minimized.

Vanuan commented Apr 19, 2017

@panuhorsemalahti AFAIK there was never a way to build Docker CE from sources. Now it seams there is.

I think it will be like Chrome vs Chromium.

@krasi-georgiev

This comment has been minimized.

Contributor

krasi-georgiev commented Apr 19, 2017

It took me a while and a bit of reading, but I think now I understand the idea and really like it.

The "docker" idea just goes a level deeper by using the Moby project to build the base OS on which to run the containers.

I think container oriented kernel changes would be one of good the side effects for this transition.

@gaocegege

This comment has been minimized.

Contributor

gaocegege commented Apr 19, 2017

Since moby and docker focus on different aspects, why not keep two repos 🤔

And, could moby build the unikernel, instead of minimal linux distribution?

Anyway, congratulations 🎉

@krasi-georgiev

This comment has been minimized.

Contributor

krasi-georgiev commented Apr 19, 2017

@gaocegege if I understand it correctly moby will use linuxkit as the moby base cli #32693 and looking at the linuxkit repo there is already support for unikernels.

https://github.com/linuxkit/linuxkit/tree/master/projects/miragesdk

@catap

This comment has been minimized.

catap commented Apr 19, 2017

@shykes

Moby = open source development
Docker CE = free product release based on Moby
Docker EE = commercial product release based on Docker EE.

Am I right that Docker EE will be something not related to Docker CE/Moby?

@thaJeztah

This comment has been minimized.

Member

thaJeztah commented Apr 19, 2017

Am I right that Docker EE will be something not related to Docker CE/Moby?

The last line should probably be;

Docker EE = commercial product release based on Docker CE.

Docker EE is on the same code base as Docker CE, so also built from Moby, with commercial components added, such as "docker data center / universal control plane"

@zaquestion

This comment has been minimized.

zaquestion commented Apr 20, 2017

classic

@shykes

This comment has been minimized.

Collaborator

shykes commented Apr 20, 2017

@gaocegege we moved the repository because the first task of the Moby project is to break up the engine monolith into smaller components, and spin them out into separate repos. Once that is complete, Moby will be an integration repository, with the tools necessary to combine upstream components into a complete system.

What we want to do (and clearly we need to communicate this better) is ask for the contributor's help in making this happen. Let's finish breaking up the monolith together - and then let's make Moby the best place to assemble the components in many different ways, so that everyone can do whatever they want with them.

@shykes

This comment has been minimized.

Collaborator

shykes commented Apr 20, 2017

To clarify the "production chain", from upstream to downstream:

upstream components (containerd, linuxkit etc) -> Moby -> Docker CE -> Docker EE.

@Vanuan

This comment has been minimized.

Vanuan commented Apr 21, 2017

@JonasT That would be great. I think it'll be that way eventually.

@Fonger

This comment has been minimized.

Fonger commented Apr 21, 2017

So it means that repository names starting with moby is not allowed either? LOL

@shykes

This comment has been minimized.

Collaborator

shykes commented Apr 21, 2017

@JonasT yes good idea, we will create a repo that can build "Docker CE for Linux".

@shykes

This comment has been minimized.

Collaborator

shykes commented Apr 21, 2017

@omeid you are correct Docker CE continues to exist as you describe. It will expand in scope to fit the equivalent of Docker for Mac, Docker for AWS... which today have more components built-in than "Docker for Linux".

@ravibhure

This comment has been minimized.

ravibhure commented Apr 21, 2017

@shykes How this meant for current docker which is in production or heavily on dev infrastructure ?

@shykes

This comment has been minimized.

Collaborator

shykes commented Apr 21, 2017

@ravibhure current Docker users are not affected, Docker remains exactly the same. You can get the latest Docker CE package for your package on docker.com, the same way as before.

@omeid

This comment has been minimized.

omeid commented Apr 21, 2017

Let me ask it like it is, I depend on docker (the current standard distribution) both as a Go package and as a running daemon for a lot of my projects and my clients. What should I plan ahead? How much work should I expect to put in?

@Wirone

This comment has been minimized.

Wirone commented Apr 21, 2017

Isn't Moby too busy with making music so he won't be able to handle all the building stuff? How this guy will do this single handedly.. All tech people, all the companies will rely on one bald guy with glasses.. I guess at some point he will say "I just cannot contain this" (pun intended).

@apakhomov

This comment has been minimized.

apakhomov commented Apr 21, 2017

Did you try to use google when choosing new name? Now everybody will have to filter a lot of search rubbish about musician, moby dick, moby explorer, mobile applications etc when searching moby-related issues? Awesome

@shykes

This comment has been minimized.

Collaborator

shykes commented Apr 21, 2017

@omeid no work to plan. Nothing will change for you. Just keep installing Docker from the docker.com packages as usual. Same for the Go packages.

@brettdh

This comment has been minimized.

Contributor

brettdh commented Apr 21, 2017

For further clarification on the respective roles of Docker and Moby, please see this link.

@shykes did you mean to include a link there? 🙂 Maybe this one?

@JonET

This comment has been minimized.

JonET commented Apr 21, 2017

@lukemarsden

This comment has been minimized.

Contributor

lukemarsden commented Apr 21, 2017

@brettdh yes that's the right link

@frekele

This comment has been minimized.

frekele commented Apr 22, 2017

As the thread was unfolding with community input and comments from @shykes I began to understand the real purpose of this change. And I truly fully agree that the moby project only brings more advantages to the open source community.

This will bring more Collaboration of the community, and each time we will have a more incredible tool beyond what it already is.

An interesting post about this.
https://thenewstack.io/docker-seeds-two-new-projects-building-containerized-infrastructure/

I believe this image says a lot about docker/moby changes.
88fd7e29-docker-upstream 1

*Thus far, Docker already has a library of more than 80 containerized components, including Swarm, containerd, Docker Build and even LinuxKit. Many third-party components are now hard-wired into Linux distributions. Docker is welcoming more contributions from the community.here

Now with moby we can also have a third-party Moby projects.
With moby, now many components that were previously attached to the docker, will be linked to the moby, thereby generating more feedback, issues, PR of the community.

I think it's going to be this way now.

moby-structure-build

Correct me if I'm wrong. Please. 🙏 😄

@fatherlinux

This comment has been minimized.

fatherlinux commented Apr 22, 2017

@frekele while, I really want to believe this is a good move for the community, there is one critical component missing. In the drawings referenced, there is no mention of the docker command, docker daemon, and Dockerfile. There is no mention of whether they stay in the Moby project/repo or are separated out into another repository.

Either way, if they remain named docker daemon, docker command, and Dockerfile, it prevents any other company from extracting any value from their contributions to these projects because only Docker Inc. retains the rights to use the docker name.

As somebody who has targeted and refined my career over the last 4 years based on Docker/Containers, I can no longer extract value from that participation directly without becoming a Docker EE reseller of some kind. I can't build a company doing consulting on the docker daemon, Dockerfiles because that would be a breach Docker Inc's usage policy.

In fact, nobody can build anything downstream from Moby that includes the docker daemon, command or Dockerfile and talk about it in any sane way. The docker daemon absolutely needs renamed or Docker Inc is the only company that can extract value.

Open source projects like Linux, Apache, KDE, etcd, etc allow ANYBODY to extract downstream value. If the goal is openness, then I ask Docker Inc. to rename the docker daemon to mobyd, the command to moby, and the Dockerfile to Mobyfile. Then, anybody can build downstream value. I personally, am very upset. (opinions here my own, not those of my employer Red Hat).

@shykes

This comment has been minimized.

Collaborator

shykes commented Apr 22, 2017

@frekele that's exactly right! Thank you for the extended drawing, and for taking the time to write this.

@shykes

This comment has been minimized.

Collaborator

shykes commented Apr 22, 2017

@fatherlinux

This comment has been minimized.

fatherlinux commented Apr 22, 2017

@zouyee

This comment has been minimized.

zouyee commented Apr 24, 2017

Moby Dick

@ozbillwang

This comment has been minimized.

Contributor

ozbillwang commented Apr 24, 2017

our model is closer to the Red Hat model: we do everything in the open, in a fast-moving project where everyone can participate. Then we integrate, test and harden downstream into a supported enterprise product.

so docker as Red Hat, moby as centos?

@Vanuan

This comment has been minimized.

Vanuan commented Apr 24, 2017

so docker as Red Hat, moby as centos?

I would think it's more like this:

Docker EE = RHEL
Docker CE = CentOS
Moby project = Fedora project

@fatherlinux

This comment has been minimized.

fatherlinux commented Apr 24, 2017

Docker CE can't be CentOS because CentOS is a downstream build of Red Hat Enterprise Linux. The Red Hat relationship is below, each step is a selection, packaging and/or hardening process of the upstream because they are ALL open source:

Linux & Community Projects-> Fedora -> Red Hat -> CentOS

With Docker it's more like:

Community Projects (Containerd, Dockerd (maybe), Runc) -> Moby -> Docker CE -> Docker EE -> Dead End (because Docker EE is not made of all open source components. Docker EE adds proprietary components, RHEL does not).

@jberkus

This comment has been minimized.

jberkus commented Apr 24, 2017

@fatherlinux although to be fair, years ago, Red Hat used to have proprietary components (I'm talking about 2002 here).

We don't know that Docker EE will have proprietary components; @shykes has said they'll be open-sourcing some of the things which are currently proprietary. Let's give them a chance, eh?

For my part, a complete Moby kit will make it easier to release the version of Docker (or Moby) we want in Fedora. At least, theoretically. Some of us have been after Docker for the last year and more to "break up the monolith", and now they're doing it, so I think at least a bit of cheering them on is in order.

@jberkus

This comment has been minimized.

jberkus commented Apr 24, 2017

BTW, @shykes, we have a bunch of community experts in Fedora-land, happy to discuss the new setup if we can help!

@shykes

This comment has been minimized.

Collaborator

shykes commented Apr 25, 2017

@jberkus I would love that, thank you for offering. I will contact you on your profile email to pick your brains, if that's OK.

@vieux vieux referenced this pull request Apr 27, 2017

Open

[META] Splitting moby and docker #32867

4 of 14 tasks complete

@l2dy l2dy referenced this pull request May 5, 2017

Merged

docker: update to 17.04.0-ce #433

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment