Building Docker should not depend on having a Docker binary

[aecolley]

It's too complicated to build Docker without using a prebuilt Docker binary and a prebuilt base image. Can an open source project really be open source if you can't build it from source?

I created a patch that introduces a "bootstrap" bundle, which builds docker and a base image from scratch. It uses a shell script as a docker emulator which uses lxc-start to containerize the build without using docker.

The patch can be seen or downloaded at https://bitbucket.org/aecolley/redeb-docker/src/tip/docker.patch?at=default -- I'll create a pull request if you think it's worth pulling.

[shykes]

Hi @aecolley, wow, you put a lot of effort into this! Respectfully, I'm not sure I understand what problem you're trying to solve. Looks like going to a lot of trouble for nothing.

Can an open source project really be open source if you can't build it from source?

You can build it from source. Like every other open-source project on the planet, it has build dependencies: executable programs which are necessary to produce a build of docker. What's so unusual about that?

If you think that having binary build dependencies is not acceptable for open-source projects, you should know that the number of open-source projects which pass your quality bar is zero.

Your script does not solve the problem, either. It introduces build dependencies of its own, including of course lxc and debootstrap (which itself adds a dependency on the hosted ubuntu package infrastructure... at every build. nice.)

So, to apply your own logic: it's too complicated to run your bootstrap script without using a prebuilt lxc and debootstrap installation, and having a working internet connection to the live ubuntu servers. Can an open-source project really be open-source if you can't build it from source? Please provide me with the sources of lxc, debootstrap, and the entire ubuntu bootstrap infrastructure, and make sure you bootstrap those too before claiming that your script is open-source. Of course you might need to use gcc, make and autotools for that. Make sure to bootstrap those too.

[shykes]

Also note that, although the official build environment that we recommend uses docker (because it's actually a useful build tool, and it seems unfair for us to not benefit from it when every other developer out there can!), it's of course possible to build docker without it. We just don't recommend it unless you have a good reason, for example if you're a distro packager and are stuck with a rigid set of build tools. We take the needs of distro packagers seriously, and spend a lot of time helping them do this. Most of the credit for this work goes to @tianon.

So if you want to keep using your crazy "fake docker" just for the satisfaction of not using docker to build docker, feel free to ping @tianon for help on using make.sh as part of that. But we're definitely not merging it as an official build tool - it would bring absolutely zero value.

[aecolley]

@shykes I distinguish between open-source projects which depend on other binaries and open-source projects which depend on themselves. A build process which depends on its own output presents a problem: you can't know what you're running, even if you have to care for security reasons.

As for lxc and debootstrap: docker itself already depends on lxc-start at runtime, and on ubuntu (and therefore debootstrap) at build time.

So you don't want the patch. That's fine.

[tianon]

Indeed, @shykes is exactly right: as part of my normal flow I build Docker without Docker many, many times every single day, and I've helped quite a few packagers get our build script working in their packaging environments.

The only reason it isn't "easy" is because Docker has somewhat interesting compilation needs, especially with regards to the binary we inject inside the container, which has to be tolerant of all kinds of different container environments (and thus is most easy to use if it's static).

If you're still interested in compiling Docker without Docker, give ./hack/make.sh dynbinary a whirl after setting up a proper GOPATH (http://golang.org/doc/code.html#Workspaces). As long as you've got the dynamic libraries and header files necessary, that should work like a champ and give you two binaries you can use together.

[shykes]

@aecolley I might be missing something, but why is it more of a security risk to depend on a pre-existing build of docker than on a pre-existing build of other tools, ie debootstrap, lxc-start and bash, which themselves had binary dependencies of their own? Isn't it fundamentally the same problem, that is to say: if you follow the build dependencies all the way down, there's always a "black box" which you cannot link back to source?

[potatoker]

@tianon ,hello,I want to install docker on my android device.But the binary package provides by docker Doc didn't work on my device,Could you please tell me about more about how to compile docker without docker on my android device?Thanks

[isilanes]

@aecolley I might be missing something, but why is it more of a security risk to depend on a pre-existing build of docker than on a pre-existing build of other tools, ie debootstrap, lxc-start and bash, which themselves had binary dependencies of their own?

Because I might trust those other tools (which were built from binaries other than themselves), whereas it is docker itself the package I do not trust, so its binaries can't be trusted either. Think of it as you giving me a product (the code) and the ruler (binary) to measure it's length (compile it). You are seriously saying that using my own ruler (debootstrap, lxc-start, bash) is as "unsafe" as using yours (docker binary), just because I bought my ruler from a third party, just as you are a third party?

Also, guess when it's most useful being able to compile some code from source? That's right, when precompiled binaries are NOT available for the platform you're on. What good is resorting to compiling from source (because docker binaries are not available for me), if to do so I need the binary installed in the first place? That's nuts.

[thaJeztah]

@isilanes please see https://github.com/docker/docker/blob/master/project/PACKAGERS.md for some information about packaging for a platform that's not yet supported