Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Include the docker client binary in images (for gcloud docker commands) #97

Closed
euank opened this Issue Aug 31, 2017 · 7 comments

Comments

Projects
None yet
2 participants
@euank
Copy link
Contributor

commented Aug 31, 2017

The cloud-sdk docker images provide access to the gcloud commands. However, they don't provide a good mechanism for executing the gcloud docker subcommand which is handy for downloading docker images from gcr.io more easily.

I've presented the two options we can recommend users take in order to successfully run the gcloud docker subcommands. I've included the good and bad of each.

Based on the below, I'd like to recommend that the cloud-sdk image, in both the debian and alpine variants, include the latest "stable" docker client binary in the image itself.
I'm happy to do the work required for that myself if you agree that's the right path.
I think the good/bad descriptions below adequately explain why I think this is a good idea.

1. Bindmount in the host's docker client

Right now, the accepted way to do this, as best I can tell, is bindmount in the host's docker client binary. A full command to do a gcr pull looks like the following:

$ docker run -v $HOME/.config/gcloud:/root/.config/gcloud \
                     -v /var/run/docker.sock:/var/run/docker.sock \
                     -v /usr/bin/docker:/usr/bin/docker \
                     google/cloud-sdk  docker -- pull gcr.io/google_containers/pause

The good

  • This works on some distros!
  • The daemon and client will have matching versions
  • Small cloud-sdk image size

The bad

  • Doesn't work with the alpine cloud-sdk on gibc distros with dynamically linked docker clients (aka almost all distros).
  • Doesn't work on Fedora 26, even with the debian based cloud-sdk image
    docker: error while loading shared libraries: libltdl.so.7: cannot open shared object file: No such file or directory
    
  • Requires the user to do an extra bindmount in addition to the docker socket

2. Include a client binary for the latest 'stable' docker version

The image itself could provide the client portion of the docker package.

This will result in a user running a command like the following:

$ docker run -v $HOME/.config/gcloud:/root/.config/gcloud \
                     -v /var/run/docker.sock:/var/run/docker.sock \
                     google/cloud-sdk  docker -- pull gcr.io/google_containers/pause

The good

  • Will work regardless of the host distro (musl / glibc mixing and matching fine)
  • Easier command for the user to type
  • If the user runs the command from option 1, it will still work just as well or badly as before (e.g. this option doesn't prevent users from using 1 if they want the good from it)

The bad

  • Increases image size by 20MB or so
  • The client and server versions will generally not be identical
    Note, however, that each docker daemon supports multiple 'api versions', and the docker client makes an attempt to negotiate the correct api version. Given the only operation here is 'pull', this difference should be fairly harmless.
@ahmetb

This comment has been minimized.

Copy link
Member

commented Aug 31, 2017

Do you know if docker offers client-only binary distributions (e.g. it does for Mac and Windows) via apt? The reason a Fedora binary doesn't work on a Debian container is mostly because of the docker-daemon code paths in the /usr/bin/docker.

If you can find a client-only distribution of the binary for Linux, I'm guessing you can just download it to the container via curl/wget and use it, or if there's an apt package, you can just extend this image by using it as a base image (something we suggest people to do all the time) and just RUN apt-get install ....

I appreciate if you can take a look if there are client-only distributions of docker. Otherwise, it's not easily doable for us to package the binary either.

@euank

This comment has been minimized.

Copy link
Contributor Author

commented Aug 31, 2017

The reason a Fedora binary doesn't work on a Debian container is mostly because of the docker-daemon code paths in the /usr/bin/docker.

@ahmetb That's not the case. Docker has long-since split out docker from dockerd. For example, on Fedora the difference between those two binaries is:

 $ rpm --query --file /usr/bin/docker 
docker-engine-17.05.0.ce-1.fc25.x86_64
$ ldd /usr/bin/docker{,d}
/usr/bin/docker:
	linux-vdso.so.1 (0x00007ffe79bad000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f0730b64000)
	libltdl.so.7 => /lib64/libltdl.so.7 (0x00007f073095a000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f0730589000)
	/lib64/ld-linux-x86-64.so.2 (0x000055b4ccc98000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f0730385000)
/usr/bin/dockerd:
	linux-vdso.so.1 (0x00007ffddf57f000)
	libsystemd.so.0 => /lib64/libsystemd.so.0 (0x00007f389a144000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f3899f25000)
	libseccomp.so.2 => /lib64/libseccomp.so.2 (0x00007f3899ce3000)
	libdevmapper.so.1.02 => /lib64/libdevmapper.so.1.02 (0x00007f3899a8f000)
	libc.so.6 => /lib64/libc.so.6 (0x00007f38996be000)
	libresolv.so.2 => /lib64/libresolv.so.2 (0x00007f38994a3000)
	libselinux.so.1 => /lib64/libselinux.so.1 (0x00007f389927b000)
	libcap.so.2 => /lib64/libcap.so.2 (0x00007f3899076000)
	librt.so.1 => /lib64/librt.so.1 (0x00007f3898e6e000)
	liblzma.so.5 => /lib64/liblzma.so.5 (0x00007f3898c48000)
	liblz4.so.1 => /lib64/liblz4.so.1 (0x00007f3898a33000)
	libgcrypt.so.20 => /lib64/libgcrypt.so.20 (0x00007f3898723000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00007f389851f000)
	libgpg-error.so.0 => /lib64/libgpg-error.so.0 (0x00007f389830b000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f38980f4000)
	/lib64/ld-linux-x86-64.so.2 (0x000055cf5f8c2000)
	libsepol.so.1 => /lib64/libsepol.so.1 (0x00007f3897e5d000)
	libudev.so.1 => /lib64/libudev.so.1 (0x00007f3897e3a000)
	libm.so.6 => /lib64/libm.so.6 (0x00007f3897b22000)
	libpcre.so.1 => /lib64/libpcre.so.1 (0x00007f38978af000)

As you can clearly see, they've already made a careful attempt to avoid linking to excessive binaries for the client.

I appreciate if you can take a look if there are client-only distributions of docker. Otherwise, it's not easily doable for us to package the binary either.

As you say, we can download the static binaries docker provides. I'd be happy to provide a patch that extracts only the client from that, does hashsum verification (since they no longer provide gpg signed binaries 😢), and cleans up appropriately.

That's what i had in mind when I offered to tackle this.

If we don't wish to trust their distribution, I'd also be happy to update the dockerfile to be a "multi-stage-build" dockerfile (dockerhub now supports them) to compile a docker client binary for a given version and include it.

I think either is an okay approach and I'd be happy to go through the work for either if they'd be welcome.
If both sound good, I'd mildly prefer doing the latter.

@ahmetb

This comment has been minimized.

Copy link
Member

commented Aug 31, 2017

As you say, we can download the static binaries docker provides. I'd be happy to provide a patch that extracts only the client from that, does hashsum verification (since they no longer provide gpg signed binaries 😢), and cleans up appropriately.

I'm not sure if we need to ship docker client in our images. I suspect not many users of gcloud might be needing docker in the same image. It'd be great to hear if you can get it to work though.

docker: error while loading shared libraries: libltdl.so.7: cannot open shared object file: No such file or directory

I suppose the problem simply might be missing libltdl on our base image. We can simply include this library in our image to enable you to install docker, but then you can probably do it yourself, too.

At this point I'm not sure supporting docker-in-docker is one of our top priorities or a direction we'd like to take right now.

euank added a commit to euank/cloud-sdk-docker that referenced this issue Aug 31, 2017

Add docker client to images
The `gcloud docker` subcommand relies on executing a docker client.
This change allows the user to bindmount in the host's docker socket,
and then interact with it using the `gcloud docker` subcommand.

This will only work if the user's host dockerd is compatible with this
docker client, but this is true for all modern versions of docker.

See additional discussion in GoogleCloudPlatform#97
for why bindmounting in the client's docker binary is non-ideal.

This change is the equivilant of simply downloading their official linux
release tarball, and picking the client binary out of it.

This is simply a more readable, and possibly more future-proof way, of
accomplishing that.
@euank

This comment has been minimized.

Copy link
Contributor Author

commented Aug 31, 2017

@ahmetb

I suppose the problem simply might be missing libltdl on our base image.

That's missing the point I'm trying to state. The problem is that you can't recommend users to bindmount in a docker binary that the host's package manager installed because you can't know its library dependencies. You can play whack-a-mole with each distro if you wish, but to me it's clearly the wrong solution.

At this point I'm not sure supporting docker-in-docker is one of our top priorities or a direction we'd like to take right now.

What most people mean by docker in docker is running the docker daemon.
My suggestions, to be clear, are entirely about using the docker client to communicate with the host's daemon.
I agree that dealing with dockerd in docker isn't something worth doing here.

Given that gcloud provides utility functions to pull from gcr, and those commands only work with a docker client present, I do think supporting that command is well within the scope of this project.

It'd be great to hear if you can get it to work though.

Here's a trivial example of how to get it working. If you trust docker-inc, then this works just fine. If you'd rather not trust them, I'd be happy to compile from source instead.

https://github.com/GoogleCloudPlatform/cloud-sdk-docker/compare/master...euank:add-docker-client?expand=1

@ahmetb

This comment has been minimized.

Copy link
Member

commented Aug 31, 2017

I meant even having docker-client shipped with this image might not be something that's necessary.

Given that gcloud provides utility functions to pull from gcr, and those commands only work with a docker client present, I do think supporting that command is well within the scope of this project.

While I certainly understand the use case, I am not sure how many people are using this image to bind mount the host docker.sock and pull/push images to GCR. I don't think we currently support all gcloud commands in the containerized image.

Here's a trivial example of how to get it working.

I am completely fine documenting the multi-step approach you showed above in README.md to copy image from official docker-client image that has the static binary. Let me ask around internally and see what others think about this.

@euank

This comment has been minimized.

Copy link
Contributor Author

commented Aug 31, 2017

Sure, let me know whether you'd like a PR with that change or not.

@ahmetb

This comment has been minimized.

Copy link
Member

commented Sep 5, 2017

It looks like we're okay with adding a docker CLI to the image.

  • We can use multi-stage builds to bring the binary from the official image and copy it to /usr/local/bin/docker.
  • I think we should do this only for images other than the :slim. By definition, the slim image has bare minimum to get gcloud main commands working (and gcloud docker is probably a secondary use case).
  • We should also look at configuring docker-credential-gcr out-of-the-box so that the users do not need gcloud docker -a command to get docker push/docker pull working out of the box. (If we're not installing docker-credential-gcr already for non-slim images today, we should.)

euank added a commit to euank/cloud-sdk-docker that referenced this issue Sep 25, 2017

Add docker client to 'thick' image
The `gcloud docker` subcommand relies on executing a docker client.
This change allows the user to bindmount in the host's docker socket,
and then interact with it using the `gcloud docker` subcommand.

This will only work if the user's host dockerd is compatible with this
docker client, but this is true for all modern versions of docker.

See additional discussion in GoogleCloudPlatform#97
for why bindmounting in the client's docker binary is non-ideal.

The `docker` image was used to source the docker client binary as a
matter of convenience.

The change is intentionally only applied to the 'alpine' and 'debian'
variants of the image, not the '-slim' one. This is to keep the 'slim'
image slim.

euank added a commit to euank/cloud-sdk-docker that referenced this issue Oct 3, 2017

Add docker client to 'thick' image
The `gcloud docker` subcommand relies on executing a docker client.
This change allows the user to bindmount in the host's docker socket,
and then interact with it using the `gcloud docker` subcommand.

This will only work if the user's host dockerd is compatible with this
docker client, but this is true for all modern versions of docker.

See additional discussion in GoogleCloudPlatform#97
for why bindmounting in the client's docker binary is non-ideal.

The `docker` image was used to source the docker client binary as a
matter of convenience.

The change is intentionally only applied to the 'debian' variant of the
image, not the 'alpine' nor '-slim' ones. This is to keep the 'slim'
image slim.

euank added a commit to euank/cloud-sdk-docker that referenced this issue Oct 3, 2017

Add docker client to 'thick' image
The `gcloud docker` subcommand relies on executing a docker client.
This change allows the user to bindmount in the host's docker socket,
and then interact with it using the `gcloud docker` subcommand.

This will only work if the user's host dockerd is compatible with this
docker client, but this is true for all modern versions of docker.

See additional discussion in GoogleCloudPlatform#97
for why bindmounting in the client's docker binary is non-ideal.

The `docker` image was used to source the docker client binary as a
matter of convenience.

The change is intentionally only applied to the 'debian' variant of the
image, not the 'alpine' nor '-slim' ones. This is to keep the 'slim'
image slim.

@ahmetb ahmetb closed this in #101 Oct 23, 2017

ahmetb added a commit that referenced this issue Oct 23, 2017

Add docker client to 'thick' image (#101)
* Add docker client to 'thick' image

The `gcloud docker` subcommand relies on executing a docker client.
This change allows the user to bindmount in the host's docker socket,
and then interact with it using the `gcloud docker` subcommand.

This will only work if the user's host dockerd is compatible with this
docker client, but this is true for all modern versions of docker.

See additional discussion in #97
for why bindmounting in the client's docker binary is non-ideal.

The `docker` image was used to source the docker client binary as a
matter of convenience.

The change is intentionally only applied to the 'debian' variant of the
image, not the 'alpine' nor '-slim' ones. This is to keep the 'slim'
image slim.

* Update docker client to 17.09

17.09 is now the current docker stable release.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.