Proposal: Override ApiVersion in client

[ahmetb]

Problem

Sometimes (or frequently) users have trouble talking to the docker engine with their docker client because they accidentally update their docker client and the server now has an older version.

It's too hard to get this right and that this usually happens by one off (e.g. client:1.18 server:1.17) and there is not major differences between these versions. This especially happens to the people who are compiling docker from master and talking to a Linux daemon in boot2docker or some cloud VM.

This happens to me a few times a day as I'm finding myself upgrading to latest docker client version and all my servers are old or CoreOS is totally lagging two versions behind or something.

Proposal

I propose a power user environment variable DOCKER_APIVERSION used in the client to override the constant in api/common.go when set:

$ ./docker info
FATA[0000] Error: client and server don't have same version (client : 1.18, server: 1.17)

$ DOCKER_APIVERSION=1.17 ./docker info
Containers: 0
Images: 0
[...]

This can easily help people to try out latest client binaries downloaded from master.dockerproject.com which has the latest bug fixes against their existing docker host deployed somewhere on the cloud.

Considerations

Since this enables the way of undefined behavior and feature set mismatch between the client and the engine this must be thoroughly documented that it is dangerous area. (and probably should exist only in development environment documentation instead of user guide).

Users will be seeing feature mismatches most of the time in the cutting-edge new features added to cli but missing from the docker host.

[phemmer]

See #7746 which has discussion on this subject.

[DieterReuter]

👍

[tknerr]

👍 would make it easier to use the latest docker client with boot2docker definitely.

Btw: is there some SemVer for docker protocol versions? Any notion of forwards or backwards compatibility? Or will it just be luck in the end ;-)

[ahmetb]

@tknerr current status is: clients (API version: x) can talk to engines (API version: y) only if x ≤ y.

[tknerr]

@ahmetalpbalkan ok understand, so that's why there should probably be a big DANGER sign when using DOCKER_APIVERSION ;-)

Is there any notion of major.minor or does it just look like so? I.e. 1.x clients can talk to 1.y engines if x ≤ y, but never to 2.z engines?

[ahmetb]

@tknerr Well, I assume 2.x would be a breaking change, that's why we have API v1.19 now. :) But I'm not sure if there's any promises around this.

[tknerr]

This is what I get with latest docker client 1.5.0-dev, build 7ecf4e5 (1.19) + boot2docker 1.5.0 (1.17):

• boot2docker init
• boot2docker up
• for /f "tokens=2" %i in ('boot2docker shellinit') do set %i (thanks @StefanScherer)
• sed -ibak -b docker.exe -e 's/1.19/1.17/g'

Then on docker ps, docker images, etc... I get this:

C:\Repos\_github\bills-kitchen\target\build>docker ps
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x1 addr=0x0 pc=0x5468b3]

goroutine 8 [running]:
math/big.(*Int).Mod(0xc0820b5340, 0xc0820b5340, 0x0, 0xc0820b5340)
        /usr/local/go/src/math/big/int.go:251 +0x233
crypto/elliptic.(*CurveParams).IsOnCurve(0xc0820fff20, 0xc0820b5300, 0xc0820b5320, 0x137)
        /usr/local/go/src/crypto/elliptic/elliptic.go:57 +0x73
crypto/elliptic.p256Curve.IsOnCurve(0xc0820fff20, 0xc0820b5300, 0xc0820b5320, 0x41)
        <autogenerated>:16 +0x59
crypto/tls.(*ecdheKeyAgreement).processServerKeyExchange(0xc082087fc0, 0xc0820b1560, 0xc082074f00, 0xc082071710, 0xc082092900, 0xc0820ffdd0, 0x0, 0x0)
        /usr/local/go/src/crypto/tls/key_agreement.go:328 +0x52b
crypto/tls.(*clientHandshakeState).doFullHandshake(0xc0820717a0, 0x0, 0x0)
        /usr/local/go/src/crypto/tls/handshake_client.go:306 +0xc1a
crypto/tls.(*Conn).clientHandshake(0xc0820a42c0, 0x0, 0x0)
        /usr/local/go/src/crypto/tls/handshake_client.go:197 +0x1988
crypto/tls.(*Conn).Handshake(0xc0820a42c0, 0x0, 0x0)
        /usr/local/go/src/crypto/tls/conn.go:977 +0xf8
net/http.func┬À022()
        /usr/local/go/src/net/http/transport.go:637 +0x43
created by net/http.(*Transport).dialConn
        /usr/local/go/src/net/http/transport.go:642 +0x893

goroutine 1 [select]:
net/http.(*Transport).getConn(0xc082071560, 0xc082026820, 0x0, 0x8a3d90, 0x5, 0xc08209c606, 0x13, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/http/transport.go:525 +0x60f
net/http.(*Transport).RoundTrip(0xc082071560, 0xc082026820, 0xc082087c00, 0x0, 0x0)
        /usr/local/go/src/net/http/transport.go:228 +0x4db
net/http.send(0xc082026820, 0x1a2330, 0xc082071560, 0x31, 0x0, 0x0)
        /usr/local/go/src/net/http/client.go:219 +0x503
net/http.(*Client).send(0xc0820ff860, 0xc082026820, 0x31, 0x0, 0x0)
        /usr/local/go/src/net/http/client.go:142 +0x162
net/http.(*Client).doFollowingRedirects(0xc0820ff860, 0xc082026820, 0x985258, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/http/client.go:367 +0xb2c
net/http.(*Client).Do(0xc0820ff860, 0xc082026820, 0xc082071560, 0x0, 0x0)
        /usr/local/go/src/net/http/client.go:174 +0xab
github.com/docker/docker/api/client.(*DockerCli).clientRequest(0xc0820a1680, 0x899b50, 0x3, 0x8d37d0, 0x11, 0x1a2c30, 0xc082013650, 0x0, 0x0, 0x0, ...)
        /go/src/github.com/docker/docker/api/client/utils.go:82 +0x7ad
github.com/docker/docker/api/client.(*DockerCli).call(0xc0820a1680, 0x899b50, 0x3, 0x8d37d0, 0x11, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /go/src/github.com/docker/docker/api/client/utils.go:170 +0x26c
github.com/docker/docker/api/client.(*DockerCli).CmdPs(0xc0820a1680, 0xc082008850, 0x0, 0x0, 0x0, 0x0)
        /go/src/github.com/docker/docker/api/client/ps.go:89 +0x12d8
reflect.callMethod(0xc0820ff680, 0xc08210fbe8)
        /usr/local/go/src/reflect/value.go:605 +0x180
reflect.methodValueCall(0xc082008850, 0x0, 0x0, 0x1, 0xc0820ff680, 0xc0820a1701, 0xc082071560, 0x5, 0xc082044001, 0x195c08, ...)
        /usr/local/go/src/reflect/asm_amd64.s:29 +0x3d
github.com/docker/docker/api/client.(*DockerCli).Cmd(0xc0820a1680, 0xc082008850, 0x1, 0x1, 0x0, 0x0)
        /go/src/github.com/docker/docker/api/client/cli.go:81 +0x385
main.main()
        /go/src/github.com/docker/docker/docker/docker.go:138 +0xb90

goroutine 5 [syscall]:
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:21 +0x26
created by os/signal.init┬À1
        /usr/local/go/src/os/signal/signal_unix.go:27 +0x3c

goroutine 6 [chan receive]:
net/http.(*Transport).dialConn(0xc082071560, 0x0, 0x8a3d90, 0x5, 0xc08209c606, 0x13, 0x0, 0x0, 0x0)
        /usr/local/go/src/net/http/transport.go:643 +0x8da
net/http.func┬À019()
        /usr/local/go/src/net/http/transport.go:520 +0x49
created by net/http.(*Transport).getConn
        /usr/local/go/src/net/http/transport.go:522 +0x33c

Not sure if this is a protocol incompatibility here or something gone wrong with sed -ibak -b docker.exe -e 's/1.19/1.17/g'...

[tknerr]

If the latter sed is the issue this would be another big 👍 for using an env var

[ahmetb]

@tknerr you should almost never edit a binary like that. :) just edit api/common.go and recompile.

[tknerr]

@ahmetalpbalkan 100% agree. Just heard some success from @StefanScherer with that and thought it would be faster to try... ;-)

[tknerr]

So I need docker to build docker? That's consequent...

[ahmetb]

@tknerr officially, yes.

Unofficially, if you want to build just the client binary, you just need the Go compiler ( see this guide –you need to slightly change for OS X/Linux obviously). That's the solution I recommend to my colleagues to get around this problem.

[tknerr]

thanks @ahmetalpbalkan!

[tknerr]

@ahmetalpbalkan worked like a charm, thanks!

Concerning the name/value of the env var, do you want the user to give a specific version?

Or should it rather be something like DOCKER_VERIFY_PROTOCOL_VERSION=0?

[tknerr]

...or akin to DOCKER_TLS_VERIFY sth like DOCKER_PROTOCOL_VERIFY?

[ahmetb]

@tknerr verification is not done on client side, it's done on server side (e.g. client calls /v1.19/images). Alternatively we can directly call /images (this also works) but I think DOCKER_API_VERSION is better in this case since it also allows you to actually test the API version sent (b/c some operations actually do different things based on the API version e.g. version ≥ 1.15 {do differently}).

[StefanScherer]

I've just tested the latest docker.exe with protocol v1.19. And yes, I get the same golang errors as above. Patching the docker.exe with sed may work. The correct sed command has a backslash: sed -ibak -b docker.exe -e 's/1\.19/1.17/g'. But you totally loose warranty by doing this.

The better way at the moment without compiling and patching docker.exe and just have a test environment for playing with docker-machine.exe and docker.exe is to update the Linux docker binary in the boot2docker VirtualBox VM.

Start the dev machine, check its IP address and SSH into the VM as described in https://github.com/boot2docker/boot2docker#ssh-into-vm

docker-machine start dev
docker-machine env dev
ssh docker@192.168.99.100
curl -o docker https://master.dockerproject.com/linux/amd64/docker
chmod +x docker
sudo mv docker /usr/local/bin/docker
sudo /etc/init.d/docker restart
exit

I've tested it on my Mac with this little helper VM ( https://github.com/StefanScherer/docker-windows-box ) I've built today

C:\Users\vagrant>docker info
Containers: 0
Images: 0
Storage Driver: aufs
 Root Dir: /mnt/sda1/var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 0
 Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.18.5-tinycore64
Operating System: Boot2Docker 1.5.0 (TCL 5.4); master : a66bce5 - Tue Feb 10 23:31:27 UTC 2015
CPUs: 2
Total Memory: 1000 MiB
Name: dev
ID: 4Q6C:PM5C:W4MT:IEIL:FJRU:TFX5:PJ4V:2SRV:DIHJ:7TGU:QZI6:NXTV
Debug mode (server): true
Debug mode (client): false
File Descriptors: 11
Goroutines: 17
System Time: Wed Apr 15 06:19:12 UTC 2015
EventsListeners: 0
Init Path: /usr/local/bin/docker
Docker Root Dir: /mnt/sda1/var/lib/docker
Labels:
 provider=virtualbox

C:\Users\vagrant>docker version
Client version: 1.5.0-dev
Client API version: 1.19
Go version (client): go1.4.2
Git commit (client): da697db
OS/Arch (client): windows/amd64
Server version: 1.5.0-dev
Server API version: 1.19
Go version (server): go1.4.2
Git commit (server): da697db
OS/Arch (server): linux/amd64

If docker 1.6 is realease there will be an updated boot2docker ISO as well.
So setting the protocol version with an environment is still useful. 👍

[tknerr]

@ahmetalpbalkan makes sense, thanks for explaining

Btw: I'm coming quite close with the help of your guide, but what makes some problems is that docker -v does not work properly:

C:\Repos\_github\bills-kitchen\target\build\repo\fundamentals-with-tests\chef-repo\cookbooks\motd>docker -v
Docker version , build

Running docker version works though:

C:\Repos\_github\bills-kitchen\target\build\repo\fundamentals-with-tests\chef-repo\cookbooks\motd>docker version
Client API version: 1.17
Go version (client): go1.4.2
OS/Arch (client): windows/amd64
Server version: 1.5.0
Server API version: 1.17
Go version (server): go1.4.1
Git commit (server): a8a31ef
OS/Arch (server): linux/amd64

Is there anyting else I need to do other than go build to get this working?

[ahmetb]

@tknerr that's expected when you're building outside of a container (unofficial build).

[ahmetb]

Also please let's keep "how can I get this working" questions out of this issue. This is a proposal for an implementation and we should discuss if we need that or not. 😄

[tknerr]

@ahmetalpbalkan yep you are right, and thanks for your help!

So as already stated I'm totally 👍 fo the env var.

[thaJeztah]

Perhaps the client should automatically switch back to the older API if it's detected (and, possibly a flag/env-var specified). Thinking of cases where a single client is used to manage multiple hosts. Some hosts are up-to-date, others use an older daemon.

Having to change the DOCKER_APIVERSION env-var could become annoying in that case; perhaps (e.g.) DOCKER_API_MINVERSION=1.17 and automatically fall back if the daemon meets that minimum version?

[ahmetb]

@thaJeztah having that sort of logic is probably 100x harder to code and maintain. Users should almost never know about this environment variable, this is for the power users and is just an override. Please refer back to the proposal above.

[arun-gupta]

Any ETA of when this will be available?

[igoratencompass]

+1 for the ETA question

[dnephin]

We recently added COMPOSE_API_VERSION to docker-compose (docker/compose#1780) which does the same thing. It's been really useful.

[duglin]

#dibs

[duglin]

here ya go: #15964

[ahmetb]

lol didn't realize I should've just done it, I was waiting for some approval I guess. @duglin thanks for taking this up.

[thaJeztah]

@ahmetalpbalkan PRs usually work best (if you're willing to take the risk of it being rejected)...the maintainers are keen on getting the number of open PRs down 😉 😇 🙊

[tiborvass]

@ahmetalpbalkan @duglin It feels like the solution would be to not have that stupid restriction and be smarter about API versioning. In fact I believe @calavera is working on the API (feel free to chime in).

Now for the immediate pain, I would NOT recommend supporting this in any way, except for docker devs, so I would be fine with an unsupported and undocumented envvar, and only an envvar. That way we can break it whenever we want.

[duglin]

@tiborvass not sure how any of the API versioning stuff would fix/help here since we're talking about allowing the user to set the version string to something different than what the CLI wants to use.

re: docs - I'm not going to demand that it be in the docs, but I'm not a fan of hiding stuff - having an advanced section is ok though. Hiding stuff got msft in trouble many moons ago :-)

[icecrime]

I understand the use case in a dev environment, but I'm afraid people will start using this for bad reasons. Instead of overriding the ApiVersion, can we make it some sort of "dev mode" that bypasses the check entirely?

Or as @crosbymichael is suggested: make this dependent on --debug?

[duglin]

I think @ahmetalpbalkan was looking for more than just turning off the check on the daemon but to also have the CLI change the URL it used. But I'll defer to @ahmetalpbalkan on this one.

[ahmetb]

Turning off the check on the daemon is way harder... For instance docker-swarm didn't have any release in the past 2 months and it still exposes the ancient API Version 1.16 which makes it impossible to use with new clients.

Also any time I need to change some daemon setting I need to ssh, update setting and restart the service (therefore restart all containers)... Seems a lot tedious/dangerous than adding it to the client.

[TomasTomecek]

If this can work in docker-py, how come it can't in official docker client?