race condition: boot2docker upgrade after docker release but before boot2docker release results in client server version mismatch

[jokeyrhyme]

Just ran boot2docker upgrade after the Docker 1.5.0 release but before the matching boot2docker release. So I was on boot2docker 1.4.1.

The upgrade process updated my docker binary to 1.5.0, but kept the boot2docker parts at 1.4.1, which is as expected.

Unfortunately, I now get the following output from `docker version:

Client version: 1.5.0
Client API version: 1.17
Go version (client): go1.4.1
Git commit (client): a8a31ef
OS/Arch (client): darwin/amd64
FATA[0000] Error response from daemon: client and server don't have same version (client : 1.17, server: 1.16)

And the following error for many docker commands:

time="2015-02-11T09:03:17+11:00" level="fatal" msg="Error: client and server don't have same version (client : 1.17, server: 1.16)"

I noticed that the old docker binary is backed up to ~/.boot2docker, so this is not the end of the world for me or anything. :)

I wonder if it's possible for boot2docker upgrade to check the API versions and automatically rollback if there is a mismatch? Or maybe somehow know that there will be a mismatch in advance? /shrug

Update: It has been well-established that once all the required versions of docker and boot2docker are actually available, that simply running boot2docker upgrade is sufficient to correct any issues. When I first reported this issue, my desire was that boot2docker upgrade should not be able to break anything in the first place. Running it a second time a few days later is known to fix it, but is little consolation if you are prevented from using Docker during this time.

[jokeyrhyme]

Loosely related to #575 .

[nathanleclaire]

@jokeyrhyme Hm, this should not happen. Sorry to hear. It sounds like the new boot2docker ISO has not been downloaded (it should be). Could you please paste the output from when you run the boot2docker upgrade command?

I noticed that the old docker binary is backed up to ~/.boot2docker, so this is not the end of the world for me or anything. :)

I love it when a plan comes together!

[jokeyrhyme]

@nathanleclaire eek. Too late now. boot2docker 1.5.0 is out now, so this exact situation is no longer repeatable. These are the prerequisites in order to reproduce:

• have boot2docker installed with a certain API version
• have a recent release of docker with a new API version
• use boot2docker upgrade immediately after the new docker release, and before the matching boot2docker release is available
• boot2docker upgrade will update the docker binary on the host to the latest docker, but will not be able to update either boot2docker or the boot2docker ISO

[nathanleclaire]

use boot2docker upgrade immediately after the new docker release, and before the matching boot2docker release is available

Ah, so here's our issue. It's a race between the releases - probably should have an additional check around that. Thanks for reporting!

[aravindgan]

hi Nathan,

i am facing the similar issue. So what is the fix? Could you please help me with that?

Regards,
Aravind

[jokeyrhyme]

@aravindgan boot2docker upgrade should backup your Docker binary into ~/.boot2docker, so you can manually restore it from there. Also, now that the latest stable releases for Docker and boot2docker match again (v1.5.0), you should be able to run boot2docker upgrade again to have everything fixed.

[aravindgan]

it does not work,

sudo boot2docker upgrade
Password:
docker is up to date (1.5.0), skipping upgrade...
boot2docker is up to date (v1.5.0), skipping upgrade...
Latest release for boot2docker/boot2docker is v1.5.0
Downloading boot2docker ISO image...
Success: downloaded https://github.com/boot2docker/boot2docker/releases/download/v1.5.0/boot2docker.iso
to /Users/aravind/.boot2docker/boot2docker.iso
Waiting for VM and Docker daemon to start...
.o
Started.
Writing /Users/aravind/.boot2docker/certs/boot2docker-vm/ca.pem
Writing /Users/aravind/.boot2docker/certs/boot2docker-vm/cert.pem
Writing /Users/aravind/.boot2docker/certs/boot2docker-vm/key.pem

To connect the Docker client to the Docker daemon, please set:
export DOCKER_HOST=tcp://192.168.59.103:2376
export DOCKER_CERT_PATH=/Users/aravind/.boot2docker/certs/boot2docker-vm
export DOCKER_TLS_VERIFY=1

bash-3.2$ docker version
Client version: 1.5.0
Client API version: 1.17
Go version (client): go1.4.1
Git commit (client): a8a31ef
OS/Arch (client): darwin/amd64
FATA[0000] Error response from daemon: client and server don't have same version (client : 1.17, server: 1.16)

[aravindgan]

how can i manually downgrade to older version of client? can you help me with the steps?

[nathanleclaire]

@aravindgan You can find a binary named docker-1.4.1 (or whatever version you upgraded from) in your ~/.boot2docker directory which is the old client version if you need it.

However, the issue you're running into shouldn't happen. Could you please share the contents of the file ~/.boot2docker/profile? I'm wondering if you're set up to use an alternative ISO or something else is off in configuration. Looks like the ISO is downloading correctly, so I am suspicious that something else is awry.

Also, I don't think this is the issue at hand, but you shouldn't need sudo on the boot2docker upgrade command- try it without.

Lastly, please ensure that your environment variables are set correctly ($(boot2docker shellinit) should do the trick) before running any Docker commands.

[SvenDowideit]

before you run docker version please run $(boot2docker shellinit) to make sure your env is right for the docker client to find your daemon. (its a long shot tho)

[arulrajnet]

Download the docker windows client from here https://master.dockerproject.com/windows/amd64/docker-1.4.1-dev.exe which has api version 1.16

[jasonferrier]

I just encountered the same error.

I had v1.4.1 installed and working. I just upgraded using the v1.5.0 release and upon running boot2docker.app, I received the error:

FATA[0000] Error response from daemon: client and server don't have same version (client : 1.17, server: 1.16)

Running $ boot2docker upgrade fixed my issue

[Pensu]

Hi,

I am having the same issue. I am running docker on Fedora 20 and I am experimenting on swarm with docker. I am able to configure swarm but can't use it.

[peeyush@localhost swarm]$ docker-machine_linux-amd64 ls
NAME ACTIVE DRIVER STATE URL SWARM
dev virtualbox Running tcp://192.168.99.100:2376
swarm-master virtualbox Running tcp://192.168.99.101:2376 swarm-master (master)
swarm-node-00 virtualbox Running tcp://192.168.99.100:2376 swarm-master
swarm-node-01 * virtualbox Running tcp://192.168.99.102:2376 swarm-master
[peeyush@localhost swarm]$ eval "$(docker-machine_linux-amd64 env swarm-master)"
[peeyush@localhost swarm]$ docker info
FATA[0000] Error response from daemon: client and server don't have same version (client : 1.18, server: 1.17)

[peeyush@localhost swarm]$ docker version
Client version: 1.5.0-dev
Client API version: 1.18
Go version (client): go1.3.3
Git commit (client): 831c796
OS/Arch (client): linux/amd64
FATA[0000] Error response from daemon: client and server don't have same version (client : 1.18, server: 1.17)

And solution? because seems like I can't do boot2docker upgrade!

[jokeyrhyme]

@Pensu are you able to install just the 1.5.0 stable client? The output says you are running 1.5.0-dev.

[Pensu]

@jokeyrhyme I tried to do sudo yum update docker, but it's still 1.4.1, not sure how to get stable 1.5.0 for Fedora 20. Right now I am just building it from git code.

[jokeyrhyme]

@Pensu building from git code is fine, but make sure that you've checked out a stable tag. Don't just build from master (unless you know that works).
.

[Pensu]

@jokeyrhyme okay, that might be the issue, let me switch to the stable branch and try it out again. Thanks :)

[Pensu]

@jokeyrhyme Thanks for the advice and finally I am not getting that error. But I am not able to see my nodes in docker info in swarm master env.

[peeyush@localhost swarm]$ docker-machine_linux-amd64 ls
NAME ACTIVE DRIVER STATE URL SWARM
swarm-master virtualbox Running tcp://192.168.99.108:2376 swarm-master (master)
swarm-node-00 * virtualbox Running tcp://192.168.99.109:2376 swarm-master

[peeyush@localhost swarm_demo]$ eval "$(docker-machine_linux-amd64 env swarm-master)"
[peeyush@localhost swarm_demo]$ docker info
Containers: 2
Images: 8
Storage Driver: aufs
Root Dir: /mnt/sda1/var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 12
Execution Driver: native-0.2
Kernel Version: 3.18.5-tinycore64
Operating System: Boot2Docker 1.5.0 (TCL 5.4); master : a66bce5 - Tue Feb 10 23:31:27 UTC 2015
CPUs: 4
Total Memory: 999.9 MiB
Name: swarm-master
ID: 6KQB:SY5U:BMTS:S4YY:ZBRJ:RKJY:HVED:XSVK:WIHT:JT24:4HOU:J3MO
Debug mode (server): true
Debug mode (client): false
Fds: 28
Goroutines: 29
EventsListeners: 1
Init Path: /usr/local/bin/docker
Docker Root Dir: /mnt/sda1/var/lib/docker
Labels:
provider=virtualbox

It's not showing me the info about the node that I added i.e. swarm-node-00. Is there any other API or something or my setup still has something wrong?

Update: Okay, got it. I also have to use --swarm while loading the environment.

[jokeyrhyme]

@Pensu all I can suggest is double-checking how you've set it up: https://docs.docker.com/machine/#using-docker-machine-with-docker-swarm
It could also be that specific non-stable versions of docker / swarm / machine are required? /shrug

[orcaman]

Same issue. Downloaded and installed boot2docker dmg file just now and encountered this when opening boot2docker:

Error response from daemon: client and server don't have same version (client : 1.17, server: 1.16)

running boot2docker upgrade from Mac terminal (not boot2docker) solved it for me.

[jokeyrhyme]

How the boot2docker maintainers decide to tackle this is their business, but I'm personally happy when boot2docker upgrade fixes things. My concern with this ticket is that boot2docker upgrade actually breaks things in the first place (depending on when you run it).

[orangewise]

I'm using docker-machine and had the same error, docker-machine upgrade fixed the problem.

[wangwenchao]

boot2docker upgrade to update the boot2docker images iso file

[jayasimhan]

Do this:

brew update
brew upgrade docker
brew upgrade boot2docker
boot2docker upgrade

[zq920320]

@jayasimhan I do as your method, and success.Thanks :)

[alexanderilyin]

I just had to stop VM and rerun upgrade and it works.

[behemphi]

I just had a problem where client version was 1.18 and server version was 1.17

FATA[0000] Error response from daemon: client and server don't have same version (client : 1.18, server: 1.17)

Running boot2docker upgrade fixed the issue.

[jokeyrhyme]

It has been well-established that once all the required versions of docker and boot2docker are actually available, that simply running boot2docker upgrade is sufficient to correct any issues. When I first reported this issue, my desire was that boot2docker upgrade should not be able to break anything in the first place. Running it a second time a few days later is known to fix it, but is little consolation if you are prevented from using Docker during this time.

[henvic]

I've just had the same issue after boot2docker upgrade and running again fixed it. I haven't stopped boot2docker before upgrading. Maybe this has something to do with it?

[osswangxining]

If you installed docker directly on native OS instead of VM, you need to restart the docker daemon service to remove this issue, e.g. using the following command in ubuntu: sudo service docker restart.

[kevinSuttle]

FATA[0000] Error response from daemon: client and server don't have same version (client : 1.18, server: 1.17)

I got this today, but running boot2docker upgrade fixed it.

[sethkrasnianski]

👍 Verifying that boot2docker upgrade did in fact fix it.

[thaJeztah]

I wonder if this is still an issue; I think the releases are now better coordinated, for example, Boot2Docker 1.7.0-rc3 was available 6 minutes after Docker 1.7.0-rc3; moby/moby#13528 (comment).

[jokeyrhyme]

@thaJeztah I've noticed this too. Why keep this as a code problem if the deployment process has solved it?

[greysAcademicCode]

This does not appear to me to be solved.

Following the official installation instructions for Windows (just now) results in FATA[0000] Error response from daemon: client and server don't have same version... when trying to run the docker version command as shown in the instructions.

This is for the current stable version of Docker, 1.6.2.

Running boot2docker upgrade then restarting the boot2docker environment fixes it, but why should the installer be bringing a version that immediately needs to be upgraded?

[leifhanack]

Still an issue when docker is released before boot2docker is released. Currently (2015-06-18 7pm GMT) docker 1.7 is released but boot2docker is only available in 1.6.2. Doing a boot2docker upgrade will install docker 1.7 (client: 1.19) but keeps the server in version 1.18.

Is it an alternative to restrict boot2docker upgrade to the same docker client major.minor version, so that in the above case only the latest 1.6.x release will be installed instead of 1.7?

Regards, Leif

[thaJeztah]

@leifhanack are you sure? 1.7 was released an hour ago; https://github.com/boot2docker/boot2docker/releases

[leifhanack]

I have had the luck of upgrading in the hour right between the two releases:)

[thaJeztah]

@leifhanack that's unfortunate, @tianon missed the GitHub notification that 1.7 was merged, resulting in a slight delay, so I pinged him on IRC, apologies for the inconvenience there.

[tianon]

It also takes time to actually build these things -- they're not instantaneous. 😄

[leifhanack]

@thaJeztah @tianon no problem at all. thanks for all your work!

btw since 1.7 I have a certificate issue.

docker version
Client version: 1.7.0
Client API version: 1.19
Go version (client): go1.4.2
Git commit (client): 0baf609
OS/Arch (client): darwin/amd64
An error occurred trying to connect: Get https://192.168.59.103:2376/v1.19/version: x509: certificate is valid for 127.0.0.1, 10.0.2.15, not 192.168.59.103

so far I couldn't find sth related here

thanks a lot, leif

[leifhanack]

when i remove my vm (boot2docker delete) and init it again it works. unfortunately i didn't backed up my vm so that i can't help fixing the problem.

[bayendor]

I can report the same issue as @leifhanack. It is solved by boot2docker delete && boot2docker init, but the changes do not persist if the boot2docker vm is halted for any reason.

[kevinSuttle]

Am I the only one who finds the messages confusing?

❯ brew install codeclimate                                                                              12:39:30 PM
==> Installing codeclimate from codeclimate/homebrew-formulae
==> Downloading https://github.com/codeclimate/codeclimate/archive/v0.0.7.tar.gz
==> Downloading from https://codeload.github.com/codeclimate/codeclimate/tar.gz/v0.0.7
######################################################################## 100.0%
==> env PATH=/usr/local/bin:/usr/local/Library/ENV/4.3:/usr/bin:/bin:/usr/sbin:/sbin PREFIX=/usr/local/Cellar/codecl
bin/check

Notice the version numbers carefully. Those specific versions (1.19, 1.18) are only ever mentioned in the error from the daemon.

Error response from daemon: client and server don't have same version (client : 1.19, server: 1.18)

Now this next one is the kicker. If boot2docker is up to date, then why is it downloading a new image of the same version?

❯ boot2docker upgrade                                                                                   12:39:37 PM
docker is up to date (1.7.0), skipping upgrade...
boot2docker is up to date (v1.7.0), skipping upgrade...
Latest release for github.com/boot2docker/boot2docker is v1.7.0
Downloading boot2docker ISO image...
Success: downloaded https://github.com/boot2docker/boot2docker/releases/download/v1.7.0/boot2docker.iso
    to /Users/kevinsuttle/.boot2docker/boot2docker.iso
...

[jokeyrhyme]

@kevinSuttle I opened #670 a while back RE: unnecessary downloads

[glaucio-melo-movile]

Hi folks,
I'm getting the following error - Error response from daemon: client and server don't have same version (client API version: 1.20, server API version: 1.19). I've read the thread above and it seems this race condition between release is a bit often - Is there any simple way to avoid that? :)

[thaJeztah]

@glaucio-melo-movile it's possible you somehow only updated your client, but not the virtual machine; this doesn't sound like the "race" described here, because docker 1.8.x has been available for a while (and so is boot2docker with docker 1.8.x)

Try updating the VM by running docker-machine upgrade <yourmachinename> or boot2docker upgrade (if you're still using boot2docker-cli)

[jm3]

@kevinSuttle you are definitely not the only one — the entire process is a living nightmare. it's like we resurrected the worst of linux sysadmin circa 1999.