lxc-docker to docker-engine upgrade on 1.10 is causing docker to hang on trusty

[SamSaffron]

Upgrading the docker from 1.0.9 in trusty at deb https://get.docker.io/ubuntu docker main to 1.10 in deb https://apt.dockerproject.org/repo ubuntu-trusty main is hanging for me.

This seems to happen quite consistently on 3 upgrades I did today.

In 1/3 installs a reboot made docker start working.

In 2/3 I had to uninstall, rm -fr /var/lib/docker and then re-install.

[GordonTheTurtle]

If you are reporting a new issue, make sure that we do not have any duplicates already open. You can ensure this by searching the issue list for this repository. If there is a duplicate, please close your issue and add a comment to the existing issue instead.

If you suspect your issue is a bug, please edit your issue description to include the BUG REPORT INFORMATION shown below. If you fail to provide this information within 7 days, we cannot debug your issue and will close it. We will, however, reopen it if you later provide the information.

For more information about reporting issues, see CONTRIBUTING.md.

You don't have to include this information if this is a feature request

(This is an automated, informational response)

---

BUG REPORT INFORMATION

Use the commands below to provide key information from your environment:

docker version:
docker info:

Provide additional environment details (AWS, VirtualBox, physical, etc.):

List the steps to reproduce the issue:
1.
2.
3.

Describe the results you received:

Describe the results you expected:

Provide additional info you think is important:

----------END REPORT ---------

#ENEEDMOREINFO

[jeremyVignelles]

EDIT: Please dont pay attention to this message, I didn't read the docker changelog : https://github.com/docker/docker/releases/tag/v1.10.0

Similar issue here on linux mint 17.1 with deb https://apt.dockerproject.org/repo ubuntu-trusty main :

After aptitude upgrade that installed a new version of docker-engine (1.10.0-0trusty), the docker commands stops working (freeze), and iotop shows high disk read usage ( 100% of disk time) for docker daemon, making my computer quite unusable.
My RAM level is normal.

Previous versions (1.9.1-0~trusty I think), worked just fine, and I didn't notice any freeze.

I tried a service docker stop, which stopped the service correctly and released my hard drive, but service docker start restarts the IO traffic.

Tried a restart without luck, but a downgrade to 1.9.1-0~trusty worked.

[rqclima]

I have the same problem. On Linux Mint 17.2, after I updated docker today, CPU usage is very, very high. Restart didn't work, I stopped the service and going to downgrade...

[SamSaffron]

Feel free to close, I guess this is just how it goes, if you upgrade 1.9 to
1.10 on digital ocean and have lots of images expect a reasonably long
outage

On Friday, 5 February 2016, rqclima notifications@github.com wrote:

I have the same problem. On Linux Mint 17.2, after I updated docker today,
CPU usage is very, very high. Restart didn't work, I stopped the service
and going to downgrade...

—
Reply to this email directly or view it on GitHub
#20021 (comment).

[vdemeester]

@SamSaffron @rqclima @jeremyVignelles There is a reason for the high io/cpu, it's documented there : https://docs.docker.com/engine/migration/.

Starting from version 1.10 of Docker Engine, we completely change the way image data is addressed on disk. Previously, every image and layer used a randomly assigned UUID. In 1.10 we implemented a content addressable method using an ID, based on a secure hash of the image and layer data.

[…]

To make your current images accessible to the new model we have to migrate them to content addressable storage. This means calculating the secure checksums for your current data.

All your current images, tags and containers are automatically migrated to the new foundation the first time you start Docker Engine 1.10. Before loading your container, the daemon will calculate all needed checksums for your current data, and after it has completed, all your images and tags will have brand new secure IDs.

While this is simple operation, calculating SHA256 checksums for your files can take time if you have lots of image data. On average you should assume that migrator can process data at a speed of 100MB/s. During this time your Docker daemon won’t be ready to respond to requests.

[cpuguy83]

Yes, this is most likely due to the migration to content addressable storage.
Docker 1.10 will block until all images have been migrated (or skipped due to an error).
The above link provides a stand-alone migration tool that can be run against live daemons prior to upgrade to pre-process these images so you don't have downtime.

[yorele]

@cpuguy83 if the daemon is upgraded on one machine, are the images useable on another which has not been upgraded?
I just tried updating my build machine. I don't want to spend hours waiting for images to migrate to realize they won't work on my hosts.

Thanks.

[cpuguy83]

@yorele The images themselves are not changed, the migration is only computing checksums for all the images (and can be run asynchronously via a separate tool, even in a container)

[yorele]

@cpuguy83 Great, thanks...