Integration tests failing in master

[technolo-g]

Hi guys,
When we were hacking yesterday at the open-source-a-thon, we couldn't get the integration tests to pass on master. We were able to write an additional one that works but the ones already there seem to be jacked (or we couldn't understand the instructions). Here is what I'm seeing:

[23:31:10 mbajor@mbajor-DraakBookPro:~/go/src/github.com/technolo-g/swarm]
$ git status
On branch master
Your branch is up-to-date with 'origin/master'.
nothing to commit, working directory clean
[23:31:15 mbajor@mbajor-DraakBookPro:~/go/src/github.com/technolo-g/swarm]
$ bats test/integration
 ✗ docker info should return the number of nodes
   (from function `wait_until_reachable' in file test/integration/helpers.bash, line 36,
    from function `start_docker' in file test/integration/helpers.bash, line 72,
    in test file test/integration/api.bats, line 11)
     `start_docker 3' failed
   time="2015-03-26T23:31:26-06:00" level="fatal" msg="Cannot connect to the Docker daemon. Is 'docker -d' running on this host?"
   Attempt to connect to 127.0.0.1:5814 failed for the 1 time
   time="2015-03-26T23:31:26-06:00" level="fatal" msg="Cannot connect to the Docker daemon. Is 'docker -d' running on this host?"
   Attempt to connect to 127.0.0.1:5814 failed for the 2 time
   time="2015-03-26T23:31:27-06:00" level="fatal" msg="Cannot connect to the Docker daemon. Is 'docker -d' running on this host?"
   Attempt to connect to 127.0.0.1:5814 failed for the 3 time
   time="2015-03-26T23:31:27-06:00" level="fatal" msg="Cannot connect to the Docker daemon. Is 'docker -d' running on this host?"
   Attempt to connect to 127.0.0.1:5814 failed for the 4 time
   time="2015-03-26T23:31:28-06:00" level="fatal" msg="Cannot connect to the Docker daemon. Is 'docker -d' running on this host?"
   Attempt to connect to 127.0.0.1:5814 failed for the 5 time
   time="2015-03-26T23:31:28-06:00" level="fatal" msg="Cannot connect to the Docker daemon. Is 'docker -d' running on this host?"
   kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
   Stopping dd8751ab663022ffdcd2a78d8c6c29738b16138c72af3ce475dae83ae9164f76
   Stopping 623cb17a4e8115ba99e89726adb53b2a6e2514833546e1e21f033cf4f1405820
   Stopping 98d931d6fecde34e85fae9b2050b1e2b19ccc0de601e762deb5ce312cac7aea2
 - docker ps -n 3 should return the 3 last containers, including non running one (skipped)
 - docker ps -l should return the last container, including non running one (skipped)
 ✗ node constraint
   (from function `wait_until_reachable' in file test/integration/helpers.bash, line 36,
    from function `start_docker' in file test/integration/helpers.bash, line 72,
    in test file test/integration/constraints.bats, line 11)
     `start_docker 2' failed
   time="2015-03-26T23:31:30-06:00" level="fatal" msg="Cannot connect to the Docker daemon. Is 'docker -d' running on this host?"
   Attempt to connect to 127.0.0.1:5046 failed for the 1 time
   time="2015-03-26T23:31:30-06:00" level="fatal" msg="Cannot connect to the Docker daemon. Is 'docker -d' running on this host?"
   Attempt to connect to 127.0.0.1:5046 failed for the 2 time
   time="2015-03-26T23:31:31-06:00" level="fatal" msg="Cannot connect to the Docker daemon. Is 'docker -d' running on this host?"
   Attempt to connect to 127.0.0.1:5046 failed for the 3 time
   time="2015-03-26T23:31:31-06:00" level="fatal" msg="Cannot connect to the Docker daemon. Is 'docker -d' running on this host?"
   Attempt to connect to 127.0.0.1:5046 failed for the 4 time
   time="2015-03-26T23:31:32-06:00" level="fatal" msg="Cannot connect to the Docker daemon. Is 'docker -d' running on this host?"
   Attempt to connect to 127.0.0.1:5046 failed for the 5 time
   time="2015-03-26T23:31:32-06:00" level="fatal" msg="Cannot connect to the Docker daemon. Is 'docker -d' running on this host?"
   kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
   Stopping e7b8fa748321465ebb0c9d75ccf31fcbec8243d6edb6cf0c4e77f2012509f956
   Stopping 3f6d597d2440fd0db0c7654442b99f65ba454719e59bf7f14666fa95be3b7788
 ✗ label constraints
   (from function `wait_until_reachable' in file test/integration/helpers.bash, line 36,
    from function `start_docker' in file test/integration/helpers.bash, line 72,
    in test file test/integration/constraints.bats, line 35)
     `start_docker 1 --label foo=a' failed
   time="2015-03-26T23:31:34-06:00" level="fatal" msg="Cannot connect to the Docker daemon. Is 'docker -d' running on this host?"
   Attempt to connect to 127.0.0.1:5268 failed for the 1 time
   time="2015-03-26T23:31:34-06:00" level="fatal" msg="Cannot connect to the Docker daemon. Is 'docker -d' running on this host?"
   Attempt to connect to 127.0.0.1:5268 failed for the 2 time
   time="2015-03-26T23:31:35-06:00" level="fatal" msg="Cannot connect to the Docker daemon. Is 'docker -d' running on this host?"
   Attempt to connect to 127.0.0.1:5268 failed for the 3 time
   time="2015-03-26T23:31:35-06:00" level="fatal" msg="Cannot connect to the Docker daemon. Is 'docker -d' running on this host?"
   Attempt to connect to 127.0.0.1:5268 failed for the 4 time
   time="2015-03-26T23:31:36-06:00" level="fatal" msg="Cannot connect to the Docker daemon. Is 'docker -d' running on this host?"
   Attempt to connect to 127.0.0.1:5268 failed for the 5 time
   time="2015-03-26T23:31:36-06:00" level="fatal" msg="Cannot connect to the Docker daemon. Is 'docker -d' running on this host?"
   kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
   Stopping b635e7a7fba0fdbeb06ac6f3b00c15ae6d30a1eff8f63fccac69a73332307b73
 ✓ test that create doesn't accept any arugments
 ✓ version string should contain a proper number with git commit

7 tests, 3 failures, 2 skipped
[23:31:37 mbajor@mbajor-DraakBookPro:~/go/src/github.com/technolo-g/swarm]
$ docker info
Containers: 0
Images: 45
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 46
Execution Driver: native-0.2
Kernel Version: 3.13.0-32-generic
Operating System: Ubuntu 14.04.1 LTS
CPUs: 8
Total Memory: 7.798 GiB
Name: vagrant-trusty64
ID: 4HYJ:SJ3B:6OQV:CEXG:ZHPV:56ZK:LIUC:V7LE:SB66:N5SH:53MU:XHM4
Username: mbajor
Registry: [https://index.docker.io/v1/]
WARNING: No swap limit support
[23:32:50 mbajor@mbajor-DraakBookPro:~/go/src/github.com/technolo-g/swarm]
$ docker version
Client version: 1.5.0
Client API version: 1.17
Go version (client): go1.4.1
Git commit (client): a8a31ef
OS/Arch (client): darwin/amd64
Server version: 1.5.0
Server API version: 1.17
Go version (server): go1.4.1
Git commit (server): a8a31ef
[23:32:52 mbajor@mbajor-DraakBookPro:~/go/src/github.com/technolo-g/swarm]
$ ./swarm --version
swarm version 0.2.0 (HEAD)
[23:32:57 mbajor@mbajor-DraakBookPro:~/go/src/github.com/technolo-g/swarm]
$ ls -lahtr swarm
-rwxr-xr-x  1 mbajor  staff   8.8M Mar 26 23:29 swarm
[23:33:01 mbajor@mbajor-DraakBookPro:~/go/src/github.com/technolo-g/swarm]
$ date
Thu Mar 26 23:33:03 MDT 2015

What docker ps looks like during the test:

root@vagrant-trusty64:~# docker ps
CONTAINER ID        IMAGE                  COMMAND                CREATED             STATUS              PORTS                      NAMES
08f748f6362f        aluzzardi/docker:1.5   "hack/dind docker -d   3 seconds ago       Up 1 seconds        127.0.0.1:5821->5821/tcp   node-2
707aef054d4d        aluzzardi/docker:1.5   "hack/dind docker -d   3 seconds ago       Up 2 seconds        127.0.0.1:5820->5820/tcp   node-1
b05a5f78bc47        aluzzardi/docker:1.5   "hack/dind docker -d   3 seconds ago       Up 2 seconds        127.0.0.1:5819->5819/tcp   node-0

[vieux]

@aluzzardi can you take a look ?

[aluzzardi]

Hey @technolo-g, silly question: Is docker reachable from the terminal you are trying to run bats from? (as in, DOCKER_HOST properly set, pointing to an actual docker daemon, not swarm itself, etc)

[chanwit]

totally guessing but @technolo-g have you got any $http_proxy setting there?

[technolo-g]

Sorry guys, missed this.

@aluzzardi yes docker was reachable and I saw the tests creating containers and destroying them on the Docker host. Also the commands in the paste were run sequentially in the same terminal.

@chanwit where would that setting live?

[chanwit]

just export http_proxy= in the shell.
but I could be wrong as you have got some test case running.

[aluzzardi]

Are you running the integration tests on an engine using devicemapper as a graph driver?

It looks like there are problems running docker-in-docker instances with devicemapper, as reported in jpetazzo/dind#19

A short term fix would be to run them on a system with aufs.

[abronan]

@technolo-g This seems to be indeed due to devicemapper, I just ran into the same issue with a Vagrant setup without aufs installed.

[abronan]

@technolo-g So your docker info shows that the local daemon uses aufs but it seems that your dind (docker in docker) images are choosing devicemapper instead..

If I run docker with devicemapper but the aufs module is loaded, dind will choose aufs by default and the tests will run fine. If I unload aufs, then dind will use devicemapper as the default storage driver and I run into the same issue as you.

Overall, it seems to be related to dind choosing devicemapper as the storage driver. You can probably confirm this by commenting the teardown() function in a test file, pointing DOCKER_HOST to one of the dind instances and run docker info :D

[aluzzardi]

Hey @technolo-g, a lot has changed since.

Can you:

• Try again exactly like you did
• Try also with test/integration/run.sh

[aluzzardi]

Closing for now. Please re-open if it happens again.