Unable to connect to controller0 while running openwhisk.yml (local deployment)

[chinnshan]

Environment details:

• local deployment
• Docker version 17.12.0-ce, build c97c6d6, ubuntu 16.04 LTS, Vagrant 1.8.1, couchdb 2.1.1

Steps to reproduce the issue:

1. Follow the steps for local development on Ubuntu
2. Use CouchDB as the database
3. The situation will occur while running openwhisk.yml with ansible-playbook.
4. It waits for the controller for all 12 retries and then gives up. However, I can see a container running controller on 0.0.0.0:10001->8080/tcp.

Provide the expected results and outputs:

The expected result is to run **sudo ansible-playbook openwhisk.yml** command without any failures.

Provide the actual results and outputs:

TASK [controller : create seed nodes list] **********************************************************************************************
Monday 05 February 2018  17:14:58 +0200 (0:00:00.179)       0:00:37.382 ******* 
ok: [controller0] => (item=(0, u'172.17.0.1'))

TASK [controller : (re)start controller] ************************************************************************************************
Monday 05 February 2018  17:14:58 +0200 (0:00:00.210)       0:00:37.592 ******* 
changed: [controller0]

TASK [controller : wait until the Controller in this host is up and running] ************************************************************
Monday 05 February 2018  17:15:02 +0200 (0:00:03.716)       0:00:41.308 ******* 
FAILED - RETRYING: wait until the Controller in this host is up and running (12 retries left).
FAILED - RETRYING: wait until the Controller in this host is up and running (11 retries left).
FAILED - RETRYING: wait until the Controller in this host is up and running (10 retries left).
FAILED - RETRYING: wait until the Controller in this host is up and running (9 retries left).
FAILED - RETRYING: wait until the Controller in this host is up and running (8 retries left).
FAILED - RETRYING: wait until the Controller in this host is up and running (7 retries left).
FAILED - RETRYING: wait until the Controller in this host is up and running (6 retries left).
FAILED - RETRYING: wait until the Controller in this host is up and running (5 retries left).
FAILED - RETRYING: wait until the Controller in this host is up and running (4 retries left).
FAILED - RETRYING: wait until the Controller in this host is up and running (3 retries left).
FAILED - RETRYING: wait until the Controller in this host is up and running (2 retries left).
FAILED - RETRYING: wait until the Controller in this host is up and running (1 retries left).
fatal: [controller0]: FAILED! => {"attempts": 12, "changed": false, "content": "", "failed": true, "msg": "Status code was not [200]: Request failed: <urlopen error [Errno 111] Connection refused>", "redirected": false, "status": -1, "url": "http://172.17.0.1:10001/ping"}

PLAY RECAP ******************************************************************************************************************************
controller0                : ok=16   changed=3    unreachable=0    failed=1   
kafka0                     : ok=8    changed=6    unreachable=0    failed=0   

Monday 05 February 2018  17:16:04 +0200 (0:01:02.233)       0:01:43.542 ******* 
=============================================================================== 
controller : wait until the Controller in this host is up and running -- 62.23s
kafka : (re)start kafka ------------------------------------------------ 13.16s
zookeeper : wait until the Zookeeper in this host is up and running ----- 7.29s
kafka : wait until the kafka server started up -------------------------- 5.34s
controller : (re)start controller --------------------------------------- 3.72s
zookeeper : (re)start zookeeper ----------------------------------------- 3.35s
zookeeper : pull the zookeeper:3.4 image -------------------------------- 2.24s
kafka : pull the wurstmeister/kafka:0.11.0.1 image ---------------------- 1.70s
Gathering Facts --------------------------------------------------------- 0.61s
controller : copy jmxremote password file ------------------------------- 0.59s
Gathering Facts --------------------------------------------------------- 0.40s
controller : copy jmxremote access file --------------------------------- 0.40s
Gathering Facts --------------------------------------------------------- 0.36s
controller : check if whisk_local_whisks with CouchDB exists ------------ 0.31s
controller : ensure controller config directory is created with permissions --- 0.31s
controller : ensure controller log directory is created with permissions --- 0.23s
controller : create seed nodes list ------------------------------------- 0.21s
controller : prepare controller ports ----------------------------------- 0.19s
controller : check if whisk_local_subjects with CouchDB exists ---------- 0.19s
controller : check if whisk_local_activations with CouchDB exists ------- 0.19s


*******************************************************************************************************************

sudo docker ps
CONTAINER ID        IMAGE                         COMMAND                  CREATED              STATUS              PORTS                                                                                                 NAMES
40caf6dde712        whisk/controller:latest       "/bin/sh -c 'exec /i…"   About a minute ago   Up About a minute   0.0.0.0:15000->15000/tcp, 0.0.0.0:16000->16000/tcp, 0.0.0.0:8000->2551/tcp, 0.0.0.0:10001->8080/tcp   controller0
e3caca9904ac        wurstmeister/kafka:0.11.0.1   "start-kafka.sh"         About a minute ago   Up About a minute   0.0.0.0:9092->9092/tcp                                                                                kafka0
6cb008c9dcb6        zookeeper:3.4                 "/docker-entrypoint.…"   About a minute ago   Up About a minute   0.0.0.0:2181->2181/tcp, 0.0.0.0:2888->2888/tcp, 0.0.0.0:3888->3888/tcp                                zookeeper0
5dd7b8271cc9        openwhisk/apigateway:0.9.8    "/usr/bin/dumb-init …"   7 minutes ago        Up 7 minutes        80/tcp, 8423/tcp, 0.0.0.0:9000->9000/tcp, 0.0.0.0:9001->8080/tcp                                      apigateway
54f5b954e9f4        redis:3.2                     "docker-entrypoint.s…"   7 minutes ago        Up 7 minutes        0.0.0.0:6379->6379/tcp                                                                                redis
8d53009037bd        apache/couchdb:2.1            "tini -- /docker-ent…"   9 minutes ago        Up 9 minutes        0.0.0.0:4369->4369/tcp, 0.0.0.0:5984->5984/tcp, 0.0.0.0:9100->9100/tcp                                couchdb

Additional information you deem important:

• I faced same issue with other components such as kafka but was able to connect in few retries.

[rabbah]

can you share wskadmin syslog get controller0? likely there's a missing configuration parameter which will prevent the controller from starting up.

[chinnshan]

Hi,

Kindly find required logs in the attached file.

controller0_syslogs.txt
Thank you.

[chinnshan]

Hi,

I am able to fix it.

Cause: Docker Engine was not installed properly.

Solution: Installed it using instructions from official docker website.

Thank you for your help.