issue about couchdb

[zhy989]

when I run "ansible-playbook -i environments/kube couchdb.yml", the error is

**TASK [couchdb : wait until the CouchDB in this host is up and running] ************************************************************************
Wednesday 21 June 2017 01:28:22 +0000 (0:00:00.681) 0:00:01.859 ********
fatal: [127.0.0.1]: FAILED! => {"changed": false, "elapsed": 60, "failed": true, "msg": "Timeout when waiting for couchdb.openwhisk:5984"}

PLAY RECAP ************************************************************************************************************************************
127.0.0.1 : ok=2 changed=1 unreachable=0 failed=1**

[zhy989]

kubectl -n openwhisk get svc

NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
consul 10.0.0.99 8300/TCP,8301/TCP,8301/UDP,8302/TCP,8302/UDP,8400/TCP,8500/TCP,8600/TCP,8600/UDP 2h
controller 10.0.0.12 10001/TCP 2h
couchdb 10.0.0.151 5984/TCP 2h
invoker None 8080/TCP 2h
kafka 10.0.0.121 9092/TCP 2h
nginx 10.0.0.7 80:31654/TCP,443:30743/TCP,8443:30395/TCP 2h
zookeeper 10.0.0.150 2181/TCP,2888/TCP,3888/TCP 2h

[zhy989]

I run CLI as below:
case 1:
$cd incubator-openwhisk-deploy-kube/.travis
$TRAVIS_KUBE_VERSION=v1.5.6 TRAVIS_ETCD_VERSION=v3.0.14 ./setup.sh
$TRAVIS_KUBE_VERSION=v1.5.6 ./build.sh
$ kubectl get job --all-namespaces -o wide --show-all
NAMESPACE NAME DESIRED SUCCESSFUL AGE CONTAINER(S) IMAGE(S) SELECTOR
openwhisk configure-openwhisk 1 0 1h configure-openwhisk danlavine/whisk_config:v1.5.6 controller-uid=03aca90e-56f3-11e7-a62c-fa163e0f1f49
$kubectl get pods --all-namespaces -o wide --show-all
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE
kube-system kube-dns-2185667875-lqfhm 0/4 Pending 0 15h
openwhisk configure-openwhisk-n4917 0/1 Pending 0 1h
$kubectl -n openwhisk logs configure-openwhisk-n4917
There is no log , the job to configure OpenWhisk did not finish
case 2:
I try to run the configure-openwhisk job in the localhost kernel not in the kubernetes pod, copy the command from file "incubator-openwhisk-deploy-kube/configure/configure.sh" step by step , when run "ansible-playbook -i environments/kube couchdb.yml",got the error:

fatal: [127.0.0.1]: FAILED! => {"changed": false, "elapsed": 60, "failed": true, "msg": "Timeout when waiting for couchdb.openwhisk:5984"}

Questions:
1.Does the job configure-openwhisk run on the cloud side or on the localhost?
2. In case 2, the couchdb service is started , why failed ? How to config the network IP?
$kubectl -n openwhisk get svc
NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
consul 10.0.0.99 8300/TCP,8301/TCP,8301/UDP,8302/TCP,8302/UDP,8400/TCP,8500/TCP,8600/TCP,8600/UDP 2h
controller 10.0.0.12 10001/TCP 2h
couchdb 10.0.0.151 5984/TCP 2h
invoker None 8080/TCP 2h
kafka 10.0.0.121 9092/TCP 2h
nginx 10.0.0.7 80:31654/TCP,443:30743/TCP,8443:30395/TCP 2h
zookeeper 10.0.0.150 2181/TCP,2888/TCP,3888/TCP 2h
3.What services should be started before deploying configure-openwhisk?
Anyone who can help me, thanks!

[ghost]

@zhy989 That first issue looks like a timeout when trying to pull the docker images.

Also, it sounds like you are running ansible-playbook -i environments/kube couchdb.yml by hand which seems a bit odd. Everything should be configured via the whisk_config image when you run the command kubectl apply -f configure/configure_whisk.yml from the instructions in configure openwhisk.

When you posted:

kubectl get pods --all-namespaces -o wide --show-all
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE
kube-system kube-dns-2185667875-lqfhm 0/4 Pending 0 15h 
openwhisk configure-openwhisk-n4917 0/1 Pending 0 1h 

It looks like your Kubernetes environment is not able to stand up correctly since the KubeDNS pods should be running by default. Specifically kube-dns-2185667875-lqfhm 0/4 Pending 0 15h, those Pods are coming from Kubernetes and without them running, I think there is some other problem going on. Unfortunately this is now debugging Kubernetes and those errors might be outside my scope of knowledge. Luckily, if you ran TRAVIS_KUBE_VERSION=v1.5.6 TRAVIS_ETCD_VERSION=v3.0.14 ./setup.sh then at the end of that output it should tell you the location for Kubernetes logs. I would start there to take a look at why Kube is not up and running successfully.

[zhy989]

Run cli "kubectl apply -f configure/configure_whisk.yml" on the local host.

Run script "/incubator-openwhisk-deploy-kube/configure/configure.sh" on the Image (danlavine/whisk_config v1.5.6), the cli "kubectl " is in the script file configure.sh.

So cli kubectl should be run successfully on both localhost and Image , right?

What components are installed on the localhost and what components are installed on the image ?

[ghost]

currently everything is installed via the image. This is because it uses KubeDNS to perform any configuration to components that come up.

To do this on local host, then all of the images would need a public endpoint so they can be configured. But that shouldn't be required and is unsafe. So everything needs to be configured from the image running in Kubernetes.

[ghost]

Also @zhy989 when checking for logs, the current configure-openwhisk-xxxxx image might not be the most current one. Check with the kubectl -n openwhisk get pods -o wide --show-all for the oldest configure-openwhisk-xxxxx image. Then you can check the logs against the oldest image with kubectl -n openwhisk logs configure-openwhisk-xxxxx.

[zhy989]

@DanLavine thanks!
Did you run CLI "kubectl apply -f configure/configure_whisk.yml" successfully with the Image danlavine/whisk_config v1.5.6 ? I think the CLI "kubectl" does not work well because there is no docker package in that image .

[ghost]

@zhy989 I was able to deploy OpenWhisk by running kubectl apply -f configure/configure_whisk.yml. That image should not need the docker package since it is using kubectl to setup the othe OpenWhisk Pods from inside of the configure_whisk image.

For example, take a look at this and this as an example what that configuration pod is trying to do.

Lastly, there are a number of things currently happening as we are trying to remove this pod and deploy OpenWhisk in a more "Kubernetes way". Only using yaml files. If you take a look at #issue22, things might need to be deployed by hand moving forward.

[zhy989]

@DanLavine

Is couchdb deployed successfully ?

• kubectl get svc --all-namespaces
  NAMESPACE NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
  default kubernetes 10.0.0.1 443/TCP 1d
  kube-system kube-dns 10.0.0.10 53/UDP,53/TCP 1d
  openwhisk consul 10.0.0.169 8300/TCP,8301/TCP,8301/UDP,8302/TCP,8302/UDP,8400/TCP,8500/TCP,8600/TCP,8600/UDP 18h
  openwhisk controller 10.0.0.70 10001/TCP 18h
  openwhisk couchdb 10.0.0.50 5984/TCP 18h
  openwhisk invoker None 8080/TCP 18h
  openwhisk kafka 10.0.0.150 9092/TCP 18h
  openwhisk nginx 10.0.0.107 80:30435/TCP,443:30521/TCP,8443:32685/TCP 18h
  openwhisk zookeeper 10.0.0.2 2181/TCP,2888/TCP,3888/TCP 18h

• kubectl get po --all-namespaces
  NAMESPACE NAME READY STATUS RESTARTS AGE
  kube-system kube-dns-2185667875-sd506 4/4 Running 1 1d
  openwhisk consul-57995027-4ngbf 0/2 ContainerCreating 0 18h
  openwhisk couchdb-109298327-rstgl 1/1 Running 2 22h

but when I run "ansible-playbook -i environments/kube couchdb.yml" , there is also "time out" like this:

• TASK [couchdb : wait until the CouchDB in this host is up and running] ************************************************************************
  Wednesday 05 July 2017 02:04:44 +0000 (0:00:00.303) 0:00:00.761 ********
  fatal: [172.17.0.1]: FAILED! => {"changed": false, "elapsed": 129, "failed": true, "msg": "Timeout when waiting for couchdb.openwhisk:5984"}

when I run "ansible-playbook -i environments/kube initdb.yml", the error is as below:

• TASK [check if openwhisk_kube_subjects with CouchDB exists] ***********************************************************************************
  Wednesday 05 July 2017 02:37:15 +0000 (0:00:00.421) 0:00:00.445 ********
  fatal: [ansible]: FAILED! => {"changed": false, "content": "", "failed": true, "msg": "Status code was not [200, 404]: Request failed: <urlopen error [Errno -2] Name or service not known>", "redirected": false, "status": -1, "url": "http://couchdb.openwhisk:5984/openwhisk_kube_subjects"}

[ghost]

@zhy989 yes CouchDB is deployed successfully and has already been configured. You should not need to run any ansible-playbook .... The configuration image already does that for you. The only commands that should be uses to manage the OpenWhisk on Kubernetes are kubectl commands.

[zhy989]

@DanLavine
I have installed some components successully, but there is no invoker component. so I can not invoke hello world example successully.

• kubectl get po --all-namespaces
  NAMESPACE NAME READY STATUS RESTARTS AGE
  kube-system kube-dns-2185667875-vm94p 4/4 Running 4 1d
  openwhisk consul-57995027-hk0dj 2/2 Running 0 2m
  openwhisk controller-3250411552-d76x4 1/1 Running 0 1m
  openwhisk couchdb-109298327-btwpv 1/1 Running 0 2m
  openwhisk kafka-1060962555-psk3j 1/1 Running 0 1m
  openwhisk nginx-1877697612-62bgw 1/1 Running 0 40s
  openwhisk zookeeper-1304892743-9f3hl 1/1 Running 0 1m

• kubectl get deployment --all-namespaces
  NAMESPACE NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
  kube-system kube-dns 1 1 1 1 1d
  openwhisk consul 1 1 1 1 2m
  openwhisk controller 1 1 1 1 1m
  openwhisk couchdb 1 1 1 1 3m
  openwhisk kafka 1 1 1 1 2m
  openwhisk nginx 1 1 1 1 1m
  openwhisk zookeeper 1 1 1 1 2m

• kubectl get svc --all-namespaces
  NAMESPACE NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
  default kubernetes 10.0.0.1 443/TCP 1d
  kube-system kube-dns 10.0.0.10 53/UDP,53/TCP 1d
  openwhisk consul 10.0.0.51 8300/TCP,8301/TCP,8301/UDP,8302/TCP,8302/UDP,8400/TCP,8500/TCP,8600/TCP,8600/UDP 8m
  openwhisk controller 10.0.0.209 8080/TCP 8m
  openwhisk couchdb 10.0.0.153 5984/TCP 8m
  openwhisk kafka 10.0.0.17 9092/TCP 8m
  openwhisk nginx 10.0.0.196 80:30622/TCP,443:31522/TCP,8443:30145/TCP 6m
  openwhisk zookeeper 10.0.0.84 2181/TCP,2888/TCP,3888/TCP 8m

• kubectl get StatefulSet --all-namespaces
  No resources found.

• wsk action list -i
  actions
  /guest/hello private nodejs:6

• wsk action create hello1 hello.js -i
  ok: created action hello1

• wsk action list -i
  actions
  /guest/hello1 private nodejs:6
  /guest/hello private nodejs:6

• wsk action invoke hello1 -i
  error: Unable to invoke action 'hello1': There was an internal server error. (code 43)
  Run 'wsk --help' for usage.

[ghost]

hi @zhy989. This is a problem with the configuration image. There is a PR that removes the Invoker, but the problem is that the configuration Image needs to be fixed so CI passes before the code goes into the repo. Just look at the PR for how to deploy the Invoker.

This will probably be a problem with other pieces as well until the configuration image goes away. PRs just need to be merged in with additional docs, so that would be the place to check if something isn't deploying anymore.

[zhy989]

@DanLavine thanks!

I download invoker script file from branch "remotes/origin/ansible_invoker_remove", and I have set up the openwhisk+k8s environment succesfully

• kubectl -n openwhisk get po
  NAME READY STATUS RESTARTS AGE
  consul-57995027-nqmnz 2/2 Running 0 56m
  controller-3250411552-dpprr 1/1 Running 0 55m
  couchdb-109298327-57cj6 1/1 Running 0 56m
  invoker-0 1/1 Running 2 54m
  kafka-1060962555-5wzkb 1/1 Running 0 55m
  nginx-1877697612-j20pc 1/1 Running 0 56s
  zookeeper-1304892743-21j8j 1/1 Running 0 55m

• kubectl -n openwhisk get svc
  NAME CLUSTER-IP EXTERNAL-IP PORT(S) AGE
  consul 10.0.0.21 8300/TCP,8301/TCP,8301/UDP,8302/TCP,8302/UDP,8400/TCP,8500/TCP,8600/TCP,8600/UDP 1h
  controller 10.0.0.245 8080/TCP 1h
  couchdb 10.0.0.216 5984/TCP 1h
  kafka 10.0.0.251 9092/TCP 1h
  nginx 10.0.0.173 80:30572/TCP,443:30209/TCP,8443:32279/TCP 24m
  zookeeper 10.0.0.16 2181/TCP,2888/TCP,3888/TCP 1h

• kubectl -n openwhisk get statefulset
  NAME DESIRED CURRENT AGE
  invoker 1 1 1h

• wsk -i action create hello1 hello.js
  ok: created action hello1

• wsk -i action invoke hello1
  ok: invoked /_/hello1 with id 20bb799dc4b447c19daa056ecfe1d9e3

• wsk -i activation result 20bb799dc4b447c19daa056ecfe1d9e3
  {
  "payload": "Hello world"
  }

[ghost]

@zhy989 👍. Glad you have everything deployed. If all your issues have been resolved, could you please close this issue. Thanks!