Error in installing route management actions under Ubuntu

[houshengbo]

TASK [routemgmt : install route management actions] ****************************
Monday 25 June 2018  01:03:17 +0000 (0:00:00.118)       0:04:50.071 *********** 
fatal: [ansible]: FAILED! => {"changed": true, "cmd": "./installRouteMgmt.sh /home/openwhisk16/workspace/incubator-openwhisk/ansible/../ansible/files/auth.whisk.system 172.17.0.1 /whisk.system /home/openwhisk16/workspace/incubator-openwhisk/ansible/../bin/wsk", "delta": "0:00:05.363667", "end": "2018-06-25 01:03:23.608836", "msg": "non-zero return code", "rc": 244, "start": "2018-06-25 01:03:18.245169", "stderr": "+ '[' 4 -eq 0 ']'\n+ AUTH=/home/openwhisk16/workspace/incubator-openwhisk/ansible/../ansible/files/auth.whisk.system\n+ APIHOST=172.17.0.1\n+ NAMESPACE=/whisk.system\n+ WSK_CLI=/home/openwhisk16/workspace/incubator-openwhisk/ansible/../bin/wsk\n+ WHISKPROPS_FILE=/home/openwhisk16/workspace/incubator-openwhisk/ansible/../whisk.properties\n+ '[' -z '' ']'\n++ cut -d= -f2\n++ fgrep apigw.auth.user= /home/openwhisk16/workspace/incubator-openwhisk/ansible/../whisk.properties\n+ GW_USER=\n+ '[' -z '' ']'\n++ cut -d= -f2-\n++ fgrep apigw.auth.pwd= /home/openwhisk16/workspace/incubator-openwhisk/ansible/../whisk.properties\n+ GW_PWD=\n+ '[' -z '' ']'\n++ cut -d= -f2\n++ fgrep apigw.host.v2= /home/openwhisk16/workspace/incubator-openwhisk/ansible/../whisk.properties\n+ GW_HOST_V2=http://172.17.0.1:9000/v2\n+ '[' -f /home/openwhisk16/workspace/incubator-openwhisk/ansible/../ansible/files/auth.whisk.system ']'\n++ cat /home/openwhisk16/workspace/incubator-openwhisk/ansible/../ansible/files/auth.whisk.system\n+ AUTH=789c46b1-71f6-4ed5-8c54-816aa4f8c502:abczO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIwP\n+ export WSK_CONFIG_FILE=\n+ WSK_CONFIG_FILE=\n+ echo Installing apimgmt package\n+ /home/openwhisk16/workspace/incubator-openwhisk/ansible/../bin/wsk -i --apihost 172.17.0.1 package update --auth 789c46b1-71f6-4ed5-8c54-816aa4f8c502:abczO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIwP --shared no /whisk.system/apimgmt -a description 'This package manages the gateway API configuration.' -p gwUser '' -p gwPwd '' -p gwUrlV2 http://172.17.0.1:9000/v2\nerror: Package update failed: The connection failed, or timed out. (HTTP status code 500)", "stderr_lines": ["+ '[' 4 -eq 0 ']'", "+ AUTH=/home/openwhisk16/workspace/incubator-openwhisk/ansible/../ansible/files/auth.whisk.system", "+ APIHOST=172.17.0.1", "+ NAMESPACE=/whisk.system", "+ WSK_CLI=/home/openwhisk16/workspace/incubator-openwhisk/ansible/../bin/wsk", "+ WHISKPROPS_FILE=/home/openwhisk16/workspace/incubator-openwhisk/ansible/../whisk.properties", "+ '[' -z '' ']'", "++ cut -d= -f2", "++ fgrep apigw.auth.user= /home/openwhisk16/workspace/incubator-openwhisk/ansible/../whisk.properties", "+ GW_USER=", "+ '[' -z '' ']'", "++ cut -d= -f2-", "++ fgrep apigw.auth.pwd= /home/openwhisk16/workspace/incubator-openwhisk/ansible/../whisk.properties", "+ GW_PWD=", "+ '[' -z '' ']'", "++ cut -d= -f2", "++ fgrep apigw.host.v2= /home/openwhisk16/workspace/incubator-openwhisk/ansible/../whisk.properties", "+ GW_HOST_V2=http://172.17.0.1:9000/v2", "+ '[' -f /home/openwhisk16/workspace/incubator-openwhisk/ansible/../ansible/files/auth.whisk.system ']'", "++ cat /home/openwhisk16/workspace/incubator-openwhisk/ansible/../ansible/files/auth.whisk.system", "+ AUTH=789c46b1-71f6-4ed5-8c54-816aa4f8c502:abczO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIwP", "+ export WSK_CONFIG_FILE=", "+ WSK_CONFIG_FILE=", "+ echo Installing apimgmt package", "+ /home/openwhisk16/workspace/incubator-openwhisk/ansible/../bin/wsk -i --apihost 172.17.0.1 package update --auth 789c46b1-71f6-4ed5-8c54-816aa4f8c502:abczO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIwP --shared no /whisk.system/apimgmt -a description 'This package manages the gateway API configuration.' -p gwUser '' -p gwPwd '' -p gwUrlV2 http://172.17.0.1:9000/v2", "error: Package update failed: The connection failed, or timed out. (HTTP status code 500)"], "stdout": "Installing apimgmt package", "stdout_lines": ["Installing apimgmt package"]}

[FAILED]
> ./installRouteMgmt.sh /home/openwhisk16/workspace/incubator-openwhisk/ansible/../ansible/files/auth.whisk.system 172.17.0.1 /whisk.system /home/openwhisk16/workspace/incubator-openwhisk/ansible/../bin/wsk
non-zero return code
Installing apimgmt package
+ '[' 4 -eq 0 ']' + AUTH=/home/openwhisk16/workspace/incubator-
openwhisk/ansible/../ansible/files/auth.whisk.system + APIHOST=172.17.0.1 +
NAMESPACE=/whisk.system + WSK_CLI=/home/openwhisk16/workspace/incubator-
openwhisk/ansible/../bin/wsk + WHISKPROPS_FILE=/home/openwhisk16/workspace
/incubator-openwhisk/ansible/../whisk.properties + '[' -z '' ']' ++ cut -d= -f2
++ fgrep apigw.auth.user= /home/openwhisk16/workspace/incubator-
openwhisk/ansible/../whisk.properties + GW_USER= + '[' -z '' ']' ++ cut -d= -f2-
++ fgrep apigw.auth.pwd= /home/openwhisk16/workspace/incubator-
openwhisk/ansible/../whisk.properties + GW_PWD= + '[' -z '' ']' ++ cut -d= -f2
++ fgrep apigw.host.v2= /home/openwhisk16/workspace/incubator-
openwhisk/ansible/../whisk.properties + GW_HOST_V2=http://172.17.0.1:9000/v2 +
'[' -f /home/openwhisk16/workspace/incubator-
openwhisk/ansible/../ansible/files/auth.whisk.system ']' ++ cat
/home/openwhisk16/workspace/incubator-
openwhisk/ansible/../ansible/files/auth.whisk.system + AUTH=789c46b1-71f6-4ed5-8
c54-816aa4f8c502:abczO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIw
P + export WSK_CONFIG_FILE= + WSK_CONFIG_FILE= + echo Installing apimgmt package
+ /home/openwhisk16/workspace/incubator-openwhisk/ansible/../bin/wsk -i
--apihost 172.17.0.1 package update --auth 789c46b1-71f6-4ed5-8c54-816aa4f8c502:
abczO3xZCLrMN6v2BKK1dXYFpXlPkccOFqm12CdAsMgRU4VrNZ9lyGVCGuMDGIwP --shared no
/whisk.system/apimgmt -a description 'This package manages the gateway API
configuration.' -p gwUser '' -p gwPwd '' -p gwUrlV2 http://172.17.0.1:9000/v2
error: Package update failed: The connection failed, or timed out. (HTTP status
code 500)

[houshengbo]

This is a known issue, which may happen when running openwhisk.yml. There could be a race condition issue with openwhisk.yml. routemgmt.yml is the last step of openwhisk.yml. When this error happens, we suggest running "ansible-playbook -i environments/local routemgmt.yml" separately.

[rabbah]

See #3760

[houshengbo]

I am not sure if this PR #3760 will resolve the issue here, but it is documented on the release 0.9.0 anyhow.

[rabbah]

It runs after post deploy - so naturally if the bug is because the system is not fully ready 3760 will surface the issue in installing the catalog. That’s why I pointed out you need to see that playbook also.

[houshengbo]

This issues is referred to in the doc for 0.9.0 release: https://github.com/apache/incubator-openwhisk-release/blob/master/releases/0.9.0-incubating/INSTALL.md#deploy-openwhisk

[rabbah]

@mdeuser what’s the status on #3760 which I think will affect this issue (which may require a different solution).

[mdeuser]

i agree that a different solution is preferred. is there a more deterministic way for the controller to indicate it's ready to accept requests? moving the .yml around only varies the timing window without guaranteeing the routemgmt package will be installed. it might be that other packages in postdeploy may experience the same issue.

[dgrove-oss]

Maybe #3838 will help? The motivation for adding this route to the controller was to give us a way to check that the system is ready to invoke actions before we start running kubernetes jobs that need to do that.

[rabbah]

I think they're orthogonal concerns and 3838 is not really related (the failure is related to CRUD operations which means the invokers are not relevant). I don't think there's an RCA for the failure in any case. It could be more that couchdb thread pool is not ready which prevents authentication. Calling wsk namespace list until it succeeds could be a more reliable health check.

I also think the playbooks which install the packages could retry on failure.

[rabbah]

@houshengbo do you have the nginx and controller logs where you observed the failure?