0.9.4 - The API gateway is unable to connect to the catalog and the catalog is unavailable

[Joe-Winchester]

**0.9.4 build - the API gateway shows the catalog as up, however the catalog isn't available. Quite a few errors in the log about SSL **

To Reproduce
Download 0.9.4, install and start Zowe.

Expected behavior
The API Gateway should show the gateway as started and clicking it to visit the gateway should show it available

Details

• Version: 0.9.4_RC1

[Colin-Stone]

stdout.txt
Included the stdout all the exceptions are at the end of the file. Spoke to Petr about it earlier as I thought it could be down to the change of config variable in api-mediation-start-gateway and updated flag to read: -Dapiml.security.verifySslCertificatesOfServices=false \ but this had no affect.

[plavjanik]

This message is repeated multiple times:

 org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://localhost:23956/api/v1/jobs/api-doc": Connect to localhost:23956 [localhost/127.0.0.1] failed: EDC8128I Connection refused. (Connection refus
 	at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:732) ~[spring-web-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
 	at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:680) ~[spring-web-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
 	at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:600) ~[spring-web-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
 	at com.ca.mfaas.apicatalog.services.status.APIDocRetrievalService.retrieveApiDocFromInstance(APIDocRetrievalService.java:147) ~[classes!/:na]
 	at com.ca.mfaas.apicatalog.services.cached.CacheRefreshService.retrieveApiDocFromInstance(CacheRefreshService.java:196) [classes!/:na]
 	at com.ca.mfaas.apicatalog.services.cached.CacheRefreshService.lambda$processInstance$3(CacheRefreshService.java:175) [classes!/:na]
 	at com.ca.mfaas.apicatalog.services.cached.CacheRefreshService$$Lambda$664.0000000076F498D0.get(Unknown Source) ~[na:na]
 	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1601) ~[na:1.8.0]
 	at java.lang.Thread.run(Thread.java:811) ~[na:2.9 (07-31-2018)]
 Caused by: org.apache.http.conn.HttpHostConnectException: Connect to localhost:23956 [localhost/127.0.0.1] failed: EDC8128I Connection refused. (Connection refused)
 	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) ~[httpclient-4.5.6.jar!/:4.5.6]
 	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) ~[httpclient-4.5.6.jar!/:4.5.6]
 	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:394) ~[httpclient-4.5.6.jar!/:4.5.6]
 	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[httpclient-4.5.6.jar!/:4.5.6]
 	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[httpclient-4.5.6.jar!/:4.5.6]
 	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.6.jar!/:4.5.6]
 	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.6.jar!/:4.5.6]
 	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.6.jar!/:4.5.6]
 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.6.jar!/:4.5.6]
 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.6.jar!/:4.5.6]
 	at org.springframework.http.client.HttpComponentsClientHttpRequest.executeInternal(HttpComponentsClientHttpRequest.java:87) ~[spring-web-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
 	at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48) ~[spring-web-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
 	at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53) ~[spring-web-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
 	at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:723) ~[spring-web-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
 	... 8 common frames omitted
 Caused by: java.net.ConnectException: EDC8128I Connection refused. (Connection refused)
 	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:380) ~[na:1.8.0]
 	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:236) ~[na:1.8.0]
 	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:218) ~[na:1.8.0]
 	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403) ~[na:1.8.0]
 	at java.net.Socket.connect(Socket.java:666) ~[na:1.8.0]
 	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:339) ~[httpclient-4.5.6.jar!/:4.5.6]
 	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.6.jar!/:4.5.6]
 	... 21 common frames omitted

[plavjanik]

Following build is used:

 09:34:20.785 [main] INFO com.ca.mfaas.product.service.BuildInfo - Service api-catalog-services version 0.2.0-SNAPSHOT #90 on null by e1f5c68a5c9e commit f7e72f9

https://wash.zowe.org:8443/job/API_Mediation/job/master/90/

[Colin-Stone]

This message is repeated multiple times:

 org.springframework.web.client.ResourceAccessException: I/O error on GET request for "https://localhost:23956/api/v1/jobs/api-doc": Connect to localhost:23956 [localhost/127.0.0.1] failed: EDC8128I Connection refused. (Connection refus
 	at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:732) ~[spring-web-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
 	at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:680) ~[spring-web-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
 	at org.springframework.web.client.RestTemplate.exchange(RestTemplate.java:600) ~[spring-web-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
 	at com.ca.mfaas.apicatalog.services.status.APIDocRetrievalService.retrieveApiDocFromInstance(APIDocRetrievalService.java:147) ~[classes!/:na]
 	at com.ca.mfaas.apicatalog.services.cached.CacheRefreshService.retrieveApiDocFromInstance(CacheRefreshService.java:196) [classes!/:na]
 	at com.ca.mfaas.apicatalog.services.cached.CacheRefreshService.lambda$processInstance$3(CacheRefreshService.java:175) [classes!/:na]
 	at com.ca.mfaas.apicatalog.services.cached.CacheRefreshService$$Lambda$664.0000000076F498D0.get(Unknown Source) ~[na:na]
 	at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1601) ~[na:1.8.0]
 	at java.lang.Thread.run(Thread.java:811) ~[na:2.9 (07-31-2018)]
 Caused by: org.apache.http.conn.HttpHostConnectException: Connect to localhost:23956 [localhost/127.0.0.1] failed: EDC8128I Connection refused. (Connection refused)
 	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:159) ~[httpclient-4.5.6.jar!/:4.5.6]
 	at org.apache.http.impl.conn.PoolingHttpClientConnectionManager.connect(PoolingHttpClientConnectionManager.java:373) ~[httpclient-4.5.6.jar!/:4.5.6]
 	at org.apache.http.impl.execchain.MainClientExec.establishRoute(MainClientExec.java:394) ~[httpclient-4.5.6.jar!/:4.5.6]
 	at org.apache.http.impl.execchain.MainClientExec.execute(MainClientExec.java:237) ~[httpclient-4.5.6.jar!/:4.5.6]
 	at org.apache.http.impl.execchain.ProtocolExec.execute(ProtocolExec.java:185) ~[httpclient-4.5.6.jar!/:4.5.6]
 	at org.apache.http.impl.execchain.RetryExec.execute(RetryExec.java:89) ~[httpclient-4.5.6.jar!/:4.5.6]
 	at org.apache.http.impl.execchain.RedirectExec.execute(RedirectExec.java:110) ~[httpclient-4.5.6.jar!/:4.5.6]
 	at org.apache.http.impl.client.InternalHttpClient.doExecute(InternalHttpClient.java:185) ~[httpclient-4.5.6.jar!/:4.5.6]
 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:83) ~[httpclient-4.5.6.jar!/:4.5.6]
 	at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:56) ~[httpclient-4.5.6.jar!/:4.5.6]
 	at org.springframework.http.client.HttpComponentsClientHttpRequest.executeInternal(HttpComponentsClientHttpRequest.java:87) ~[spring-web-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
 	at org.springframework.http.client.AbstractBufferingClientHttpRequest.executeInternal(AbstractBufferingClientHttpRequest.java:48) ~[spring-web-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
 	at org.springframework.http.client.AbstractClientHttpRequest.execute(AbstractClientHttpRequest.java:53) ~[spring-web-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
 	at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:723) ~[spring-web-5.0.8.RELEASE.jar!/:5.0.8.RELEASE]
 	... 8 common frames omitted
 Caused by: java.net.ConnectException: EDC8128I Connection refused. (Connection refused)
 	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:380) ~[na:1.8.0]
 	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:236) ~[na:1.8.0]
 	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:218) ~[na:1.8.0]
 	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403) ~[na:1.8.0]
 	at java.net.Socket.connect(Socket.java:666) ~[na:1.8.0]
 	at org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(SSLConnectionSocketFactory.java:339) ~[httpclient-4.5.6.jar!/:4.5.6]
 	at org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:142) ~[httpclient-4.5.6.jar!/:4.5.6]
 	... 21 common frames omitted

The repeated message is as it tries to contact other api-defs in addition to discovery api docs etc

[plavjanik]

I have installed this PAX file: https://gizaartifactory.jfrog.io/gizaartifactory/libs-release-local/org/zowe/nightly/zowe-0.9.4-20181126.160305-198.pax on my system with build 91 of APIML. It contains different messages with SSL problems when gateway tries to access API catalog. I need to double check the API catalog template.

[plavjanik]

Fix is in #140