Errors (SourceHandler I/O error: Received fatal alert: unknown_ca) for POD apim-gw-worker

[yhilem]

Hi,

For pattern-2 (https://github.com/wso2/kubernetes-apim/tree/2.1.0/pattern-2), these exception is continously thrown for POD apim-gw-worker :

[2017-09-16 07:15:56,676] INFO - ChannelOpenHandler Connecting to: carbon
[2017-09-16 07:15:56,788] INFO - AndesChannel Channel created (ID: X.X.X.X:47620)
[2017-09-16 07:15:56,965] WARN - JMSUtils Cannot locate destination : throttleData
[2017-09-16 07:15:57,015] INFO - RegistryEventingServiceComponent Successfully Initialized Eventing on Registry
[2017-09-16 07:15:57,164] INFO - QueueDeclareHandler Queue tmp_X.X.X.X_47620_1 bound to default exchange(<>)
[2017-09-16 07:15:57,164] INFO - QueueDeclareHandler Queue tmp_X.X.X.X_47620_1 declared successfully
[2017-09-16 07:15:57,228] INFO - QueueBindHandler Binding queue tmp_X.X.X.X_47620_1 to exchange TopicExchange[amq.topic] with routing key throttleData
[2017-09-16 07:15:57,323] INFO - SubscriptionEngine Local subscription ADDED [throttleData]ID=0@NODEwso2apim-worker-1-2o4mh/X.X.X.X/T=1505546157288/D=false/X=true/O=clientid/E=amq.topic/ET=org.wso2.andes.server.exchange.TopicExchange$1@5f40412b/EUD=0/S=true
[2017-09-16 07:15:57,419] ERROR - SourceHandler I/O error: Received fatal alert: unknown_ca
javax.net.ssl.SSLException: Received fatal alert: unknown_ca
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634)
at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1800)
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1083)
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907)
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
at org.apache.http.nio.reactor.ssl.SSLIOSession.doUnwrap(SSLIOSession.java:245)
at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:280)
at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:410)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:119)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:159)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:338)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:316)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:277)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:105)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:586)
at java.lang.Thread.run(Thread.java:745)
[2017-09-16 07:15:57,542] INFO - JMXServerManager JMX Service URL : service:jmx:rmi://localhost:11111/jndi/rmi://localhost:9999/jmxrmi
[2017-09-16 07:15:57,543] INFO - StartupFinalizerServiceComponent Server : WSO2 API Manager-2.1.0
[2017-09-16 07:15:57,544] INFO - StartupFinalizerServiceComponent WSO2 Carbon started in 111 sec
[2017-09-16 07:15:57,797] INFO - JMSListener Started to listen on destination : throttleData of type topic for listener Siddhi-JMS-Consumer#throttleData
[2017-09-16 07:15:58,412] INFO - CarbonUIServiceComponent Mgt Console URL : https://X.X.X.X:9443/carbon/
[2017-09-16 07:15:58,414] INFO - CarbonUIServiceComponent API Publisher Default Context : https://X.X.X.X:9443/publisher
[2017-09-16 07:15:58,414] INFO - CarbonUIServiceComponent API Store Default Context : https://X.X.X.X:9443/store
[2017-09-16 07:15:58,518] ERROR - SourceHandler I/O error: Received fatal alert: unknown_ca
javax.net.ssl.SSLException: Received fatal alert: unknown_ca
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634)
at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1800)
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1083)
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907)
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
at org.apache.http.nio.reactor.ssl.SSLIOSession.doUnwrap(SSLIOSession.java:245)
at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:280)
at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:410)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:119)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:159)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:338)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:316)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:277)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:105)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:586)
at java.lang.Thread.run(Thread.java:745)
[2017-09-16 07:15:59,075] ERROR - SourceHandler I/O error: Received fatal alert: unknown_ca
javax.net.ssl.SSLException: Received fatal alert: unknown_ca
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634)
at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1800)
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1083)
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907)
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781)
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624)
at org.apache.http.nio.reactor.ssl.SSLIOSession.doUnwrap(SSLIOSession.java:245)
at org.apache.http.nio.reactor.ssl.SSLIOSession.doHandshake(SSLIOSession.java:280)
at org.apache.http.nio.reactor.ssl.SSLIOSession.isAppInputReady(SSLIOSession.java:410)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:119)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:159)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:338)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:316)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:277)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:105)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:586)
at java.lang.Thread.run(Thread.java:745)
[2017-09-16 07:15:59,421] ERROR - SourceHandler I/O error: Received fatal alert: unknown_ca
javax.net.ssl.SSLException: Received fatal alert: unknown_ca
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634)
at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1800)
....
....

To facilitate the analysis, I stopped all other PODs.
As a result, the exchanges that cause this exception are within the only POD that remains: apim-gw-worker

One solution that satisfies us is not to use SSL when exchanges are intra-POD.
Where to apply changes And so quickly fix this problem ?

Apparently, this happens when a pub / subscribe channel is created between the client (gw) and the broker node. But this connection is not ssl :

kubernetes-apim/pattern-2/confs/apim-gw-worker/repository/conf/jndi.properties

Line 19 in 6ebb2b6

connectionfactory.TopicConnectionFactory = amqp://admin:admin@clientid/carbon?brokerlist='tcp://localhost:5672'

Regards,
Youcef HILEM

[yhilem]

Hi,

MeaCulpa;)

This is a configuration error (mountpaths point to the wrong path) on my part which is the cause of this error.

Here is the new log :

Using Java memory options: -Xms256m -Xmx1024m
[2017-09-16 21:47:07,371] INFO - CarbonCoreActivator Starting WSO2 Carbon...
[2017-09-16 21:47:07,375] INFO - CarbonCoreActivator Operating System : Linux 3.10.0-514.21.1.el7.x86_64, amd64
[2017-09-16 21:47:07,376] INFO - CarbonCoreActivator Java Home : /home/o2_adm/java/jre
[2017-09-16 21:47:07,376] INFO - CarbonCoreActivator Java Version : 1.8.0_121
[2017-09-16 21:47:07,376] INFO - CarbonCoreActivator Java VM : Java HotSpot(TM) 64-Bit Server VM 25.121-b13,Oracle Corporation
....
....
[2017-09-16 21:47:07,377] INFO - CarbonCoreActivator User : ?, en-US, Etc/UTC
[2017-09-16 21:47:18,456] INFO - EmbeddedRegistryService Configured Registry in 467ms
[2017-09-16 21:47:18,755] INFO - EmbeddedRegistryService Connected to mount at govregistry in 3ms
[2017-09-16 21:47:19,169] INFO - EmbeddedRegistryService Connected to mount at govregistry in 1ms
[2017-09-16 21:47:19,498] INFO - RegistryCoreServiceComponent Registry Mode : READ-WRITE
[2017-09-16 21:47:20,183] INFO - JmxReporterBuilder Creating JMX reporter for Metrics with domain 'org.wso2.carbon.metrics'
[2017-09-16 21:47:20,188] INFO - JDBCReporterBuilder Creating JDBC reporter for Metrics with source 'wso2apim-worker-1-5aol8', data source 'jdbc/WSO2MetricsDB' and 60 seconds polling period
[2017-09-16 21:47:20,190] INFO - AbstractReporter Started JMX reporter for Metrics
[2017-09-16 21:47:20,196] INFO - AbstractReporter Started JDBC reporter for Metrics
[2017-09-16 21:47:30,225] INFO - SolrClient Default Embedded Solr Server Initialized
[2017-09-16 21:47:30,951] INFO - UserStoreMgtDSComponent Carbon UserStoreMgtDSComponent activated successfully.
[2017-09-16 21:48:02,782] INFO - TaglibUriRule TLD skipped. URI: http://tiles.apache.org/tags-tiles is already defined
[2017-09-16 21:48:04,356] INFO - ClusterBuilder Clustering has been enabled
[2017-09-16 21:48:04,390] INFO - ClusterBuilder Running in application mode
[2017-09-16 21:48:05,483] INFO - UserStoreConfigurationDeployer User Store Configuration Deployer initiated.
[2017-09-16 21:48:05,517] INFO - UserStoreConfigurationDeployer User Store Configuration Deployer initiated.
[2017-09-16 21:48:07,411] INFO - PassThroughHttpSender Initializing Pass-through HTTP/S Sender...
[2017-09-16 21:48:07,426] INFO - PassThroughHttpSender No proxy configuration found
[2017-09-16 21:48:07,553] INFO - PassThroughHttpSender Pass-through HTTP Sender started...
[2017-09-16 21:48:07,554] INFO - PassThroughHttpSSLSender Initializing Pass-through HTTP/S Sender...
[2017-09-16 21:48:07,566] INFO - PassThroughHttpSSLSender No proxy configuration found
[2017-09-16 21:48:07,583] INFO - ClientConnFactoryBuilder HTTPS Loading Identity Keystore from : repository/resources/security/api-mgt.jks
[2017-09-16 21:48:07,596] INFO - ClientConnFactoryBuilder HTTPS Loading Trust Keystore from : repository/resources/security/client-truststore.jks
[2017-09-16 21:48:07,616] INFO - PassThroughHttpSSLSender Pass-through HTTPS Sender started...
[2017-09-16 21:48:07,625] INFO - WebsocketTransportSender WS Sender started
[2017-09-16 21:48:07,659] INFO - PassThroughHttpListener Initializing Pass-through HTTP/S Listener...
[2017-09-16 21:48:07,835] INFO - PassThroughHttpSSLListener Initializing Pass-through HTTP/S Listener...
[2017-09-16 21:48:09,115] INFO - ModuleDeployer Deploying module: addressing-1.6.1-wso2v20 - file:/home/o2_adm/wso2am-2.1.0/repository/deployment/client/modules/addressing-1.6.1-wso2v20.mar
[2017-09-16 21:48:11,039] INFO - DeploymentEngine Deploying Web service: org.wso2.carbon.identity.application.authentication.framework-5.7.5 -
[2017-09-16 21:48:11,269] INFO - DeploymentEngine Deploying Web service: org.wso2.carbon.identity.discovery-5.3.4 -
[2017-09-16 21:48:11,547] INFO - DeploymentEngine Deploying Web service: org.wso2.carbon.identity.webfinger-5.3.4 -
[2017-09-16 21:48:12,314] INFO - DeploymentEngine Deploying Web service: org.wso2.carbon.message.processor-4.6.10 -
[2017-09-16 21:48:12,329] INFO - DeploymentEngine Deploying Web service: org.wso2.carbon.message.store-4.6.10 -
[2017-09-16 21:48:13,287] INFO - DeploymentEngine Deploying Web service: org.wso2.carbon.registry.ws.api-4.6.0 -
[2017-09-16 21:48:14,555] INFO - CarbonServerManager Repository : /home/o2_adm/wso2am-2.1.0/repository/deployment/server/
[2017-09-16 21:48:14,908] INFO - JMSConnectionFactory JMS ConnectionFactory : Siddhi-JMS-Consumer initialized
[2017-09-16 21:48:15,213] INFO - JMSTransportHandler Starting jms topic consumer thread...
[2017-09-16 21:48:15,260] ERROR - DataEndpointConnectionWorker Error while trying to connect to ssl://wso2apim-analytics-1:7712
org.wso2.carbon.databridge.agent.exception.DataEndpointSecurityException: Error while trying to connect to ssl://wso2apim-analytics-1:7712
at org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftSecureClientPoolFactory.createClient(ThriftSecureClientPoolFactory.java:81)
at org.wso2.carbon.databridge.agent.client.AbstractClientPoolFactory.makeObject(AbstractClientPoolFactory.java:39)
at org.apache.commons.pool.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:1212)
at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.connect(DataEndpointConnectionWorker.java:91)
at org.wso2.carbon.databridge.agent.endpoint.DataEndpointConnectionWorker.run(DataEndpointConnectionWorker.java:42)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.thrift.transport.TTransportException: Could not connect to wso2apim-analytics-1 on port 7712
at org.apache.thrift.transport.TSSLTransportFactory.createClient(TSSLTransportFactory.java:237)
at org.apache.thrift.transport.TSSLTransportFactory.getClientSocket(TSSLTransportFactory.java:169)
at org.wso2.carbon.databridge.agent.endpoint.thrift.ThriftSecureClientPoolFactory.createClient(ThriftSecureClientPoolFactory.java:63)
... 9 more
Caused by: java.net.UnknownHostException: wso2apim-analytics-1
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:184)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at sun.security.ssl.SSLSocketImpl.connect(SSLSocketImpl.java:668)
at sun.security.ssl.SSLSocketImpl.(SSLSocketImpl.java:427)
at sun.security.ssl.SSLSocketFactoryImpl.createSocket(SSLSocketFactoryImpl.java:88)
at org.apache.thrift.transport.TSSLTransportFactory.createClient(TSSLTransportFactory.java:233)
... 11 more

............................

The exception is normal because I stopped all other POD.

Thanks,
Youcef HILEM