Opendistro helm chart not getting deployed in IPv6 k8s cluster

[GaneshbabuRamamoorthy]

Hi All,

I tried deploying opendistro helm chart in the IPv6 k8s cluster and I am getting the below responses in pod logs

pod status were showing as running

[root@k8s-rmp-master-0 opendistro-es]$ kubectl get pods -w -n elastic
NAME                                                  READY   STATUS    RESTARTS   AGE
elasticsearch-opendistro-es-client-7fbc9b877-h8jjx    1/1     Running   0          8m18s
elasticsearch-opendistro-es-data-0                    1/1     Running   0          8m18s
elasticsearch-opendistro-es-kibana-5c454cb6bc-k6j4t   1/1     Running   0          8m17s
elasticsearch-opendistro-es-master-0                  1/1     Running   0          8m18s

below is the logs,

data pod logs

[2021-09-30T07:00:19,318][WARN ][o.e.c.c.ClusterFormationFailureHelper] [elasticsearch-opendistro-es-data-0] master not discovered yet: have discovered [{elasticsearch-opendistro-es-data-0}{g9vpjnGhQQSZXm7TlzqHdA}{UNIUsbx3SEaXPduXxEXdIw}{127.0.0.1}{127.0.0.1:9300}{dr}]; discovery will continue using [[fd74:ca9b:3a09:868c:172:18:0:4fce]:9300] from hosts providers and [] from last-known cluster state; node term 0, last-accepted version 0 in term 0
[2021-09-30T07:00:19,550][WARN ][o.e.d.HandshakingTransportAddressConnector] [elasticsearch-opendistro-es-data-0] [connectToRemoteMasterNode[[fd74:ca9b:3a09:868c:172:18:0:4fce]:9300]] completed handshake with [{elasticsearch-opendistro-es-master-0}{OqYEghRrTByIBtH3cdIulQ}{2P7FkFCdRbiAKWKVRlResw}{127.0.0.1}{127.0.0.1:9300}{mr}] but followup connection failed
org.elasticsearch.transport.ConnectTransportException: [elasticsearch-opendistro-es-master-0][127.0.0.1:9300] handshake failed. unexpected remote node {elasticsearch-opendistro-es-data-0}{g9vpjnGhQQSZXm7TlzqHdA}{UNIUsbx3SEaXPduXxEXdIw}{127.0.0.1}{127.0.0.1:9300}{dr}
        at org.elasticsearch.transport.TransportService.lambda$connectionValidator$5(TransportService.java:389) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.ActionListener$4.onResponse(ActionListener.java:157) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.transport.TransportService$5.onResponse(TransportService.java:476) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.transport.TransportService$5.onResponse(TransportService.java:466) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:54) [elasticsearch-7.10.2.jar:7.10.2]
        at com.amazon.opendistroforelasticsearch.security.transport.OpenDistroSecurityInterceptor$RestoringTransportResponseHandler.handleResponse(OpenDistroSecurityInterceptor.java:278) [opendistro_security-1.13.1.0.jar:1.13.1.0]
        at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1171) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:253) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.transport.InboundHandler.lambda$handleResponse$1(InboundHandler.java:247) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:684) [elasticsearch-7.10.2.jar:7.10.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-09-30T07:00:19,565][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [elasticsearch-opendistro-es-data-0] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)
javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)
        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:369) ~[?:?]

master pod logs,

[2021-09-30T06:49:44,180][INFO ][o.e.h.AbstractHttpServerTransport] [elasticsearch-opendistro-es-master-0] publish_address {127.0.0.1:9200}, bound_addresses {[::]:9200}
[2021-09-30T06:49:44,180][INFO ][o.e.n.Node               ] [elasticsearch-opendistro-es-master-0] started
[2021-09-30T06:49:44,181][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [elasticsearch-opendistro-es-master-0] Node started
[2021-09-30T06:49:44,182][INFO ][c.a.o.s.c.ConfigurationRepository] [elasticsearch-opendistro-es-master-0] Will attempt to create index .opendistro_security and default configs if they are absent
[2021-09-30T06:49:44,182][INFO ][c.a.o.s.c.ConfigurationRepository] [elasticsearch-opendistro-es-master-0] Background init thread started. Install default config?: true
[2021-09-30T06:49:44,183][INFO ][c.a.o.s.OpenDistroSecurityPlugin] [elasticsearch-opendistro-es-master-0] 0 Open Distro Security modules loaded so far: []
[2021-09-30T06:49:44,211][INFO ][o.e.g.GatewayService     ] [elasticsearch-opendistro-es-master-0] recovered [0] indices into cluster_state
[2021-09-30T06:49:44,353][INFO ][o.e.c.m.MetadataCreateIndexService] [elasticsearch-opendistro-es-master-0] [.opendistro_security] creating index, cause [api], templates [], shards [1]/[1]
[2021-09-30T06:49:44,371][INFO ][o.e.c.r.a.AllocationService] [elasticsearch-opendistro-es-master-0] Cluster health status changed from [YELLOW] to [RED] (reason: [index [.opendistro_security] created]).
[2021-09-30T06:50:14,471][INFO ][c.a.o.s.c.ConfigurationRepository] [elasticsearch-opendistro-es-master-0] Index .opendistro_security created?: true
[2021-09-30T06:50:14,481][INFO ][c.a.o.s.s.ConfigHelper   ] [elasticsearch-opendistro-es-master-0] Will update 'config' with /usr/share/elasticsearch/plugins/opendistro_security/securityconfig/config.yml and populate it with empty doc if file missing and populateEmptyIfFileMissing=false
[2021-09-30T06:54:43,588][INFO ][c.a.o.j.s.JobSweeper     ] [elasticsearch-opendistro-es-master-0] Running full sweep
[2021-09-30T06:59:43,592][INFO ][c.a.o.j.s.JobSweeper     ] [elasticsearch-opendistro-es-master-0] Running full sweep

client pod logs

[2021-09-30T06:49:36,120][DEPRECATION][o.e.d.c.s.Settings       ] [elasticsearch-opendistro-es-client-7fbc9b877-h8jjx] [node.master] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version.
[2021-09-30T06:49:36,422][INFO ][o.e.b.BootstrapChecks    ] [elasticsearch-opendistro-es-client-7fbc9b877-h8jjx] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2021-09-30T06:49:44,911][WARN ][o.e.d.HandshakingTransportAddressConnector] [elasticsearch-opendistro-es-client-7fbc9b877-h8jjx] [connectToRemoteMasterNode[[fd74:ca9b:3a09:868c:172:18:0:4fce]:9300]] completed handshake with [{elasticsearch-opendistro-es-master-0}{OqYEghRrTByIBtH3cdIulQ}{2P7FkFCdRbiAKWKVRlResw}{127.0.0.1}{127.0.0.1:9300}{mr}] but followup connection failed
org.elasticsearch.transport.ConnectTransportException: [elasticsearch-opendistro-es-master-0][127.0.0.1:9300] handshake failed. unexpected remote node {elasticsearch-opendistro-es-client-7fbc9b877-h8jjx}{FJuEhVeiQ7OrP9re1Zrx5A}{A_lQTVXVSQWFQkQOsfj22A}{127.0.0.1}{127.0.0.1:9300}{ir}
        at org.elasticsearch.transport.TransportService.lambda$connectionValidator$5(TransportService.java:389) ~[elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.ActionListener$4.onResponse(ActionListener.java:157) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.transport.TransportService$5.onResponse(TransportService.java:476) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.transport.TransportService$5.onResponse(TransportService.java:466) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.action.ActionListenerResponseHandler.handleResponse(ActionListenerResponseHandler.java:54) [elasticsearch-7.10.2.jar:7.10.2]
        at com.amazon.opendistroforelasticsearch.security.transport.OpenDistroSecurityInterceptor$RestoringTransportResponseHandler.handleResponse(OpenDistroSecurityInterceptor.java:278) [opendistro_security-1.13.1.0.jar:1.13.1.0]
        at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleResponse(TransportService.java:1171) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.transport.InboundHandler.doHandleResponse(InboundHandler.java:253) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.transport.InboundHandler.lambda$handleResponse$1(InboundHandler.java:247) [elasticsearch-7.10.2.jar:7.10.2]
        at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:684) [elasticsearch-7.10.2.jar:7.10.2]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1130) [?:?]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:630) [?:?]
        at java.lang.Thread.run(Thread.java:832) [?:?]
[2021-09-30T06:49:45,002][ERROR][c.a.o.s.s.t.OpenDistroSecuritySSLNettyTransport] [elasticsearch-opendistro-es-client-7fbc9b877-h8jjx] Exception during establishing a SSL connection: javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)
javax.net.ssl.SSLHandshakeException: Insufficient buffer remaining for AEAD cipher fragment (2). Needs to be more than tag size (16)
        at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:369) ~[?:?]

checked the cluster status inside the master pod and below is the response I got,

[root@elasticsearch-opendistro-es-master-0 elasticsearch]# curl -k -u admin:admin https://elasticsearch-opendistro-es-client-service:9200
Open Distro Security not initialized.[root@elasticsearch-opendistro-es-master-0 elasticsearch]#
[root@elasticsearch-opendistro-es-master-0 elasticsearch]#
[root@elasticsearch-opendistro-es-master-0 elasticsearch]#
[root@elasticsearch-opendistro-es-master-0 elasticsearch]# curl -k -u admin:admin https://elasticsearch-opendistro-es-client-service:9200/_cluster/health?pretty=true
Open Distro Security not initialized.

Attaching the opendistro-es helm which I tried in my IPv6 environment and the same was working in IPv4 k8s cluster.

Does opendistro elasticsearch helm chart not supported in IPv6 k8s cluster?

opendistro-es.zip

Please share your thoughts.

Thanks,
Ganeshbabu R

[tricky10]

I'm having the same issue. 1.13.1 works fine but 1.13.2 and 1.13.3 crash with this error