Kubernetes Cassandra-Service fails when configured using cassandra docker image

[pranav-patil]

The cassandra non-seed pods fails with the error Unable to gossip with any peers, after cassandra seed pods starts up when deploying kubernetes cassandra-service. Some solutions recommend setting CASSANDRA_LISTEN_ADDRESS in cassandra docker environment variables to pod IP address as in Issue#95 but this does not work. This issue is raised in Cassandra-Jira and claimed to be resolved but yet persists. Refer to the cassandra-service.yaml for kubernetes configuration.

INFO  [HANDSHAKE-cassandra-0.cassandra.default.svc.cluster.local/192.168.232.42] 2019-09-15 01:02:10,508 OutboundTcpConnection.java:561 - Handshaking version with cassandra-0.cassandra.default.svc.cluster.local/192.168.232.42
Exception (java.lang.RuntimeException) encountered during startup: Unable to gossip with any peers
java.lang.RuntimeException: Unable to gossip with any peers
        at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1435)
        at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:566)
        at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:823)
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:683)
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:632)
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:388)
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:620)
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:732)
ERROR [main] 2019-09-15 01:03:11,507 CassandraDaemon.java:749 - Exception encountered during startup
java.lang.RuntimeException: Unable to gossip with any peers
        at org.apache.cassandra.gms.Gossiper.doShadowRound(Gossiper.java:1435) ~[apache-cassandra-3.11.4.jar:3.11.4]
        at org.apache.cassandra.service.StorageService.checkForEndpointCollision(StorageService.java:566) ~[apache-cassandra-3.11.4.jar:3.11.4]
        at org.apache.cassandra.service.StorageService.prepareToJoin(StorageService.java:823) ~[apache-cassandra-3.11.4.jar:3.11.4]
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:683) ~[apache-cassandra-3.11.4.jar:3.11.4]
        at org.apache.cassandra.service.StorageService.initServer(StorageService.java:632) ~[apache-cassandra-3.11.4.jar:3.11.4]
        at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:388) [apache-cassandra-3.11.4.jar:3.11.4]
        at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:620) [apache-cassandra-3.11.4.jar:3.11.4]
        at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:732) [apache-cassandra-3.11.4.jar:3.11.4]
INFO  [StorageServiceShutdownHook] 2019-09-15 01:03:11,806 HintsService.java:209 - Paused hints dispatch
WARN  [StorageServiceShutdownHook] 2019-09-15 01:03:11,807 Gossiper.java:1559 - No local state, state is in silent shutdown, or node hasn't joined, not announcing shutdown
INFO  [StorageServiceShutdownHook] 2019-09-15 01:03:11,812 MessagingService.java:981 - Waiting for messaging service to quiesce
INFO  [ACCEPT-/192.168.41.235] 2019-09-15 01:03:11,813 MessagingService.java:1336 - MessagingService has terminated the accept() thread
INFO  [StorageServiceShutdownHook] 2019-09-15 01:03:16,822 HintsService.java:209 - Paused hints dispatch

[pranav-patil]

Updated cassandra-service to use google-samples kubernetes images in commit cd2eff7 instead of using original cassandra docker images due to peer communication issue between the pods which run on different machines. Downgrading this issue from bug to help wanted.