Kong connectivity issue with Cassandra

[venusf1ytrap]

Summary

Kong start fails to start when it tries to connect to Cassandra running inside a container, because its trying to connect to Cassandra on its Private ip (when doing kong start) not on the IP mentioned as a part of the KONG_CASSANDRA_CONTACT_POINTS. Where as kong migrations up works. In my current setup the Cassandra Container is deployed using Mesos/Marathon and the container is exposed using a service port. Same issue can be replicated using just Docker and try connecting to the hostip:hostport where kong fails to start, because it will try to connect on the container ip.

So, i'd like to know why or how Kong is picking up Containers private ip to connect to Cassandra,while this isn't the case during kong migrations up. Any pointers will be greatly appreciated as we're not sure if its Kong or Cassandra specific.

Steps To Reproduce

1. Run Cassandra on a Docker container.

2. _sudo docker run --rm -e "KONG_DATABASE=cassandra" -e "KONG_CASSANDRA_CONTACT_POINTS=172.27.10.x" -e "CASSANDRA_PORT=31035" kong:latest kong migrations up_ [WORKS]

3. docker run -d -e "KONG_DATABASE=cassandra" -e "KONG_CASSANDRA_CONTACT_POINTS=172.27.10.x" -e "KONG_CASSANDRA_PORT=31035" -e "KONG_PROXY_ACCESS_LOG=/dev/stdout" -e "KONG_ADMIN_ACCESS_LOG=/dev/stdout" -e "KONG_PROXY_ERROR_LOG=/dev/stderr" -e "KONG_ADMIN_ERROR_LOG=/dev/stderr" -e "KONG_ADMIN_LISTEN=0.0.0.0:8001, 0.0.0.0:8444 ssl" -p 8000:8000 -p 8443:8443 -p 8001:8001 -p 8444:8444 kong:latest [FAILS]

Additional Details & Logs

• Kong version - 0.14.x
• Kong debug-level startup logs

2018/09/21 11:27:50 [warn] You are using Cassandra but your 'db_update_propagation' setting is set to '0' (default). Due to the distributed nature of Cassandra, you should increase this value.
2018/09/21 11:27:50 [warn] 1#0: [lua] log.lua:63: log(): You are using Cassandra but your 'db_update_propagation' setting is set to '0' (default). Due to the distributed nature of Cassandra, you should increase this value.
nginx: [warn] [lua] log.lua:63: log(): You are using Cassandra but your 'db_update_propagation' setting is set to '0' (default). Due to the distributed nature of Cassandra, you should increase this value.
2018/09/21 11:27:50 [warn] 1#0: [lua] cluster.lua:136: set_peer_down(): [lua-cassandra] setting host at 192.168.1.15 DOWN
nginx: [warn] [lua] cluster.lua:136: set_peer_down(): [lua-cassandra] setting host at 192.168.1.15 DOWN
2018/09/21 11:27:50 [error] 1#0: init_by_lua error: /usr/local/share/lua/5.1/kong/init.lua:200: [cassandra error] could not retrieve current migrations: [cassandra error] all hosts tried for query failed. 192.168.1.15: host seems unhealthy, considering it down (timeout)
stack traceback:
[C]: in function 'assert'
/usr/local/share/lua/5.1/kong/init.lua:200: in function 'init'
init_by_lua:3: in main chunk
nginx: [**error**] init_by_lua error: /usr/local/share/lua/5.1/kong/init.lua:200: [**cassandra error**] could not retrieve current migrations: [**cassandra error**] all hosts tried for query failed. 192.168.1.15: host seems unhealthy, considering it down (timeout)
stack traceback:
[C]: in function 'assert'
/usr/local/share/lua/5.1/kong/init.lua:200: in function 'init'
init_by_lua:3: in main chunk

In the above log, the ip 192.168.1.15 is containers private ip on the Agent running the container and it won't be routable outside the Agent node.

[Tieske]

first thing that comes to mind: #3377

[venusf1ytrap]

Hello Tieske,

Thanks for pointing me to the issue. I briefly went through that GitHub issue and i think its sort of unrelated. The current issue is nothing specific to DNS. What we're trying to figure out is How/Why Kong (kong start) is connecting to Cassandra on its private container IP but not on the supplied KONG_CASSANDRA_CONTACT_POINTS IP address ? Any idea, how Kong is picking up the private ip of the container post Kong migrations up which successfully connects to Cassandra ?

[venusf1ytrap]

Update:

I tried playing around with some Cassandra Parameters like Broadcast_address and Broadcast_rpc_address and set it to my Host / Agent servers IP, but still when kong start happens, i end up getting the same error i.e

could not retrieve current migrations: [**cassandra error**] all hosts tried for query failed. 192.168.1.15: host seems unhealthy, considering it down (timeout).

AFAIK, the above error seems very generic and doesn't point directly to the root cause of the error message. Any pointers to what may be causing this?

[Tieske]

I'm not sure, but iirc the cassandra contact points are used to access the cluster, and then query the cluster for the cassandra nodes in the cluster. So that logic leads me to believe the cassandra nodes return different IP addresses from the contact points provided

Is that correct @thibaultcha ?

[supritshah1289]

@Tieske @thibaultcha I am also getting a similar error.
I am deploying Cassandra and kong in a docker swarm mode. I have two different YAML's 1 for cassandra and another one for Kong and Kong migration.

Cassandra works perfectly fine. After running docker stack deploy kong yaml, it successfully creates kong keyspace in Cassandra docker instance. However, kong fails with

nginx: [error] init_by_lua error: /usr/local/share/lua/5.1/kong/init.lua:200: [cassandra error] could not retrieve current migrations: [cassandra error] all hosts tried for query failed. SERVER_IP_ADDRESS: host seems unhealthy, considering it down (timeout)

I logged into Cassandra container and I have verified kong database in it.

[osamaahf]

Hello,
I have the same error, I am doing a migration from 0.10.3 to 0.14.1, the migration up works fine, but starting kong gave me the same error as @venusf1ytrap describe.
@venusf1ytrap @supritshah1289 did you get any solution for this issue ?

Thanks

[onfla11]

I am facing similar issue on single node setup, kong is coming up fine, however, when status API is invoked along with 200 OK following error is seen.

10.3.50.66 [11/Sep/2019:17:35:49 +0000] "GET /status HTTP/1.1" 200
2019/09/11 17:36:18 [error] 321#0: *8943 [lua] cluster_events.lua:360: [cluster_events] failed to poll: failed to retrieve events from DB: all hosts tried for query failed. 10.30.96.159: host seems unhealthy, considering it down (timeout), context: ngx.timer

Not able to write/read from the cassandra db. Cassandra/Kong both are up and running.

[gszr]

Hey @venusf1ytrap,

Thanks for opening this issue. Is this still valid with newer Kong versions?

Thanks!

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.