Kong 0.12.2 and 0.13.0 fails against DSE 6.0 #3410

mprimeaux · 2018-04-23T19:50:36Z

Summary

kong migrations up --vv fails to complete against Cassandra 4.x (DataStax DSE 6.0).

Steps To Reproduce

Install DataStax DSE 6.0 (Cassandra only).
Install either Kong version 0.12.2 or 0.13.0.
Configure Kong to use Cassandra.
From a terminal on the Kong server run sudo kong migrations up --vv

Additional Details & Logs

Kong version is 0.12.2 (and 0.13.0)
Operating system is Ubuntu 16.04 LTS

The output from bullet 4 follows.

ubuntu@api-gateway-1:~$ sudo kong migrations up --vv
2018/04/23 19:38:53 [verbose] Kong: 0.12.2
2018/04/23 19:38:53 [debug] ngx_lua: 10010
2018/04/23 19:38:53 [debug] nginx: 1011002
2018/04/23 19:38:53 [debug] Lua: LuaJIT 2.1.0-beta3
2018/04/23 19:38:53 [verbose] reading config file at /etc/kong/kong.conf
2018/04/23 19:38:53 [debug] admin_access_log = "logs/admin_access.log"
2018/04/23 19:38:53 [debug] admin_error_log = "logs/error.log"
2018/04/23 19:38:53 [debug] admin_http2 = false
2018/04/23 19:38:53 [debug] admin_listen = "0.0.0.0:8001"
2018/04/23 19:38:53 [debug] admin_listen_ssl = "127.0.0.1:8444"
2018/04/23 19:38:53 [debug] admin_ssl = true
2018/04/23 19:38:53 [debug] anonymous_reports = true
2018/04/23 19:38:53 [debug] cassandra_consistency = "ONE"
2018/04/23 19:38:53 [debug] cassandra_contact_points = {"172.31.34.104"}
2018/04/23 19:38:53 [debug] cassandra_data_centers = {"dc1:2","dc2:3"}
2018/04/23 19:38:53 [debug] cassandra_keyspace = "pulse_api_gateway"
2018/04/23 19:38:53 [debug] cassandra_lb_policy = "RoundRobin"
2018/04/23 19:38:53 [debug] cassandra_password = "******"
2018/04/23 19:38:53 [debug] cassandra_port = 9042
2018/04/23 19:38:53 [debug] cassandra_repl_factor = 1
2018/04/23 19:38:53 [debug] cassandra_repl_strategy = "SimpleStrategy"
2018/04/23 19:38:53 [debug] cassandra_schema_consensus_timeout = 10000
2018/04/23 19:38:53 [debug] cassandra_ssl = false
2018/04/23 19:38:53 [debug] cassandra_ssl_verify = false
2018/04/23 19:38:53 [debug] cassandra_timeout = 5000
2018/04/23 19:38:53 [debug] cassandra_username = "pulse"
2018/04/23 19:38:53 [debug] client_body_buffer_size = "8k"
2018/04/23 19:38:53 [debug] client_max_body_size = "0"
2018/04/23 19:38:53 [debug] client_ssl = false
2018/04/23 19:38:53 [debug] custom_plugins = {}
2018/04/23 19:38:53 [debug] database = "cassandra"
2018/04/23 19:38:53 [debug] db_cache_ttl = 3600
2018/04/23 19:38:53 [debug] db_update_frequency = 5
2018/04/23 19:38:53 [debug] db_update_propagation = 600
2018/04/23 19:38:53 [debug] dns_error_ttl = 1
2018/04/23 19:38:53 [debug] dns_hostsfile = "/etc/hosts"
2018/04/23 19:38:53 [debug] dns_no_sync = false
2018/04/23 19:38:53 [debug] dns_not_found_ttl = 30
2018/04/23 19:38:53 [debug] dns_order = {"LAST","SRV","A","CNAME"}
2018/04/23 19:38:53 [debug] dns_resolver = {}
2018/04/23 19:38:53 [debug] dns_stale_ttl = 4
2018/04/23 19:38:53 [debug] error_default_type = "text/plain"
2018/04/23 19:38:53 [debug] http2 = false
2018/04/23 19:38:53 [debug] latency_tokens = true
2018/04/23 19:38:53 [debug] log_level = "notice"
2018/04/23 19:38:53 [debug] lua_package_cpath = ""
2018/04/23 19:38:53 [debug] lua_package_path = "./?.lua;./?/init.lua;"
2018/04/23 19:38:53 [debug] lua_socket_pool_size = 30
2018/04/23 19:38:53 [debug] lua_ssl_verify_depth = 1
2018/04/23 19:38:53 [debug] mem_cache_size = "128m"
2018/04/23 19:38:53 [debug] nginx_daemon = "on"
2018/04/23 19:38:53 [debug] nginx_optimizations = true
2018/04/23 19:38:53 [debug] nginx_user = "nobody nobody"
2018/04/23 19:38:53 [debug] nginx_worker_processes = "auto"
2018/04/23 19:38:53 [debug] pg_database = "kong"
2018/04/23 19:38:53 [debug] pg_host = "127.0.0.1"
2018/04/23 19:38:53 [debug] pg_port = 5432
2018/04/23 19:38:53 [debug] pg_ssl = false
2018/04/23 19:38:53 [debug] pg_ssl_verify = false
2018/04/23 19:38:53 [debug] pg_user = "kong"
2018/04/23 19:38:53 [debug] prefix = "/usr/local/kong/"
2018/04/23 19:38:53 [debug] proxy_access_log = "logs/access.log"
2018/04/23 19:38:53 [debug] proxy_error_log = "logs/error.log"
2018/04/23 19:38:53 [debug] proxy_listen = "0.0.0.0:8000"
2018/04/23 19:38:53 [debug] proxy_listen_ssl = "0.0.0.0:8443"
2018/04/23 19:38:53 [debug] real_ip_header = "X-Forwarded-For"
2018/04/23 19:38:53 [debug] real_ip_recursive = "off"
2018/04/23 19:38:53 [debug] server_tokens = true
2018/04/23 19:38:53 [debug] ssl = true
2018/04/23 19:38:53 [debug] ssl_cipher_suite = "modern"
2018/04/23 19:38:53 [debug] ssl_ciphers = "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256"
2018/04/23 19:38:53 [debug] trusted_ips = {"172.31.0.0/24"}
2018/04/23 19:38:53 [debug] upstream_keepalive = 60
2018/04/23 19:38:53 [verbose] prefix in use: /usr/local/kong
2018/04/23 19:38:53 [verbose] running datastore migrations
Error:
/usr/local/share/lua/5.1/kong/cmd/migrations.lua:34: [cassandra error] could not retrieve current migrations: [cassandra error] [Invalid] table schema_keyspaces does not exist
stack traceback:
	[C]: in function 'assert'
	/usr/local/share/lua/5.1/kong/cmd/migrations.lua:34: in function 'cmd_exec'
	/usr/local/share/lua/5.1/kong/cmd/init.lua:87: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:87>
	[C]: in function 'xpcall'
	/usr/local/share/lua/5.1/kong/cmd/init.lua:87: in function </usr/local/share/lua/5.1/kong/cmd/init.lua:44>
	/usr/local/bin/kong:7: in function 'file_gen'
	init_worker_by_lua:46: in function <init_worker_by_lua:44>
	[C]: in function 'xpcall'
	init_worker_by_lua:53: in function <init_worker_by_lua:51>
ubuntu@api-gateway-1:~$

Issue Details

The logic problem in line 700 of /kong/dao/db/cassandra.lua is the code accounts for the release version being 3.x and 2.x but unfortunately assumes that if the version is not 3.x then treat it as Cassandra 2.x, which fails since the DSE 6.0 version is now Cassandra 4.x.

Here is the DSE 6.0 cqlsh output.

Connected to Pulse-OLTP-Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 4.0.0.2284 | DSE 6.0.0 | CQL spec 3.4.5 | DSE protocol v2]
Use HELP for help.

The tactical fix is to change line 700 of /kong/dao/db/cassandra.lua from if self.major_version_n == 3 then to if self.major_version_n >= 3 then. This change allows migrations to work as expected.

Longer term, we should consider a more resilient approach.

The text was updated successfully, but these errors were encountered:

thibaultcha · 2018-04-23T19:56:13Z

Hi,

Officially, Kong is not compatible with Cassandra 4.x (it isn't even tested against it). Because of this, the code did not want to make assumptions as to how 4.x will behave/is behaving (there was no guarantee that 4.x would preserve the 3.x schema). I am not sure what the release schedule of 4.x looks like right now, but once we get a chance to investigate it, we'll make the necessary changes for compatibility with it (although we do have quite a lot on our hands right now already, so help would be welcome).

Longer term, we should consider a more resilient approach.

What is your suggestion?

mprimeaux · 2018-04-23T20:00:25Z

Hi. Thanks for replying.

Regarding a possible longer term solution, we might consider abstracting the Cassandra db to select a version-specific provider for schema and similar operations rather than using the current branch logic.

mprimeaux · 2018-04-23T20:02:59Z

I'm more than happy to contribute time for Cassandra 4.x support.

If folks are supportive then I'd like to bracket the solution into a tactical fix for the current blocking issue and then look to spend time on a more considered solution.

thibaultcha · 2018-04-23T20:26:47Z

I am willing to incorporate such a hotfix in the 0.13.1 release which is being prepared right now, if you are willing to contribute it in the next few hours. You can send such a patch against the master branch if you decide to go with it. It'd be appreciated! (We do have a Kong Contributor T-shirt for merged PRs since recently :) )

mprimeaux · 2018-04-23T20:57:06Z

Thank you. I will work on this now.

mprimeaux · 2018-04-23T21:30:34Z

@thibaultcha Do you want me to send the patch to you?

I've read the contribution guidelines and am available to submit a PR as it seems that's the preferred manner based on what I have read in the Submitting a Patch section.

Regardless, I've attached the patch here in case you'd like to review.

If you do prefer the patch then here's my commit message:

fix(dao) tactical change to support Cassandra 4

This change alters the logic for selecting the appropriate Cassandra
schema by assuming Cassandra 3.x and greater adhere to the same 
structure. This is a tactical fix. A more considered design will be 
considered in a future release. 

Fix #3410

Otherwise, I believe I'll need write access to the Kong repository in order to push my branch for PR submission.

Please let me know your preference.

thibaultcha · 2018-04-23T22:48:23Z

The typical GitHub process is for you to fork this repository, and then open a Pull Request from your fork's branch to this remote's master branch.

mprimeaux · 2018-04-23T23:29:08Z

I’ll take care of this now.

thibaultcha · 2018-04-24T00:39:49Z

@mprimeaux No worries, I will apply the patch you provided above.

mprimeaux · 2018-04-24T00:47:27Z

Thanks much.

thibaultcha · 2018-04-24T00:49:35Z

Patched applied to master, thanks!

mprimeaux · 2018-04-24T02:22:01Z

@thibaultcha I’ll now turn my attention to a more considered approach.

thibaultcha closed this as completed in 1d16030 Apr 24, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Kong 0.12.2 and 0.13.0 fails against DSE 6.0 #3410

Kong 0.12.2 and 0.13.0 fails against DSE 6.0 #3410

mprimeaux commented Apr 23, 2018 •

edited

Loading

thibaultcha commented Apr 23, 2018

mprimeaux commented Apr 23, 2018

mprimeaux commented Apr 23, 2018 •

edited

Loading

thibaultcha commented Apr 23, 2018

mprimeaux commented Apr 23, 2018 •

edited

Loading

mprimeaux commented Apr 23, 2018 •

edited

Loading

thibaultcha commented Apr 23, 2018

mprimeaux commented Apr 23, 2018

thibaultcha commented Apr 24, 2018

mprimeaux commented Apr 24, 2018

thibaultcha commented Apr 24, 2018

mprimeaux commented Apr 24, 2018

Kong 0.12.2 and 0.13.0 fails against DSE 6.0 #3410

Kong 0.12.2 and 0.13.0 fails against DSE 6.0 #3410

Comments

mprimeaux commented Apr 23, 2018 • edited Loading

Summary

Steps To Reproduce

Additional Details & Logs

Issue Details

thibaultcha commented Apr 23, 2018

mprimeaux commented Apr 23, 2018

mprimeaux commented Apr 23, 2018 • edited Loading

thibaultcha commented Apr 23, 2018

mprimeaux commented Apr 23, 2018 • edited Loading

mprimeaux commented Apr 23, 2018 • edited Loading

thibaultcha commented Apr 23, 2018

mprimeaux commented Apr 23, 2018

thibaultcha commented Apr 24, 2018

mprimeaux commented Apr 24, 2018

thibaultcha commented Apr 24, 2018

mprimeaux commented Apr 24, 2018

mprimeaux commented Apr 23, 2018 •

edited

Loading

mprimeaux commented Apr 23, 2018 •

edited

Loading

mprimeaux commented Apr 23, 2018 •

edited

Loading

mprimeaux commented Apr 23, 2018 •

edited

Loading