Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Periodically refresh cassandra clusters topology #47

Open
wants to merge 2 commits into
base: master
from

Conversation

Projects
None yet
3 participants
@drolando
Copy link
Member

drolando commented Dec 21, 2018

Fixes #46

lua-cassandra doesn't automatically detect changes in the ring topology. You have to manually call cluster:refresh() for it to pick up any change. This is a problem when we're replacing nodes in the C* cluster since the driver will keep trying to connect to the old ones and ignore any new host.

With this change we call refresh() even N seconds, always after the response has been returned.

@drolando drolando requested review from thebostik and curatorman Dec 21, 2018

@drolando

This comment has been minimized.

Copy link
Member Author

drolando commented Dec 21, 2018

cc @thibaultcha: did I understand correctly lua-cassandra's behavior?

@thibaultcha

This comment has been minimized.

Copy link

thibaultcha commented Dec 21, 2018

@drolando Correct, lua-cassandra does not listen for cluster topology events as of today (although contributions would be welcome).

However, I am not sure that refresh() can be called at runtime. Have you tried? Since the refresh process starts by deleting the health boolean and info records from the shm, see here, any query made with an existing load balancing policy would likely fail since next_coordinator depends on those shm values to decide whether a node is healthy and if we can connect to it. I have suggested contributions in that regard in the past already, but have received none as of today.

Have you given a try to your patch in an environment receiving traffic while swapping C* nodes at the same time?

@@ -53,6 +53,9 @@ function caching_handlers._post_request_callback(response, request_info, cacheab
local cache_uri = caching_handlers._get_cache_uri(request_info)
local success, err = xpcall(
function()
-- Refresh cluster topology before writing the result

This comment has been minimized.

Copy link
@thebostik

thebostik Dec 21, 2018

Member

is it not possible to have openresty do this outside of the scope of a request for us?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.