New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Run time configuration reload #321
base: master
Are you sure you want to change the base?
Conversation
+1 |
Here's what happens when I add servers and run
Config that crashed nutcracker: memcached_one:
listen: 127.0.0.1:12335
hash: fnv1a_64
distribution: ketama
auto_eject_hosts: false
servers:
- web321:31545:1 twemproxy_memcached_one.841cae5c-bb4f-11e4-8669-56847afe9799
- web227:31710:1 twemproxy_memcached_one.8bf224cd-bb4f-11e4-8669-56847afe9799
- web363:31002:1 twemproxy_memcached_one.8bf24bde-bb4f-11e4-8669-56847afe9799
- web540:31094:1 twemproxy_memcached_one.8bf24bdf-bb4f-11e4-8669-56847afe9799
- web338:31418:1 twemproxy_memcached_one.8bf272f0-bb4f-11e4-8669-56847afe9799
- web322:31513:1 twemproxy_memcached_one.8bf29a01-bb4f-11e4-8669-56847afe9799
- web172:31592:1 twemproxy_memcached_one.8bf29a02-bb4f-11e4-8669-56847afe9799
- web453:31636:1 twemproxy_memcached_one.8bf2c113-bb4f-11e4-8669-56847afe9799
- web447:31582:1 twemproxy_memcached_one.8bf2e824-bb4f-11e4-8669-56847afe9799
- web397:31048:1 twemproxy_memcached_one.8bf30f35-bb4f-11e4-8669-56847afe9799
- web358:31000:1 twemproxy_memcached_one.8bf33646-bb4f-11e4-8669-56847afe9799
- web13:31755:1 twemproxy_memcached_one.8bf35d57-bb4f-11e4-8669-56847afe9799
- web424:31944:1 twemproxy_memcached_one.8bf35d58-bb4f-11e4-8669-56847afe9799
- web448:31356:1 twemproxy_memcached_one.8bf38469-bb4f-11e4-8669-56847afe9799
- web457:31854:1 twemproxy_memcached_one.8bf3ab7a-bb4f-11e4-8669-56847afe9799
- web300:31940:1 twemproxy_memcached_one.8bf3d28b-bb4f-11e4-8669-56847afe9799
- web305:31128:1 twemproxy_memcached_one.8bf3d28c-bb4f-11e4-8669-56847afe9799
- web385:31837:1 twemproxy_memcached_one.8bf3f99d-bb4f-11e4-8669-56847afe9799
- web324:31160:1 twemproxy_memcached_one.8bf420ae-bb4f-11e4-8669-56847afe9799
- web311:31173:1 twemproxy_memcached_one.8bf420af-bb4f-11e4-8669-56847afe9799
redis_one:
listen: 127.0.0.1:12334
hash: fnv1a_64
distribution: ketama
redis: true
auto_eject_hosts: false
servers:
- web203:31962:1 twemproxy_redis_one.18ac61be-bb47-11e4-8669-56847afe9799 This happens from time to time, not always. |
Well, it actually happens with this config all the time. Steps to reproduce: Run with dummy config: dummy:
listen: /tmp/dummy-listen
servers:
- /tmp/dummy-server:1 Replace it with actual config from previous comment.
|
@bobrik: Fixed, try again! |
Any reason this hasn't been merged in yet? We really need it.. |
+1 |
1 similar comment
+1 |
Due to this issue, and the topology of the stack we've switched to Redis Cluster and implemented specialised clients that support it. All of our problems have gone away and things seem much more reliable now and also faster! (though, to clarify, the stack was app -> local haproxy -> twemproxy -> redis) Feedback: For twemproxy to be a viable option for the corporate infrastructure, it should not only support reloading of the config but also one of the following:
|
Is something holding this up? |
+1 for this patch. Would love to see it merged in so we can run off of the mainline master branch again. |
+1 |
5 similar comments
+1 |
+1 |
+1 |
+1 |
+1 |
+1, anything holding up this patch? |
@andyqzb - can you look into this? |
@manjuraj, let me finish this up. I have a list of things I promised you to do about it. |
any news on this? |
it will be great to have this functionality added... since in the marathon/mesos world the hosts are dynamic... thanks |
Sorry to ask again after all the pings but.. Any news? Really looking forward to see this feature merged! |
+1 Any updates for this PR? |
+1 |
2 similar comments
+1 |
+1 |
hello? is this going to get merged? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM! 👍
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍
+1 |
I'm maintaining a fork of twemproxy and have merged this commit (https://github.com/xginn8/twemproxy). |
Lev Walkin seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account. You have signed the CLA already but the status is still pending? Let us recheck it. |
Is this dynamic config reload option planned to be merged to the main branch? |
This patch introduces a runtime configuration reload.
killall -USR1 nutcracker
would cause a configuration reload.Changes
nc_connection.h
to a more precise enumeration of what a connection is. This also allows significantly more descriptive logging.struct conn
, in a way that is not very conducive to allowing multiple different kinds of entities receiving file system events. This did not work well when I tried to safely pause the statistics gathering events. I restructured that part by removing the part of event loop which deals with statistics and moving it into the statistics thread itself.Model of operation
The new config is read into memory, and the new pools are allocated. If the new pools cannot be allocated for some reasons, the configuration change is not done and the configuration state is rolled back. Once the new pools are allocated, we pause all the ingress data transfers from the clients and drain the output queues. Once the client queues are drained (and we have sent out all the outstanding server responses), we replace the old pools with the newly configured pools. Then we unblock the clients so they can send new requests.
I am ok restructuring this patch however you see fit. Looking forward to your feedback.