(title changed) Problems with multiplexing and XREAD/XADD commands from 2 different connections #18
Comments
|
ps: for info, I have a python service which uses 'manual' sharding, and I'm trying to simplify it by using redis-cluster. But the library I'm using now is aioredis, and it does not support redis-cluster. |
|
(FYI here are all the commands sent, when things work, on my mac). I'm going to try to simplify and write a repro. |
|
cluster status, if that helps |
|
If you want to log received queries you can use the
The proxy will print something like this in its log: |
|
Ah awesome let me try that.
Developers never read the doc, I wrote some doc and was complaining yesterday to a user who asking a question and didn't read it :)
… On Jan 16, 2020, at 2:43 AM, artix ***@***.***> wrote:
If you want to log received queries you can use the --dump-queries options (note: it still needs --log-level debug in order to work).
This option will print the received query in the proxy's log, ie:
> set mykey 10
The proxy will print something like this in its log:
[2020-01-16 11:42:40] Req. 0:0:3 ARGV[0]: 'set'
[2020-01-16 11:42:40] Req. 0:0:3 ARGV[1]: 'mykey'
[2020-01-16 11:42:40] Req. 0:0:3 ARGV[2]: '10'
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub <#18>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC2O6UNOFEID3G4V7ISADYDQ6A267ANCNFSM4KHNGB7A>.
|
|
Here's more debugging output. First I 'subscribe to stream' with XREAD BLOCK, and then I write to the stream with XADD. I'm expecting something to come out, but nothing comes. Maybe it's a timing issue. I'll try to add more debugging (logging buffers ?). |
|
@bsergean You should receive the reply. I'll take a look at it, it's probably a bug. I'll take a look at it, could you send me the complete log? |
|
If I restart the redis-proxy before I do my run, I actually have another behavior (which might be easier to diagnose).
I swear I've seen that on my mac before. I'll try to see if I can reproduce it and send you steps if possible. Server logClient log |
|
@bsergean Ok, I've found the issue. Since XREAD is a used in the blocking way, the client that sends the query is removed from the multiplexing context (shared connection) and it privately connects to the cluster. In a "non-proxy" context, a similar situation could even happen when:
In the proxy's context, this effect could be mitigated by using a pool of already-connected connections, so when a client reconnects using a private connection, it could use an already-connected socket and deliver its query in a faster way. Anyway, this would not completely solve your issue, since client A and client B are actually using different connections to the node. I've already planned to implement a connection pool before the final release. |
|
I'm trying to run with only one thread to see if it makes a difference. Now the second client can connect, and we're back to the previous behavior, where the XADD makes it through, but the XREAD does not pick it up. |
|
Cool just read your previous answer. You're right that even in a non proxy way it's possible to run into this problem, but practically I haven't yet. I'll see what I can do to make the XREAD connection wait a bit ; or change the ordering so that the message is published/xadd'ed first, and then we xread it. That order should work fine as we get an answer from xadd when it has completed. |
|
@bsergean Yeah, the reason why it practically never happens when you're not using the proxy is that Redis Cluster Proxy actually reconnects the clients that call the blocking XREAD. So this is what is happening:
The connection pool that I will implement will provide clients that have been removed from the shared connection with a private connection that has its sockets already connected to the nodes of the cluster: this will result in a much faster query execution that would be more similar to what happens when you execute those queries without the proxy. As for now, you can force all clients that must send a blocking XREAD query to be removed from the shared connection before sending XREAD by using the special
This will remove the calling client from the shared connection and will provide a private connection to the client itself. Anyway, try this: You can also disable multiplexing by default for all the clients by launching the proxy with the |
bsergean
changed the title
|
Adding PROXY MULTIPLEXING OFF is slightly involved, so I tried to use the --disable-multiplexing always option. It 'almost' make it works :) The first request still yield a weird disconnection, but then the subsequent request work as expected, which is great. This allowed me to move on to other kind of testing ... Given that this option can trigger behavior changes, I wonder if it should be opt-in instead of opt-out, to ease the adoption of the proxy. |
|
@bsergean The |
|
Ok get it. Thanks for the explanation.
… On Jan 17, 2020, at 8:31 AM, artix ***@***.***> wrote:
@bsergean <https://github.com/bsergean> The --disable-multiplexing always is only for debugging purposes. If multiplexing is always disabled, every client will get its own private connections that could result in a huge amount of open sockets (number of nodes * number of clients).
The best way to handle issues like the one you described is the "connection pool" I was talking about in the previous comment. It will be probably implemented after the first beta is released in the next days.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#18>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC2O6UJCAP24Y5K2PBKOIMDQ6HMORANCNFSM4KHNGB7A>.
|
|
@bsergean I'm implementing the connections pool feature in the connections_pool branch. |
|
Great I'll give it a try. (found a typo in a header file btw: clients with priovate connections). The program that was triggering the problem was an internal healthcheck, and I have changed based on your feedback to do the XADD first, and then the XREAD, so maybe regardless of the connection pool the bug could have disappeared. I have a branch with the old behavior and I'll see if this helps. |
|
Unfortunately I'm still getting the 'hang'... I removed the --disable-multiplexing option ; the connection pool is on by default right ? Full log |
|
@bsergean Yes, the connections pool is enabled by default, but it doesn't seem to be active from what I read in your log. Please ensure that:
If connections pool is active, you should read its size with |
|
Ok I double checked I'm on the right branch, and since I'm using docker I'm pretty sure my build instructions do a clean build. Now I see this message: In the 'incipit'. Similar to when I was using the debugging disable multi-plexing option: it looks like
I wonder if something could be done to fix the 'first ever case', or if this won't be a problem in practice. |
|
@bsergean I need to understand the 'first ever case' scenario and what really happens under the hood. Could you attach a complete log? Since the buffer output is not needed for this kind of issue and logs with
|
|
I'm trying to get you repro steps using cobra, the python server/client I'm working on.
In tha process I was starting/shutting down my redis cluster, while the proxy is still running, and noticed that redis-cli hangs when that happens. (I'm using the connection pool branch)
$ redis-cli -p 7777
127.0.0.1:7777> set foo bra
OK
127.0.0.1:7777>
127.0.0.1:7777>
127.0.0.1:7777> set foo bra
(error) ERR Cluster node disconnected: 127.0.0.1:10006
127.0.0.1:7777>
127.0.0.1:7777>
127.0.0.1:7777> set foo bra
(error) ERR Cluster node disconnected: 127.0.0.1:10006
127.0.0.1:7777>
127.0.0.1:7777>
127.0.0.1:7777> set foo bar
.... here I have restarted the cluster, so I expect the proxy to be able to reach it now.
.... but redis-cli is hanged.
… On Jan 23, 2020, at 7:21 AM, artix ***@***.***> wrote:
@bsergean <https://github.com/bsergean> I need to understand the 'first ever case' scenario and what really happens under the hood. Could you attach a complete log?
Since the buffer output is not needed for this kind of issue and logs with --dump-buffer are usually very big, you can omit --dump-buffer to have a lighter log, ie.
redis-cluster-proxy --log-level debug --dump-queries --dump-queues
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub <#18>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AC2O6UP44QTQ5AJBEBYDS73Q7GYZDANCNFSM4KHNGB7A>.
|
|
@bsergean I made some tests executing the XREAD(blocking)/XADD sequence from different clients. Using the connections_pool branch, the problem is highly mitigated, anyway, with many clients, it sometimes hangs too since the connections pool runs out of available connections. It's possible to increase the pool's size and/or change the "respawn" criteria. By default, the pool's size is 10 connections per thread (created when the proxy starts), the minimum size is 10, the default spawning interval is 50ms and the default spawning rate is 2. You can fine-tune these parameters by using the following options:
So, ie.
It means: "create a pool of 50 connections and start respawning 5 connections every 2ms when connections drop below 40" By fine-tuning those parameters, I finally managed to get all my tests executed without hangs on the blocking XREAD commands. |
|
Thanks a lot for the very detailed explanation. The connection pool work highly mitigates this problem which I hope will help adoption. Based on your input I have changed my scripts to XADD first and then XREAD, which sound way more reliable. |
Hi there,
I'm trying to debug a problem where things work on my mac, but not on Linux/Openshift on the 'real' thing. I have tried to set the log level to debug but I don't see received commands being printed on the console. It looks like this would help quite a bit in troubleshooting.
Thanks !
The text was updated successfully, but these errors were encountered: