Cluster: remove hardcoded Cluster port offset #1556

Closed
mattsta opened this Issue Feb 19, 2014 · 14 comments

Comments

Projects
None yet
5 participants
Contributor

mattsta commented Feb 19, 2014

Steps to remove the hard-coded REDIS_CLUSTER_PORT_INCR limit currently requiring Redis always uses a port at least 10,000 less than the highest port number:

  • Covert to zero: listenToPort(server.port+REDIS_CLUSTER_PORT_INCR, ... -> listenToPort(0, ...
  • When cluster mode is initialized with an OS-assigned cluster management port, save the port number to internal server state (server->cluster_port or somesuch).
  • Add cluster_port to output of an INFO section.
  • When other nodes communicate by:
    • connect to the redis instance on the regular Redis port
    • run INFO CLUSTER PORT
    • get the port
    • connect to the cluster port of the server

Reasons for this change: The 10,000 port offset is slightly arbitrary, and disallowing Redis to run on ports higher than 55535 isn't very clear at first.

Reasons against this change: Using a randomly assigned OS port for cluster communication makes firewalling and network permissions difficult. Options: allow a user-specified cluster port (which would still require custom cluster port discovery as above), or use a smaller port offset (next port up instead of 10,000 ports up).

Reasons against a cluster port at all: I haven't looked into it, but I assume muxing cluster communications over the regular Redis port would cause performance problems somewhere.

Owner

antirez commented Mar 10, 2014

Hello Matt, thanks for analyzing the question. I still think the fixed offset is a good idea, you have plenty of opportunities to find a set of port/port+10k that are free, and it makes the Redis Cluster universe a bit simpler: nodes always communicate their primary port that is used everywhere, from cluster nodes, to logging of messages, It is implicit that the binary-chat port for the cluster bus is always at a fixed offset.

If you check your message above, it shows how complexity must be added to implement your proposal. However it is not clear for what gain, since you can run your 2, 20, or 100 cluster instances easily in any reasonable box.

Btw brainstorming is always good!

@antirez antirez closed this Mar 10, 2014

mkralka commented Nov 24, 2015

We're beginning to use Redis Cluster in a mesos+aurora cluster, where ports are assigned to services at random, from a pool of available ports, and registered in zookeeper (which clients use to discover where the nodes are in the cluster).

The cluster_port == port+10000 assumptions causes problems because:

  • port+10000 might be in the pool of ports assigned by mesos to other applications.
  • port+10000 might be in the ephemeral port range used by the OS and currently in use by another application.

I'm working on a patch that will address this problem. Would this patch be of interest for being merged into Redis?

My initial thoughts on how to do this are I bit different than @mattsta suggests, but would look something like:

  • Add a cluster-port configuration option, squirrel that away in struct clusterNode. If not specified, get the default behavior of port+10000.
  • Update CLUSTER MEET to take an optional cluster bus port (becoming CLUSTER MEET ip port [cluster_port])
  • Extend clusterMsgDataGossip in backwards compatible way, to include the cluster bus port (possibly turning notused1 and notused2 into some sort of gossip protocol versioning information and the explicit cluster_port)
  • Populate clusterMsgDataGossip for self indicating that the Redis instance supports an explicit cluster_port (whether or not it's configured to use an explicit/explicit port).
  • If an explicit cluster_port is specified, refused to participate in any cluster that has a node that doesn't support explicit ports. If not specified, participation is OK (since the port is what all nodes expect).

Thoughts?

roboslone commented Jul 29, 2016

you have plenty of opportunities to find a set of port/port+10k that are free, and it makes the Redis Cluster universe a bit simpler

Well, sometimes you just don't. And does that bit really worth it?

This is a real issue, not every environment can guarantee such port set. Please allow to change bus port via config option or at least bus port offset.

This is a stopper for me to deploy Redis Cluster in production environment.

mkralka commented Jul 30, 2016

We have a patch that adds this functionality in a backwards compatible way. It's seems to work, but has not been battle-tested.

If you're not averse to running a patched version of redis, perhaps this will be helpful: tellapart/redis@0f6a94e

@mkralka, that's great! Unfortunately I'm on vacation for the next two weeks and won't be able to test it. But I think I'll be able to send some feedback in the end of the month.

tjxs commented Aug 1, 2016

uh_mm

roboslone notifications@github.com编写:

@mkralka, that's great! Unfortunately I'm on vacation for the next two weeks and won't be able to test it. But I think I'll be able to send some feedback in the end of the month.


You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
#1556 (comment)

roboslone commented Aug 15, 2016

@mkralka , I'm about to start testing your patched version of redis. Is that failed test is something I should worry about?

*** [err]: Test replication partial resync: ok psync (diskless: yes, reconnect: 1) in tests/integration/replication-psync.tcl
Expected condition '[s -1 sync_partial_ok] > 0' to be true ([s -1 sync_partial_ok] > 0)

UPDATE
Looks like #2715, also reproduced in 3.2.3.

roboslone commented Aug 16, 2016

Well, first of all, protected-mode directive doesn't work:

*** FATAL CONFIG FILE ERROR ***
Reading the configuration file, at line 7
>>> 'protected-mode no'
Bad directive or wrong number of arguments

Even if it set to default value:

*** FATAL CONFIG FILE ERROR ***
Reading the configuration file, at line 7
>>> 'protected-mode yes'
Bad directive or wrong number of arguments

And removing protected-mode from config I got segfault on all instances, masters and slaves alike:

=== REDIS BUG REPORT START: Cut & paste starting from here ===
622471:M 16 Aug 12:09:35.057 #     Redis 3.0.5-ta.2 crashed by signal: 11
622471:M 16 Aug 12:09:35.057 #     Failed assertion: <no assertion failed> (<no file>:0)
622471:M 16 Aug 12:09:35.057 # --- STACK TRACE
../../no-fixed-offset/src/redis-server(logStackTrace+0x3e)[0x451bde]
/lib/x86_64-linux-gnu/libc.so.6(+0x148c7d)[0x7f89cc9ffc7d]
/lib/x86_64-linux-gnu/libpthread.so.0(+0xfcb0)[0x7f89ccc84cb0]
/lib/x86_64-linux-gnu/libc.so.6(+0x148c7d)[0x7f89cc9ffc7d]
../../no-fixed-offset/src/redis-server(clusterLoadConfig+0x2b2)[0x4568f2]
../../no-fixed-offset/src/redis-server(clusterInit+0xfd)[0x45998d]
../../no-fixed-offset/src/redis-server(initServer+0x3f4)[0x424334]
../../no-fixed-offset/src/redis-server(main+0x255)[0x41a1b5]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7f89cc8d87ed]
../../no-fixed-offset/src/redis-server[0x41a42d]
622471:M 16 Aug 12:09:35.058 # --- INFO OUTPUT
zsh: segmentation fault  ../../no-fixed-offset/src/redis-server ./redis.conf

Configuration file looks like that:

port 7000
cluster-port 18000
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes
#protected-mode no

UPDATE
Apparently I've been running with old nodes.conf from my earlier tests. Removing nodes.conf and appenonly.aof fixed that.

roboslone commented Aug 16, 2016

Okay, that's what I've got right away:

  • cluster seemed to start correctly, I could see something like 645157:M 16 Aug 12:15:14.049 * The server is now ready to accept connections on port 7000 on every instance
  • then I started redis-cli like that: ./redis-cli -c -p 7000 and tried to store some value:
127.0.0.1:7000> set 'foo' 'bar'
(error) CLUSTERDOWN The cluster is down
  • some commands do work though:
127.0.0.1:7000> ping
PONG
127.0.0.1:7000> dbsize
(integer) 0

At that point I stopped all my instances and tried to start them again and got this:

664407:M 16 Aug 12:20:33.937 # Unrecoverable error: corrupted cluster config file.

nodes.conf looked like that:

6295fe08a8cfb10552133aa250869fd709cf75a9 :0:0 myself,master - 0 0 0 connected
vars currentEpoch 0 lastVoteEpoch 0

And after I've removed nodes.conf and restarted instances I got this:

675729:M 16 Aug 12:23:15.269 # Fatal error loading the DB: Invalid argument. Exiting.

Turned out I forgot to create cluster =/
utils/create-cluster/create-cluster doesn't seem to work with custom cluster-port.

Well, there are two issues right now:

  1. protected-mode directive doesn't work
  2. create-cluster util doesn't work and using nodes.conf from stock redis crashes server

roboslone commented Aug 16, 2016

Trying to create cluster manually. Docs on CLUSTER MEET say:

As a side effect of A knowing and being known by all the other nodes, it will send gossip sections in the heartbeat packets that will allow each other node to create a link with each other one, forming a full mesh in a matter of seconds, even if the cluster is large. Moreover CLUSTER MEET does not need to be reciprocal. If I send the command to A in order to join B, I don't need to also send it to B in order to join A.

So I've manually ran CLUSTER MEET <other-node-host> <other-node-port> for all neighbour nodes on first node. CLUSTER NODES shows that first node knows every other node, but no other node actually knows first one:

127.0.0.1:7000> cluster nodes
3e83a695ef3c77a426d51078d4409390a5614c27 127.0.0.1:7004:7014 master - 0 1471344483825 0 connected
e8424710c8128c5a4dc75699649f30358a155dde 127.0.0.1:7000:7010 myself,master - 0 0 1 connected
88891480710e01c17347d3895a7b01258a2108eb 127.0.0.1:7002:7012 master - 0 1471344483324 0 connected
a71365ae28e7a02a78ad8a3d02c09bdd8b5b3f13 127.0.0.1:7003:7013 master - 0 1471344484827 0 connected
e9ce7763ce24fd0b47c6420a8ab81400cce67e5d 127.0.0.1:7001:7011 master - 0 1471344484827 0 connected
b306d6cae562dbe624a1132206838ddd78b230b4 127.0.0.1:7005:7015 master - 0 1471344483324 0 connected

127.0.0.1:7001> cluster nodes
e9ce7763ce24fd0b47c6420a8ab81400cce67e5d 127.0.0.1:7001:7011 myself,master - 0 0 0 connected

Also they all think they're masters and SLAVEOF doesn't work in cluster mode :( Still trying to find a workaround here.

roboslone commented Aug 16, 2016

I guess something goes wrong at a handshake:

127.0.0.1:7002> cluster nodes
ff4bf369a799f6be15575f114252b49e095d8c4b 127.0.0.1:7000 handshake - 1471346669669 0 0 disconnected
55c58e09f22ac50a7d0ef0ee264aee322f68cdfb :7002:7012 myself,master - 0 0 3 connected 10923-16383
127.0.0.1:7002> cluster nodes
55c58e09f22ac50a7d0ef0ee264aee322f68cdfb :7002:7012 myself,master - 0 0 3 connected 10923-16383

roboslone commented Aug 16, 2016

Well I dirty-hacked redis-trib.rb to use CLUSTER MEET with cluster port:

diff --git a/src/redis-trib.rb b/src/redis-trib.rb
index 068e60d..d205d0e 100755
--- a/src/redis-trib.rb
+++ b/src/redis-trib.rb
@@ -54,12 +54,19 @@ class ClusterNode
            puts "Invalid IP or Port (given as #{addr}) - use IP:Port format"
            exit 1
         end
-        port = s.pop # removes port from split array
+        ports = s.pop.split(",")
+        port = ports[0]
+        if ports.length == 1
+            cluster_port = port + 10000
+        elsif ports.length == 2
+            cluster_port = ports[1]
+        end
         ip = s.join(":") # if s.length > 1 here, it's IPv6, so restore address
         @r = nil
         @info = {}
         @info[:host] = ip
         @info[:port] = port
+        @info[:cluster_port] = cluster_port
         @info[:slots] = {}
         @info[:migrating] = {}
         @info[:importing] = {}
@@ -699,7 +706,13 @@ class RedisTrib
         first = false
         @nodes.each{|n|
             if !first then first = n.info; next; end # Skip the first node
-            n.r.cluster("meet",first[:host],first[:port])
+            redis_cmd = "cluster meet " + first[:host] + " " + first[:port] + " " + first[:cluster_port]
+            xputs redis_cmd
+            shell_cmd = "./redis-cli -c -h " + n.info[:host] + " -p " + n.info[:port] + " " + redis_cmd
+            xputs shell_cmd
+            resp = `#{shell_cmd}`
+            xputs resp
         }
     end

So it kinda works with this call:

./redis-trib.rb create --replicas 1 127.0.0.1:7000,7010 127.0.0.1:7001,7011 127.0.0.1:7002,7012 127.0.0.1:7003,7013 127.0.0.1:7004,7014 127.0.0.1:7005,7015

But handshakes still don't work:

==> 7000
66d9ec06ee522d2cbb7e7eafa29a9491f4abd3a7 127.0.0.1:7000:7010 myself,master - 0 0 1 connected 0-5460
4338ee7008899fa7e1ff5c6d44456e8c218a0b08 127.0.0.1:7005:0 handshake - 1471350478601 0 0 disconnected
276f8f7f9e4fc174f123a8d46521439b13e3b2ee 127.0.0.1:7001:0 handshake - 1471350478701 0 0 disconnected
7a3e470942ad46ada20f011ca59c56db6b5ae1c2 127.0.0.1:7004:0 handshake - 1471350478601 0 0 disconnected
138345777c1f531bdc0f5a3c585592467d82ce3c 127.0.0.1:7003:0 handshake - 1471350478601 0 0 disconnected
71a2985d061ea98b2853d6bd7d47747070e9c23b 127.0.0.1:7002:0 handshake - 1471350478701 0 0 disconnected

==> 7001
9f267eb97576ac3e9f603c481367282280b0e0d2 :7001:7011 myself,master - 0 0 2 connected 5461-10922
66d9ec06ee522d2cbb7e7eafa29a9491f4abd3a7 127.0.0.1:7000:7010 master - 0 1471350622455 1 connected 0-5460

==> 7002
66d9ec06ee522d2cbb7e7eafa29a9491f4abd3a7 127.0.0.1:7000:7010 master - 0 1471350622434 1 connected 0-5460
fa6df0cc7dc633a9211f94071c31119098260d87 :7002:7012 myself,master - 0 0 3 connected 10923-16383

==> 7003
66d9ec06ee522d2cbb7e7eafa29a9491f4abd3a7 127.0.0.1:7000:7010 master - 0 1471350622449 1 connected 0-5460
13efa6d8dcbd10f15cc3f7fb35d077090f07831e :7003:7013 myself,master - 0 0 4 connected

==> 7004
66d9ec06ee522d2cbb7e7eafa29a9491f4abd3a7 127.0.0.1:7000:7010 master - 0 1471350622445 1 connected 0-5460
98e3ec2d700d9a6351cbb495c220505ecde84d3d :7004:7014 myself,master - 0 0 5 connected

==> 7005
21651683e3b3df2c58949055681e902ac6ec4b8c :7005:7015 myself,master - 0 0 6 connected
66d9ec06ee522d2cbb7e7eafa29a9491f4abd3a7 127.0.0.1:7000:7010 master - 0 1471350622454 1 connected 0-5460

UPDATE
Just noticed that cluster port is wrong: 4338ee7008899fa7e1ff5c6d44456e8c218a0b08 127.0.0.1:7005:0, looking into it...

UPDATE 2
From what I see CLUSTER MEET <host> <port> <cluster_port> doesn't seem to work correctly. Sender info is okay:

127.0.0.1:7005> cluster meet 127.0.0.1 7000 7010
OK
127.0.0.1:7005> cluster nodes
4654a568c15667d29895fcd937b4c07db466ec67 127.0.0.1:7000:7010 master - 0 1471352267075 1 connected 0-5460
a66d44efb7348655f262e24dde78e5ccb9d17e82 :7005:7015 myself,master - 0 0 6 connected

But reciever's isn't:

127.0.0.1:7000> cluster nodes
f35208db877d650f7ba321cd576a9adfc465159d 127.0.0.1:7005:0 handshake - 1471352006667 0 0 disconnected
4654a568c15667d29895fcd937b4c07db466ec67 127.0.0.1:7000:7010 myself,master - 0 0 1 connected 0-5460
...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment