New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Redis cluster should support announce-ip & announce-port #2527

Open
kremso opened this Issue Apr 15, 2015 · 81 comments

Comments

Projects
None yet
@kremso

kremso commented Apr 15, 2015

We're running into problems when trying to run redis cluster from docker on different hosts. Dockerized redis node advertises its internal docker address, which is unreachable from other nodes in the cluster. I'm aware of various hacks and workarounds, but I'm proposing that redis cluster supports announce-ip and announce-port configuration options, similarly to redis sentinel.

@kremso

This comment has been minimized.

Show comment
Hide comment
@kremso

kremso Apr 16, 2015

I did try the recommended workaround of specifying an explicit bind, but it fails too. It seems that it won't work without setting IP_TRANSPARENT option on the socket, which redis currently does not.

kremso commented Apr 16, 2015

I did try the recommended workaround of specifying an explicit bind, but it fails too. It seems that it won't work without setting IP_TRANSPARENT option on the socket, which redis currently does not.

@harley84

This comment has been minimized.

Show comment
Hide comment
@harley84

harley84 commented May 7, 2015

Related to #2335

@antirez

This comment has been minimized.

Show comment
Hide comment
@antirez

antirez Jun 3, 2015

Owner

Agreed, will write here a design spec in the next days. p.s. in the meanwhile you may want to prevent Docker from mapping ports, there is a specific feature for 1:1 mapping.

Owner

antirez commented Jun 3, 2015

Agreed, will write here a design spec in the next days. p.s. in the meanwhile you may want to prevent Docker from mapping ports, there is a specific feature for 1:1 mapping.

@KekSfabrik

This comment has been minimized.

Show comment
Hide comment
@KekSfabrik

KekSfabrik Jun 15, 2015

I'm having the same trouble - see #2621
copy/pasta from there:
[...] Master instances know the slaves running on the same machine by local (172.*) IPs and vice versa. This may possibly lead to getting the MOVED response to an IP that is not reachable by the client:

192.168.1.245 / 172.17.42.1 # redis-cli -h 192.168.1.245 -p 6379 cluster nodes
8bd0598380fc05aa374ef22a74da1a691963cfbb 192.168.1.246:6380 slave e1fc77808551cccf5efe6bcbb5816b624bff9633 0 1434352847839 5 connected
5bd7fbc8a4dfe96667dfd668746c100f5829a0e3 192.168.1.247:6380 slave e5337aa24d19a177a6e90bcb1e144eedb436a0c7 0 1434352848039 6 connected
e5337aa24d19a177a6e90bcb1e144eedb436a0c7 192.168.1.246:6379 master - 0 1434352847638 2 connected 5461-10922
e1fc77808551cccf5efe6bcbb5816b624bff9633 192.168.1.245:6379 myself,master - 0 0 1 connected 0-5460
721e4986d1cc2dbf87ac11054242233ac3d67ede 172.17.42.1:6380 slave 5042f162cd5e057df377dd473359972f1bb7e287 0 1434352847139 4 connected
5042f162cd5e057df377dd473359972f1bb7e287 192.168.1.247:6379 master - 0 1434352847839 3 connected 10923-16383
192.168.1.245 / 172.17.42.1 # redis-cli -h 192.168.1.245 -p 6380 cluster nodes
e1fc77808551cccf5efe6bcbb5816b624bff9633 172.17.42.1:6379 master - 0 1434352872207 1 connected 0-5460
8bd0598380fc05aa374ef22a74da1a691963cfbb 192.168.1.246:6380 slave e1fc77808551cccf5efe6bcbb5816b624bff9633 0 1434352872509 5 connected
5042f162cd5e057df377dd473359972f1bb7e287 192.168.1.247:6379 master - 0 1434352872308 3 connected 10923-16383
e5337aa24d19a177a6e90bcb1e144eedb436a0c7 192.168.1.246:6379 master - 0 1434352872108 2 connected 5461-10922
5bd7fbc8a4dfe96667dfd668746c100f5829a0e3 192.168.1.247:6380 slave e5337aa24d19a177a6e90bcb1e144eedb436a0c7 0 1434352872509 6 connected
721e4986d1cc2dbf87ac11054242233ac3d67ede 192.168.1.245:6380 myself,slave 5042f162cd5e057df377dd473359972f1bb7e287 0 1434101988241 4 connected

however i've noticed they all act the same -- the ones on the 192.168.1.246 IP behave exactly the same (for their own IP). Relevant bits as seen by 192.168.1.246:6379:

e5337aa24d19a177a6e90bcb1e144eedb436a0c7 192.168.1.246:6379 myself,master - 0 1434101985463 2 connected 5461-10922
8bd0598380fc05aa374ef22a74da1a691963cfbb 172.17.42.1:6380 slave e1fc77808551cccf5efe6bcbb5816b624bff9633 0 1434375798687 5 connected

So they each have a (machine local) 172.17.* net that is not available from outside the machine

KekSfabrik commented Jun 15, 2015

I'm having the same trouble - see #2621
copy/pasta from there:
[...] Master instances know the slaves running on the same machine by local (172.*) IPs and vice versa. This may possibly lead to getting the MOVED response to an IP that is not reachable by the client:

192.168.1.245 / 172.17.42.1 # redis-cli -h 192.168.1.245 -p 6379 cluster nodes
8bd0598380fc05aa374ef22a74da1a691963cfbb 192.168.1.246:6380 slave e1fc77808551cccf5efe6bcbb5816b624bff9633 0 1434352847839 5 connected
5bd7fbc8a4dfe96667dfd668746c100f5829a0e3 192.168.1.247:6380 slave e5337aa24d19a177a6e90bcb1e144eedb436a0c7 0 1434352848039 6 connected
e5337aa24d19a177a6e90bcb1e144eedb436a0c7 192.168.1.246:6379 master - 0 1434352847638 2 connected 5461-10922
e1fc77808551cccf5efe6bcbb5816b624bff9633 192.168.1.245:6379 myself,master - 0 0 1 connected 0-5460
721e4986d1cc2dbf87ac11054242233ac3d67ede 172.17.42.1:6380 slave 5042f162cd5e057df377dd473359972f1bb7e287 0 1434352847139 4 connected
5042f162cd5e057df377dd473359972f1bb7e287 192.168.1.247:6379 master - 0 1434352847839 3 connected 10923-16383
192.168.1.245 / 172.17.42.1 # redis-cli -h 192.168.1.245 -p 6380 cluster nodes
e1fc77808551cccf5efe6bcbb5816b624bff9633 172.17.42.1:6379 master - 0 1434352872207 1 connected 0-5460
8bd0598380fc05aa374ef22a74da1a691963cfbb 192.168.1.246:6380 slave e1fc77808551cccf5efe6bcbb5816b624bff9633 0 1434352872509 5 connected
5042f162cd5e057df377dd473359972f1bb7e287 192.168.1.247:6379 master - 0 1434352872308 3 connected 10923-16383
e5337aa24d19a177a6e90bcb1e144eedb436a0c7 192.168.1.246:6379 master - 0 1434352872108 2 connected 5461-10922
5bd7fbc8a4dfe96667dfd668746c100f5829a0e3 192.168.1.247:6380 slave e5337aa24d19a177a6e90bcb1e144eedb436a0c7 0 1434352872509 6 connected
721e4986d1cc2dbf87ac11054242233ac3d67ede 192.168.1.245:6380 myself,slave 5042f162cd5e057df377dd473359972f1bb7e287 0 1434101988241 4 connected

however i've noticed they all act the same -- the ones on the 192.168.1.246 IP behave exactly the same (for their own IP). Relevant bits as seen by 192.168.1.246:6379:

e5337aa24d19a177a6e90bcb1e144eedb436a0c7 192.168.1.246:6379 myself,master - 0 1434101985463 2 connected 5461-10922
8bd0598380fc05aa374ef22a74da1a691963cfbb 172.17.42.1:6380 slave e1fc77808551cccf5efe6bcbb5816b624bff9633 0 1434375798687 5 connected

So they each have a (machine local) 172.17.* net that is not available from outside the machine

@drshade

This comment has been minimized.

Show comment
Hide comment
@drshade

drshade Jul 7, 2015

+1 please

drshade commented Jul 7, 2015

+1 please

@Ducatel

This comment has been minimized.

Show comment
Hide comment
@Ducatel

Ducatel Jul 8, 2015

If have the same problem. It isn't possible to create a redis cluster on different servers with docker.
Redis send the container IP instead of the host IP.
So for this Redis should add the support of announce-ip like the sentinel.

+1 please

Ducatel commented Jul 8, 2015

If have the same problem. It isn't possible to create a redis cluster on different servers with docker.
Redis send the container IP instead of the host IP.
So for this Redis should add the support of announce-ip like the sentinel.

+1 please

@bjackson

This comment has been minimized.

Show comment
Hide comment
@bjackson

bjackson Jul 9, 2015

+1
This would be amazing.

bjackson commented Jul 9, 2015

+1
This would be amazing.

@antirez

This comment has been minimized.

Show comment
Hide comment
@antirez

antirez Jul 9, 2015

Owner

Prioritized in order to be shipped with the next release (3.2). Thanks for the feedbacks.

Owner

antirez commented Jul 9, 2015

Prioritized in order to be shipped with the next release (3.2). Thanks for the feedbacks.

@ychrysler

This comment has been minimized.

Show comment
Hide comment
@ychrysler

ychrysler Jul 11, 2015

👍 i there a proposed work-around available for this issue?

ychrysler commented Jul 11, 2015

👍 i there a proposed work-around available for this issue?

@ZhuoRoger

This comment has been minimized.

Show comment
Hide comment
@ZhuoRoger

ZhuoRoger commented Jul 16, 2015

+1

@mistypig

This comment has been minimized.

Show comment
Hide comment
@mistypig

mistypig commented Aug 24, 2015

+1

@devaos

This comment has been minimized.

Show comment
Hide comment
@devaos

devaos commented Sep 12, 2015

+1

@turbospaces

This comment has been minimized.

Show comment
Hide comment

turbospaces commented Sep 29, 2015

+1

@vital-st1x

This comment has been minimized.

Show comment
Hide comment
@vital-st1x

vital-st1x commented Sep 29, 2015

+1

@fab-io

This comment has been minimized.

Show comment
Hide comment
@fab-io

fab-io Sep 30, 2015

As a workaround, if possible in your env, run the containers with the option --net=host (see https://docs.docker.com/articles/networking/#container-networking)

fab-io commented Sep 30, 2015

As a workaround, if possible in your env, run the containers with the option --net=host (see https://docs.docker.com/articles/networking/#container-networking)

@rpannell

This comment has been minimized.

Show comment
Hide comment
@rpannell

rpannell commented Oct 6, 2015

+1

@antirez

This comment has been minimized.

Show comment
Hide comment
@antirez

antirez Oct 15, 2015

Owner

Scheduled for Redis 3.2 that will go in RC in a matter of weeks (~4 hopefully).

Owner

antirez commented Oct 15, 2015

Scheduled for Redis 3.2 that will go in RC in a matter of weeks (~4 hopefully).

@antirez antirez added this to the Redis >= 3.2 milestone Oct 15, 2015

@antirez

This comment has been minimized.

Show comment
Hide comment
@antirez

antirez Oct 15, 2015

Owner

p.s. this is going to be backported into 3.0.x anyway...

Owner

antirez commented Oct 15, 2015

p.s. this is going to be backported into 3.0.x anyway...

@himulawang

This comment has been minimized.

Show comment
Hide comment
@himulawang

himulawang commented Nov 19, 2015

+1

@fatduo

This comment has been minimized.

Show comment
Hide comment
@fatduo

fatduo Dec 9, 2015

Hi @antirez @harley84
Do you have some work-around solutions?
I think you can get the hostname instead of ip address. In docker container, I can pass hostname env to the container.

fatduo commented Dec 9, 2015

Hi @antirez @harley84
Do you have some work-around solutions?
I think you can get the hostname instead of ip address. In docker container, I can pass hostname env to the container.

@MrMMorris

This comment has been minimized.

Show comment
Hide comment
@MrMMorris

MrMMorris Dec 9, 2015

@fatduo the workaround is using --net=host as mentioned here: #2527 (comment)

MrMMorris commented Dec 9, 2015

@fatduo the workaround is using --net=host as mentioned here: #2527 (comment)

@srinikalyanaraman

This comment has been minimized.

Show comment
Hide comment
@srinikalyanaraman

srinikalyanaraman Dec 16, 2015

Hi @antirez @harley84
I have put in a fix in a fork and opened a PR (as per the CONTRIBUTING document). I have tested them in kubernetes on openstack. Please review. Thanks!!

srinikalyanaraman commented Dec 16, 2015

Hi @antirez @harley84
I have put in a fix in a fork and opened a PR (as per the CONTRIBUTING document). I have tested them in kubernetes on openstack. Please review. Thanks!!

@antirez

This comment has been minimized.

Show comment
Hide comment
@antirez

antirez Dec 23, 2015

Owner

I'm going to release RC1 without this feature, to add it in the course of the next RC so that we'll have it. However there is some design effort to do about it. It's not a matter of writing an implementation, but to understand how to implement it. For instance, the implementation provided by @srinikalyanaraman breaks binary compatibility with the Redis Cluster bus since it modifies the header structure without upgrading the version, or handling the older version format. Also the final implementation will allow to remap both the IP and port. So thanks for providing a PR, but the issue was marked as design needed because there were a few problems to fix. This is why I cannot merge the PR.

So, if there is to do a design effort, let's do the design effort:

Design proposal

In order to make the new implementation binary compatible with the past, it is possible, in theory, to use the unused field in the message structure. It's 32 bytes so enough to put inside the address (including the IPv6 address) in binary form. It's not a great solution however... IPs travel in cleartext and normalized in the rest of the protocol, and we have already the conversion functions in place.

If there is to break binary compatibility, we'll have to upgrade the protocol version in the header, so that older cluster implementations will discard the packets. Instead in 3.2 we'll have to handle the older version of the protocol as well, which is not great, but is needed.

In short:

  1. We add a new field in the header, containing the announced IP.
  2. We don't add a new port field, we already announce it currently. We'll just overwrite this field if a different port is announced as well.
  3. We upgrade the version of the protocol CLUSTER_PROTO_VER from 0 to 1.
  4. We make sure that when IPs are not remapped, the new field containing the announced IP is all zeroed. It means: auto detect it, dear receiver. Like 3.0 is doing by default.
  5. The new field will be called just myip. Without references to the fact is an announced one. Simply if it's zeroed, we ask for auto detection to the receiver.

Now in order to handle the older version of the protocol in 3.2, the simplest thing to do is the following:

  1. We read the packet as usually.
  2. In clusterProcessPacket() what we do is to check the version of the protocol. If it's 0, we know we read an older protocol with different offsets because of the missing field. But fortunately we can just memmove to fix it all, and memset the new "hole" left with zeroes. And this will make the older protocol compatible with the newer protocol.
  3. The unused field we already have, of 32 bytes, will be left as it is. We'll need it soon or later, as this issue shows. Unfortunately this time it was short, but 32 bytes are a lot, normally.

Exposed configuration options:

cluster-announce-ip
cluster-announce-port

Both can be provided independently.

Implementation

Given that it's very little, but very sensible code, I'll implement it myself so PRs will not be accepted in this context since to proof-read them is more work than writing it from scratch with the above proposal in mind. Instead what would be extremely valuable is ideas and analysis of the proposal in order to identify if it's good enough or not.

Documentation

The features must be documented in:

  1. The cluster tutorial.
  2. The example redis.conf file.
Owner

antirez commented Dec 23, 2015

I'm going to release RC1 without this feature, to add it in the course of the next RC so that we'll have it. However there is some design effort to do about it. It's not a matter of writing an implementation, but to understand how to implement it. For instance, the implementation provided by @srinikalyanaraman breaks binary compatibility with the Redis Cluster bus since it modifies the header structure without upgrading the version, or handling the older version format. Also the final implementation will allow to remap both the IP and port. So thanks for providing a PR, but the issue was marked as design needed because there were a few problems to fix. This is why I cannot merge the PR.

So, if there is to do a design effort, let's do the design effort:

Design proposal

In order to make the new implementation binary compatible with the past, it is possible, in theory, to use the unused field in the message structure. It's 32 bytes so enough to put inside the address (including the IPv6 address) in binary form. It's not a great solution however... IPs travel in cleartext and normalized in the rest of the protocol, and we have already the conversion functions in place.

If there is to break binary compatibility, we'll have to upgrade the protocol version in the header, so that older cluster implementations will discard the packets. Instead in 3.2 we'll have to handle the older version of the protocol as well, which is not great, but is needed.

In short:

  1. We add a new field in the header, containing the announced IP.
  2. We don't add a new port field, we already announce it currently. We'll just overwrite this field if a different port is announced as well.
  3. We upgrade the version of the protocol CLUSTER_PROTO_VER from 0 to 1.
  4. We make sure that when IPs are not remapped, the new field containing the announced IP is all zeroed. It means: auto detect it, dear receiver. Like 3.0 is doing by default.
  5. The new field will be called just myip. Without references to the fact is an announced one. Simply if it's zeroed, we ask for auto detection to the receiver.

Now in order to handle the older version of the protocol in 3.2, the simplest thing to do is the following:

  1. We read the packet as usually.
  2. In clusterProcessPacket() what we do is to check the version of the protocol. If it's 0, we know we read an older protocol with different offsets because of the missing field. But fortunately we can just memmove to fix it all, and memset the new "hole" left with zeroes. And this will make the older protocol compatible with the newer protocol.
  3. The unused field we already have, of 32 bytes, will be left as it is. We'll need it soon or later, as this issue shows. Unfortunately this time it was short, but 32 bytes are a lot, normally.

Exposed configuration options:

cluster-announce-ip
cluster-announce-port

Both can be provided independently.

Implementation

Given that it's very little, but very sensible code, I'll implement it myself so PRs will not be accepted in this context since to proof-read them is more work than writing it from scratch with the above proposal in mind. Instead what would be extremely valuable is ideas and analysis of the proposal in order to identify if it's good enough or not.

Documentation

The features must be documented in:

  1. The cluster tutorial.
  2. The example redis.conf file.
@allanwax

This comment has been minimized.

Show comment
Hide comment
@allanwax

allanwax Dec 28, 2015

@antirez any thought to having cluster-announce accept multiple addresses as failovers in case one of the 'channels' goes bad?

allanwax commented Dec 28, 2015

@antirez any thought to having cluster-announce accept multiple addresses as failovers in case one of the 'channels' goes bad?

@allanwax

This comment has been minimized.

Show comment
Hide comment
@allanwax

allanwax commented Dec 28, 2015

SEE also #2109

@karthimohan

This comment has been minimized.

Show comment
Hide comment
@karthimohan

karthimohan Jan 5, 2016

@antirez Could you please let us know when this will feature be available?

karthimohan commented Jan 5, 2016

@antirez Could you please let us know when this will feature be available?

@antirez

This comment has been minimized.

Show comment
Hide comment
@antirez

antirez Jan 19, 2016

Owner

Hello folks, I'm working at this stuff right now and I'm 90% done, and I've a few bad news.

  1. I don't think there is a simple way to make this backward compatible at binary level. I tried but it adds a lot of complexity, so basically this feature will not back ported to 3.0, will be 3.2 only, and when upgrading a cluster there will be to restart all the nodes with 3.2, since 3.0 <-> 3.2 chat will be impossible.
  2. I'm also implementing cluster-announce-port, since it's the most useful thing with Docker, for instance. However the problem is that we have a base port, and a cluster bus port, and I don't believe they are guaranteed to have the offset of 10000 after remapping. So this means that cluster-announce-port will have to take two arguments, not just one: the base port and the bus port. Not just that, the header will also have an additional field to announce the bus port.

Does the above makes sense? I'm not finalizing the implementation until I get feedbacks. Thanks.

Owner

antirez commented Jan 19, 2016

Hello folks, I'm working at this stuff right now and I'm 90% done, and I've a few bad news.

  1. I don't think there is a simple way to make this backward compatible at binary level. I tried but it adds a lot of complexity, so basically this feature will not back ported to 3.0, will be 3.2 only, and when upgrading a cluster there will be to restart all the nodes with 3.2, since 3.0 <-> 3.2 chat will be impossible.
  2. I'm also implementing cluster-announce-port, since it's the most useful thing with Docker, for instance. However the problem is that we have a base port, and a cluster bus port, and I don't believe they are guaranteed to have the offset of 10000 after remapping. So this means that cluster-announce-port will have to take two arguments, not just one: the base port and the bus port. Not just that, the header will also have an additional field to announce the bus port.

Does the above makes sense? I'm not finalizing the implementation until I get feedbacks. Thanks.

@kremso

This comment has been minimized.

Show comment
Hide comment
@kremso

kremso Jan 20, 2016

@antirez Makes sense to me.

kremso commented Jan 20, 2016

@antirez Makes sense to me.

@drshade

This comment has been minimized.

Show comment
Hide comment
@drshade

drshade Jan 21, 2016

Sounds reasonable. You are correct the bus port will no longer be a
guaranteed offset from the base port so making that specifiable is good.
Looking forward to testing this when you are ready.
On Thu, 21 Jan 2016 at 01:04 Tomáš Kramár notifications@github.com wrote:

@antirez https://github.com/antirez Makes sense to me.


Reply to this email directly or view it on GitHub
#2527 (comment).

drshade commented Jan 21, 2016

Sounds reasonable. You are correct the bus port will no longer be a
guaranteed offset from the base port so making that specifiable is good.
Looking forward to testing this when you are ready.
On Thu, 21 Jan 2016 at 01:04 Tomáš Kramár notifications@github.com wrote:

@antirez https://github.com/antirez Makes sense to me.


Reply to this email directly or view it on GitHub
#2527 (comment).

@jamespedwards42

This comment has been minimized.

Show comment
Hide comment
@jamespedwards42

jamespedwards42 Feb 10, 2016

Contributor

Would it be possible to also add 'client-cluster-announce-port/ip' configurations? This would be useful in scenarios where clients are tunneling in to the redis cluster. That way 'cluster-announce-port/ip' would expose private network ip's and 'client-cluster-announce-port/ip' would expose something like '127.0.0.1:7443' or however the client should have the local tunnel mapping configured.

Contributor

jamespedwards42 commented Feb 10, 2016

Would it be possible to also add 'client-cluster-announce-port/ip' configurations? This would be useful in scenarios where clients are tunneling in to the redis cluster. That way 'cluster-announce-port/ip' would expose private network ip's and 'client-cluster-announce-port/ip' would expose something like '127.0.0.1:7443' or however the client should have the local tunnel mapping configured.

@ramonsnir

This comment has been minimized.

Show comment
Hide comment
@ramonsnir

ramonsnir Feb 23, 2016

Contributor

@antirez Makes sense.

Contributor

ramonsnir commented Feb 23, 2016

@antirez Makes sense.

@korvus81

This comment has been minimized.

Show comment
Hide comment
@korvus81

korvus81 Mar 22, 2016

@antirez That implementation sounds good to me. Is this still planned for 3.2?

korvus81 commented Mar 22, 2016

@antirez That implementation sounds good to me. Is this still planned for 3.2?

@jessee9

This comment has been minimized.

Show comment
Hide comment
@jessee9

jessee9 commented May 12, 2016

+1

@coryjamesfisher

This comment has been minimized.

Show comment
Hide comment
@coryjamesfisher

coryjamesfisher May 17, 2016

@ovanes
Looks like this is the pull request for the merge to 3.2 (has conflicts that need resolution)
#3230

coryjamesfisher commented May 17, 2016

@ovanes
Looks like this is the pull request for the merge to 3.2 (has conflicts that need resolution)
#3230

@jamespedwards42

This comment has been minimized.

Show comment
Hide comment
@jamespedwards42

jamespedwards42 Jul 11, 2016

Contributor

With the new Swarm integration for Docker, it is possible to use an overlay network so that each container task will use that same network rather than a completely isolated internal network. Each Redis node will be reachable from the same IP that it will naturally announce. Here is a complete walkthrough for using it: jamespedwards42.github.io/2016/06/21/docker-swarm-redis-cluster/.

However, if cluster-announce-ip supported hostnames, this would be much easier because you wouldn't have to hack/parse out the IP address for each container service on the overlay network.

Contributor

jamespedwards42 commented Jul 11, 2016

With the new Swarm integration for Docker, it is possible to use an overlay network so that each container task will use that same network rather than a completely isolated internal network. Each Redis node will be reachable from the same IP that it will naturally announce. Here is a complete walkthrough for using it: jamespedwards42.github.io/2016/06/21/docker-swarm-redis-cluster/.

However, if cluster-announce-ip supported hostnames, this would be much easier because you wouldn't have to hack/parse out the IP address for each container service on the overlay network.

@ovanes

This comment has been minimized.

Show comment
Hide comment
@ovanes

ovanes Jul 11, 2016

@jamespedwards42 It's not only about docker/swarm :( I use Kubernetes and I am sure there are other technologies in field which have proprietary networking layers and need this features.

ovanes commented Jul 11, 2016

@jamespedwards42 It's not only about docker/swarm :( I use Kubernetes and I am sure there are other technologies in field which have proprietary networking layers and need this features.

@jamespedwards42

This comment has been minimized.

Show comment
Hide comment
@jamespedwards42

jamespedwards42 Jul 11, 2016

Contributor

@ovanes Yes, completely understand/agree. I also wanted to point out the pain that is caused by not supporting hostnames. Would that make life any easier in the Kubernetes world?

Contributor

jamespedwards42 commented Jul 11, 2016

@ovanes Yes, completely understand/agree. I also wanted to point out the pain that is caused by not supporting hostnames. Would that make life any easier in the Kubernetes world?

@ramonsnir

This comment has been minimized.

Show comment
Hide comment
@ramonsnir

ramonsnir Jul 11, 2016

Contributor

This also happens when doing cross-data-center work, where each data center has its own internal network. VPNs/IPSecs are expensive to maintain (and keep HA), and slow stuff down.

Contributor

ramonsnir commented Jul 11, 2016

This also happens when doing cross-data-center work, where each data center has its own internal network. VPNs/IPSecs are expensive to maintain (and keep HA), and slow stuff down.

@andrewmichaelsmith

This comment has been minimized.

Show comment
Hide comment
@andrewmichaelsmith

andrewmichaelsmith Jul 11, 2016

@jamespedwards42 Can confirm that a very common method for service discovery in Kubernetes land is hostnames.

andrewmichaelsmith commented Jul 11, 2016

@jamespedwards42 Can confirm that a very common method for service discovery in Kubernetes land is hostnames.

@ovanes

This comment has been minimized.

Show comment
Hide comment
@ovanes

ovanes Jul 12, 2016

Well, in Kubernetes a pod can't be trivially addressed via a name. Only a service can select using the selectors to which pod to direct the data. Thus having N nodes requires one to create N Kubernetes services, which is not maintainable. The approach here is to retrieve the pod IP (of internal Kubernetes network) and announce it. If pod is rescheduled the startup script will re-determine the IP.

ovanes commented Jul 12, 2016

Well, in Kubernetes a pod can't be trivially addressed via a name. Only a service can select using the selectors to which pod to direct the data. Thus having N nodes requires one to create N Kubernetes services, which is not maintainable. The approach here is to retrieve the pod IP (of internal Kubernetes network) and announce it. If pod is rescheduled the startup script will re-determine the IP.

@jamespedwards42

This comment has been minimized.

Show comment
Hide comment
@jamespedwards42

jamespedwards42 Jul 12, 2016

Contributor

@ovanes I like that approach as well, but I was worried that changing the IP of an existing node would mess things up. Have you had a chance to test this out against unstable?

Ultimately what kept me from using that approach is that I want to use network attached storage. With Swarm, volumes are configured at the service level. Not sure about all of the cloud platforms, but GCE only allows a drive to be attached to a single server for writes.

Just general curiosity, do you plan to use Slave nodes as well, or just rely on Kubernetes re-scheduling in conjunction with an NFS? I keep going back and forth on this because failover will probably be faster with Slave nodes depending on the size of your database.

Apologies if this is spamming up this thread, but I'm hoping as a community we can come up with a best-practice for managing Redis Cluster with container orchestration.

Contributor

jamespedwards42 commented Jul 12, 2016

@ovanes I like that approach as well, but I was worried that changing the IP of an existing node would mess things up. Have you had a chance to test this out against unstable?

Ultimately what kept me from using that approach is that I want to use network attached storage. With Swarm, volumes are configured at the service level. Not sure about all of the cloud platforms, but GCE only allows a drive to be attached to a single server for writes.

Just general curiosity, do you plan to use Slave nodes as well, or just rely on Kubernetes re-scheduling in conjunction with an NFS? I keep going back and forth on this because failover will probably be faster with Slave nodes depending on the size of your database.

Apologies if this is spamming up this thread, but I'm hoping as a community we can come up with a best-practice for managing Redis Cluster with container orchestration.

@ovanes

This comment has been minimized.

Show comment
Hide comment
@ovanes

ovanes Jul 12, 2016

@jamespedwards42

but I was worried that changing the IP of an existing node would mess things up. Have you had a chance to test this out against unstable?

Yes. I did and it works pretty well. I scheduled 3 masters and each master had 3 slaves. Unfortunately, this script is proprietary work, that's why I can't disclose it.

But in general, Kubernetes has a nice feature, you can enforce a physical port to be allocated on the Host node. Using that, gives you a very simple way to ensure that each master and its slaves run on the same physical port. Forcing them to use that port on the underlying host node, will prevent Kubernetes to schedule the same master and it's slaves on the same node :) and give you a failover distribution.

Finally, from time to time I schedule to run a pod which mounts S3 bucket using s3fs (you can ask Redis to run always as slave). This node just gets the entire-db from one of the distributed master/slaves and shuts downs/sleeps again.

In Redis one can use a config to acknowledge the write request only if one or multiple slaves have also acknowledged it. IMO, you don't need NFS etc, but just analyse number of potential failures and introduce the right amount of slaves with ephemeral local storage, which you don't care about :) As you ensure that most of the infrastructure runs and just in case you do S3 backups.

This setup also works because Kubernetes features a replication controller, where you can setup that you want to have at least N instances of particular pod to be running. This can ensure that you have at least one master and two slaves running on your infrastructure. There are some edge cases, where the entire cluster can become so busy that a new/restarted POD is not going to be started and just hanging in the distributed scheduler's queue for startup, but again this issue can be solved, by labelling the physical node. These labels can than be applied to control which POD are allowed to be run on which nodes.

ovanes commented Jul 12, 2016

@jamespedwards42

but I was worried that changing the IP of an existing node would mess things up. Have you had a chance to test this out against unstable?

Yes. I did and it works pretty well. I scheduled 3 masters and each master had 3 slaves. Unfortunately, this script is proprietary work, that's why I can't disclose it.

But in general, Kubernetes has a nice feature, you can enforce a physical port to be allocated on the Host node. Using that, gives you a very simple way to ensure that each master and its slaves run on the same physical port. Forcing them to use that port on the underlying host node, will prevent Kubernetes to schedule the same master and it's slaves on the same node :) and give you a failover distribution.

Finally, from time to time I schedule to run a pod which mounts S3 bucket using s3fs (you can ask Redis to run always as slave). This node just gets the entire-db from one of the distributed master/slaves and shuts downs/sleeps again.

In Redis one can use a config to acknowledge the write request only if one or multiple slaves have also acknowledged it. IMO, you don't need NFS etc, but just analyse number of potential failures and introduce the right amount of slaves with ephemeral local storage, which you don't care about :) As you ensure that most of the infrastructure runs and just in case you do S3 backups.

This setup also works because Kubernetes features a replication controller, where you can setup that you want to have at least N instances of particular pod to be running. This can ensure that you have at least one master and two slaves running on your infrastructure. There are some edge cases, where the entire cluster can become so busy that a new/restarted POD is not going to be started and just hanging in the distributed scheduler's queue for startup, but again this issue can be solved, by labelling the physical node. These labels can than be applied to control which POD are allowed to be run on which nodes.

@laurie71

This comment has been minimized.

Show comment
Hide comment
@laurie71

laurie71 Jul 13, 2016

This proposal sounds like it would solve the problem for a native Docker configuration, but I don't think it would be sufficient when using docker-machine, as the ports exposed by the Docker image get remapped by the docker machine virtual host. Am I missing something? Is there some way the correct IP/port numbers could be discovered and configured to be the advertised host/port?

laurie71 commented Jul 13, 2016

This proposal sounds like it would solve the problem for a native Docker configuration, but I don't think it would be sufficient when using docker-machine, as the ports exposed by the Docker image get remapped by the docker machine virtual host. Am I missing something? Is there some way the correct IP/port numbers could be discovered and configured to be the advertised host/port?

@msch3ung

This comment has been minimized.

Show comment
Hide comment
@msch3ung

msch3ung Aug 18, 2016

Does anyone know if there are any updates on this? 3.2 went out and it looks like the cluster-announce-ip is not a valid configuration option.

msch3ung commented Aug 18, 2016

Does anyone know if there are any updates on this? 3.2 went out and it looks like the cluster-announce-ip is not a valid configuration option.

@antirez

This comment has been minimized.

Show comment
Hide comment
@antirez

antirez Aug 25, 2016

Owner

This proposal is implemented in Redis unstable branch. The first stable version of Redis to support that will be 4.0, released as RC1 15th of October 2016. Anyone testing this support with success in testing environments? Feedbacks appreciated.

Owner

antirez commented Aug 25, 2016

This proposal is implemented in Redis unstable branch. The first stable version of Redis to support that will be 4.0, released as RC1 15th of October 2016. Anyone testing this support with success in testing environments? Feedbacks appreciated.

@ovanes

This comment has been minimized.

Show comment
Hide comment
@ovanes

ovanes Aug 25, 2016

I tested it in our Kubernetes cluster deployment and it worked. It was last time 2 months ago. Worked just fine, and I was able to run a cluster with docker packaged Redis instances.

Also intentionally crashing instances and letting Kubernetes reschedule the entire container worked fine with announcing completely new ip and re-electing a new master. I might be able to retest this starting in september, but can't promise it right now.

ovanes commented Aug 25, 2016

I tested it in our Kubernetes cluster deployment and it worked. It was last time 2 months ago. Worked just fine, and I was able to run a cluster with docker packaged Redis instances.

Also intentionally crashing instances and letting Kubernetes reschedule the entire container worked fine with announcing completely new ip and re-electing a new master. I might be able to retest this starting in september, but can't promise it right now.

@antirez

This comment has been minimized.

Show comment
Hide comment
@antirez

antirez Aug 25, 2016

Owner

@ovanes this is a very useful info, thanks!

Owner

antirez commented Aug 25, 2016

@ovanes this is a very useful info, thanks!

@mostolog

This comment has been minimized.

Show comment
Hide comment
@mostolog

mostolog Aug 29, 2016

Hi.

Probably this comment doesn't match exactly the topic, but I hope is enough close-related.

AFAIK, redis requires different node.conf cluster files and ports.
For that reason, when using docker, entrypoint scripts, variable substitution...are required.

This seems quite incompatible with new docker swarm mode and I was wondering if a gossip protocol, multicast or something else instead of static configurations could be used to create/handle the cluster on an easier way. (ie: the way elasticsearch does).
Is this what you have planned for Redis 4.0?

If I understood correctly, this is an example of such trickyness https://jamespedwards42.github.io/2016/06/21/docker-swarm-redis-cluster/

Regards

mostolog commented Aug 29, 2016

Hi.

Probably this comment doesn't match exactly the topic, but I hope is enough close-related.

AFAIK, redis requires different node.conf cluster files and ports.
For that reason, when using docker, entrypoint scripts, variable substitution...are required.

This seems quite incompatible with new docker swarm mode and I was wondering if a gossip protocol, multicast or something else instead of static configurations could be used to create/handle the cluster on an easier way. (ie: the way elasticsearch does).
Is this what you have planned for Redis 4.0?

If I understood correctly, this is an example of such trickyness https://jamespedwards42.github.io/2016/06/21/docker-swarm-redis-cluster/

Regards

@ovanes

This comment has been minimized.

Show comment
Hide comment
@ovanes

ovanes Aug 29, 2016

@mostolog Can you explain how elastic search does this and what is incompatible with swarm mode? Swarm Mode is smth. Kubernetes does for a long time :) As soon as ip announcement works everything is fine, cluster nodes can be found by contacting an infrastructure service which routes the request to the appropriate node.

ovanes commented Aug 29, 2016

@mostolog Can you explain how elastic search does this and what is incompatible with swarm mode? Swarm Mode is smth. Kubernetes does for a long time :) As soon as ip announcement works everything is fine, cluster nodes can be found by contacting an infrastructure service which routes the request to the appropriate node.

@mostolog

This comment has been minimized.

Show comment
Hide comment
@mostolog

mostolog Aug 31, 2016

Hi

Consider the following docker command:

docker service create --name my-redis --replicas 3 redis:3.2.1 redis-server /etc/redis/redis.conf

Instead of creating a 3 node redis cluster, it creates a 3 node redis array (3 standalone instances of redis).

It would be great to have an option to configure a cluster name, multicast group, SSDP...to have a "PnP" cluster, if they shared the same password..
In other words: https://en.wikipedia.org/wiki/Zero-configuration_networking

Elasticsearch had multicast support enabled by default on previous versions, but they just moved to unicast, and we are experiencing the same issues: "the need of address to be able to form a cluster"
IIRC, docker is considering something like apple bonjour mDNS-SD

Regards

PS: Even more: it would be great allowing redis listen on a ip/mask (eg: 172.16.0.2/16) instead of fixed IP. this would enable dynamic IP configuration while preserving the redis-network isolation

mostolog commented Aug 31, 2016

Hi

Consider the following docker command:

docker service create --name my-redis --replicas 3 redis:3.2.1 redis-server /etc/redis/redis.conf

Instead of creating a 3 node redis cluster, it creates a 3 node redis array (3 standalone instances of redis).

It would be great to have an option to configure a cluster name, multicast group, SSDP...to have a "PnP" cluster, if they shared the same password..
In other words: https://en.wikipedia.org/wiki/Zero-configuration_networking

Elasticsearch had multicast support enabled by default on previous versions, but they just moved to unicast, and we are experiencing the same issues: "the need of address to be able to form a cluster"
IIRC, docker is considering something like apple bonjour mDNS-SD

Regards

PS: Even more: it would be great allowing redis listen on a ip/mask (eg: 172.16.0.2/16) instead of fixed IP. this would enable dynamic IP configuration while preserving the redis-network isolation

@ovanes

This comment has been minimized.

Show comment
Hide comment
@ovanes

ovanes Aug 31, 2016

I don't think multicast is an option :( It'd mean a lot of redesign in Redis, afaik Redis is TCP/IP only, on top this answer might be interesting:

https://groups.google.com/forum/#!topic/coreos-user/tbh1op__qNM

Multicast is very hard to implement with cloud providers and from all available networking layers for docker only weave supports multicast.

ovanes commented Aug 31, 2016

I don't think multicast is an option :( It'd mean a lot of redesign in Redis, afaik Redis is TCP/IP only, on top this answer might be interesting:

https://groups.google.com/forum/#!topic/coreos-user/tbh1op__qNM

Multicast is very hard to implement with cloud providers and from all available networking layers for docker only weave supports multicast.

@mostolog

This comment has been minimized.

Show comment
Hide comment
@mostolog

mostolog Aug 31, 2016

Ok about multicast...but what about the other options?

Have you considered an easier way of deploying a cluster not involving redis-trib/utilities?
For example: a unicast list of hostname:port in the config file, sharing the same password and creating a cluster+slots...working just out of the box

If you consider other proposals (bonjour or listening on network/mask) are interesting I can create those issues

Regards

mostolog commented Aug 31, 2016

Ok about multicast...but what about the other options?

Have you considered an easier way of deploying a cluster not involving redis-trib/utilities?
For example: a unicast list of hostname:port in the config file, sharing the same password and creating a cluster+slots...working just out of the box

If you consider other proposals (bonjour or listening on network/mask) are interesting I can create those issues

Regards

@vvanholl

This comment has been minimized.

Show comment
Hide comment
@vvanholl

vvanholl Sep 23, 2016

@mostolog, another solution could be the usage of services like Consul, Etcd or Zookeeper

vvanholl commented Sep 23, 2016

@mostolog, another solution could be the usage of services like Consul, Etcd or Zookeeper

@mostolog

This comment has been minimized.

Show comment
Hide comment
@mostolog

mostolog Sep 26, 2016

@vvanholl I didn't mention those, 'cause as a docker user, swarm mode is replacing external registry needs (IMHO: that's the way to take)

mostolog commented Sep 26, 2016

@vvanholl I didn't mention those, 'cause as a docker user, swarm mode is replacing external registry needs (IMHO: that's the way to take)

@yank1

This comment has been minimized.

Show comment
Hide comment
@yank1

yank1 Nov 22, 2016

Docker Swarm is Good to me

yank1 commented Nov 22, 2016

Docker Swarm is Good to me

@bironran

This comment has been minimized.

Show comment
Hide comment
@bironran

bironran Feb 17, 2017

An additional feedback based on my own try-and-fail attempt with Redis cluster and K8s:
We need to be able to distinguish between the internal Redis network and the external one. Specifically, we need to be able to have each Redis server identify by two IPs - one for it's fellow cluster members (local network) and that would be reported to clients using the "cluster nodes" and other cluster commands, as "moved" response and when using "asking".
Because some commands rely on IP rather than node IDs this is where it gets complicated. With the current implementation I see no easy way out for the general case. However, if we can relax the requirements and commit to no overlap in networks, we can simply use each additional IP as an alias to the original IP the cluster would use, then add alias resolution to commands and add all known IP aliases to each response. As this is breaking protocol, perhaps toggle this by flag.

bironran commented Feb 17, 2017

An additional feedback based on my own try-and-fail attempt with Redis cluster and K8s:
We need to be able to distinguish between the internal Redis network and the external one. Specifically, we need to be able to have each Redis server identify by two IPs - one for it's fellow cluster members (local network) and that would be reported to clients using the "cluster nodes" and other cluster commands, as "moved" response and when using "asking".
Because some commands rely on IP rather than node IDs this is where it gets complicated. With the current implementation I see no easy way out for the general case. However, if we can relax the requirements and commit to no overlap in networks, we can simply use each additional IP as an alias to the original IP the cluster would use, then add alias resolution to commands and add all known IP aliases to each response. As this is breaking protocol, perhaps toggle this by flag.

zuxqoj pushed a commit to zuxqoj/kubernetes-redis-cluster that referenced this issue Jul 17, 2017

Shekhar Bansal
Kubernetes deployment - Moved to Redis-4.0.0
It supports cluster-announce-ip and cluster-announce-port
antirez/redis#2527

@Druotic Druotic referenced this issue Aug 6, 2017

Merged

Initial project #1

@drnybble

This comment has been minimized.

Show comment
Hide comment
@drnybble

drnybble Aug 17, 2017

Can someone summarize -- can redis be run under Docker Swarm in the master/slave/sentinel configuration? I have tried (redis 3.2) but the cluster seems to get befuddled as containers change IP addresses when they are restarted. It seems to get into a state where it will never elect a master even though there are two redis servers running + 3 sentinels.

drnybble commented Aug 17, 2017

Can someone summarize -- can redis be run under Docker Swarm in the master/slave/sentinel configuration? I have tried (redis 3.2) but the cluster seems to get befuddled as containers change IP addresses when they are restarted. It seems to get into a state where it will never elect a master even though there are two redis servers running + 3 sentinels.

@ovanes

This comment has been minimized.

Show comment
Hide comment
@ovanes

ovanes Aug 17, 2017

@drnybble If I am not mistaken, this feature was first introduced in Redis 4.0.

Here is what I read from the release history log's section "Redis 4.0.0-RC1 Released Fri Dec 2 10:40:01 CEST 2016" located in (https://raw.githubusercontent.com/antirez/redis/4.0/00-RELEASENOTES):

  • Redis Cluster support for NAT / Docker. There are new functionalities in order to force cluster instances to announce specific sets of IP address, client and bus ports, to the rest of the cluster, regardless of the auto detected IP. This required a bus protocol change that will force users to mass-restart all the nodes of a Redis 3.2 installation in order to upgrade to 4.0.

ovanes commented Aug 17, 2017

@drnybble If I am not mistaken, this feature was first introduced in Redis 4.0.

Here is what I read from the release history log's section "Redis 4.0.0-RC1 Released Fri Dec 2 10:40:01 CEST 2016" located in (https://raw.githubusercontent.com/antirez/redis/4.0/00-RELEASENOTES):

  • Redis Cluster support for NAT / Docker. There are new functionalities in order to force cluster instances to announce specific sets of IP address, client and bus ports, to the rest of the cluster, regardless of the auto detected IP. This required a bus protocol change that will force users to mass-restart all the nodes of a Redis 3.2 installation in order to upgrade to 4.0.
@drnybble

This comment has been minimized.

Show comment
Hide comment
@drnybble

drnybble Aug 17, 2017

Thanks -- I am talking master/slave/sentinel configuration (replication) though, not cluster.

I seem to have some success specifying this for each master/slave:

slave-announce-ip [my-virtual-ip]

I specified the hostname of the service -- there still seems to be some confusion here since the sentinels seem to recognize both the hostname and the ip address as separate entities:

1:X 17 Aug 18:04:37.371 # -sdown slave redis1:6379 10.0.0.3 6379 @ redis-cluster 10.0.0.7 6379
1:X 17 Aug 18:04:37.372 # -sdown slave 10.0.0.3:6379 10.0.0.3 6379 @ redis-cluster 10.0.0.7 6379
1:X 17 Aug 18:06:05.395 * +reboot slave redis2:6379 10.0.0.5 6379 @ redis-cluster 10.0.0.7 6379
1:X 17 Aug 18:06:06.255 * +reboot slave 10.0.0.5:6379 10.0.0.5 6379 @ redis-cluster 10.0.0.7 6379

I can always resolve the hostname before writing it to the .conf file if it is a problem.

I was able to force update services for masters and slaves (docker stack update --force redis_redis1 for instance) and so far it seems to be working as expected.

drnybble commented Aug 17, 2017

Thanks -- I am talking master/slave/sentinel configuration (replication) though, not cluster.

I seem to have some success specifying this for each master/slave:

slave-announce-ip [my-virtual-ip]

I specified the hostname of the service -- there still seems to be some confusion here since the sentinels seem to recognize both the hostname and the ip address as separate entities:

1:X 17 Aug 18:04:37.371 # -sdown slave redis1:6379 10.0.0.3 6379 @ redis-cluster 10.0.0.7 6379
1:X 17 Aug 18:04:37.372 # -sdown slave 10.0.0.3:6379 10.0.0.3 6379 @ redis-cluster 10.0.0.7 6379
1:X 17 Aug 18:06:05.395 * +reboot slave redis2:6379 10.0.0.5 6379 @ redis-cluster 10.0.0.7 6379
1:X 17 Aug 18:06:06.255 * +reboot slave 10.0.0.5:6379 10.0.0.5 6379 @ redis-cluster 10.0.0.7 6379

I can always resolve the hostname before writing it to the .conf file if it is a problem.

I was able to force update services for masters and slaves (docker stack update --force redis_redis1 for instance) and so far it seems to be working as expected.

@ovanes

This comment has been minimized.

Show comment
Hide comment
@ovanes

ovanes Aug 18, 2017

@drnybble I am a bit mislead: I think this is wrong issue to discuss about announce-ip in HA deployment, as this issue explicitly deals with Redis Cluster.

ovanes commented Aug 18, 2017

@drnybble I am a bit mislead: I think this is wrong issue to discuss about announce-ip in HA deployment, as this issue explicitly deals with Redis Cluster.

@drnybble

This comment has been minimized.

Show comment
Hide comment
@drnybble

drnybble Aug 18, 2017

@ovanes You are correct. However, I think this information is also relevant for running Redis Cluster in Docker Swarm. The announce-ip should announce the Virtual IP of the service so it is stable over container updates.

drnybble commented Aug 18, 2017

@ovanes You are correct. However, I think this information is also relevant for running Redis Cluster in Docker Swarm. The announce-ip should announce the Virtual IP of the service so it is stable over container updates.

@ovanes

This comment has been minimized.

Show comment
Hide comment
@ovanes

ovanes Aug 18, 2017

@drnybble Yes, you are right. But than it should be tested with the version where it was introduced than, i.e. 4.0 and not 3.2. And redis release docs explicitly state, that there were bus protocol updates which are incompatible with version 3.2, etc. IMO, all the findings presented for 3.2 are pointless :( As one can't make any kind of conclusion to the version 4.0 because of severe changes.

ovanes commented Aug 18, 2017

@drnybble Yes, you are right. But than it should be tested with the version where it was introduced than, i.e. 4.0 and not 3.2. And redis release docs explicitly state, that there were bus protocol updates which are incompatible with version 3.2, etc. IMO, all the findings presented for 3.2 are pointless :( As one can't make any kind of conclusion to the version 4.0 because of severe changes.

@DrGunjah

This comment has been minimized.

Show comment
Hide comment
@DrGunjah

DrGunjah Oct 6, 2017

So is this supposed to work in redis 4.0.2? There are cluster-announce directives now though I still can't get it to work.
I have a docker container which contains three redis cluster nodes. I've set up the container in bridge mode and exposed the redis server and bus ports. All ports can be reached from the host machine and from within the container, however if I try to set up the cluster with redis-trib.rb the cluster won't join.
I get this message in the redis.log:
Unable to connect to Cluster Node [1.2.3.4]:12345 -> Name or service not known
(dummy ip and port)
However, the address is reachable from host and container. I'm only a bit skeptical towards the [] brackets in the error message. It seems a bit like at some stage these are added and the dns then fails because of them. I could be totally wrong though.

Cheers

DrGunjah commented Oct 6, 2017

So is this supposed to work in redis 4.0.2? There are cluster-announce directives now though I still can't get it to work.
I have a docker container which contains three redis cluster nodes. I've set up the container in bridge mode and exposed the redis server and bus ports. All ports can be reached from the host machine and from within the container, however if I try to set up the cluster with redis-trib.rb the cluster won't join.
I get this message in the redis.log:
Unable to connect to Cluster Node [1.2.3.4]:12345 -> Name or service not known
(dummy ip and port)
However, the address is reachable from host and container. I'm only a bit skeptical towards the [] brackets in the error message. It seems a bit like at some stage these are added and the dns then fails because of them. I could be totally wrong though.

Cheers

@wizzu

This comment has been minimized.

Show comment
Hide comment
@wizzu

wizzu Oct 6, 2017

I'm only a bit skeptical towards the [] brackets in the error message. It seems a bit like at some stage these are added and the dns then fails because of them. I could be totally wrong though.

The square brackets are part of the error message formatting, so the address is fine.
See

"Cluster Node [%s]:%d -> %s", node->ip,
(I think this is the spot, but could be wrong).

As for the actual problem, sorry but don't have any idea about that.

wizzu commented Oct 6, 2017

I'm only a bit skeptical towards the [] brackets in the error message. It seems a bit like at some stage these are added and the dns then fails because of them. I could be totally wrong though.

The square brackets are part of the error message formatting, so the address is fine.
See

"Cluster Node [%s]:%d -> %s", node->ip,
(I think this is the spot, but could be wrong).

As for the actual problem, sorry but don't have any idea about that.

@DrGunjah

This comment has been minimized.

Show comment
Hide comment
@DrGunjah

DrGunjah Oct 9, 2017

@wizzu thank you for pointing this out. So if the ip address is correct I have no idea what else could be wrong at this point.

DrGunjah commented Oct 9, 2017

@wizzu thank you for pointing this out. So if the ip address is correct I have no idea what else could be wrong at this point.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment