Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Redis cluster should support announce-ip & announce-port #2527

Open
kremso opened this issue Apr 15, 2015 · 82 comments
Open

Redis cluster should support announce-ip & announce-port #2527

kremso opened this issue Apr 15, 2015 · 82 comments

Comments

@kremso
Copy link

@kremso kremso commented Apr 15, 2015

We're running into problems when trying to run redis cluster from docker on different hosts. Dockerized redis node advertises its internal docker address, which is unreachable from other nodes in the cluster. I'm aware of various hacks and workarounds, but I'm proposing that redis cluster supports announce-ip and announce-port configuration options, similarly to redis sentinel.

@kremso
Copy link
Author

@kremso kremso commented Apr 16, 2015

I did try the recommended workaround of specifying an explicit bind, but it fails too. It seems that it won't work without setting IP_TRANSPARENT option on the socket, which redis currently does not.

@harley84
Copy link

@harley84 harley84 commented May 7, 2015

Related to #2335

@antirez
Copy link
Contributor

@antirez antirez commented Jun 3, 2015

Agreed, will write here a design spec in the next days. p.s. in the meanwhile you may want to prevent Docker from mapping ports, there is a specific feature for 1:1 mapping.

@KekSfabrik
Copy link

@KekSfabrik KekSfabrik commented Jun 15, 2015

I'm having the same trouble - see #2621
copy/pasta from there:
[...] Master instances know the slaves running on the same machine by local (172.*) IPs and vice versa. This may possibly lead to getting the MOVED response to an IP that is not reachable by the client:

192.168.1.245 / 172.17.42.1 # redis-cli -h 192.168.1.245 -p 6379 cluster nodes
8bd0598380fc05aa374ef22a74da1a691963cfbb 192.168.1.246:6380 slave e1fc77808551cccf5efe6bcbb5816b624bff9633 0 1434352847839 5 connected
5bd7fbc8a4dfe96667dfd668746c100f5829a0e3 192.168.1.247:6380 slave e5337aa24d19a177a6e90bcb1e144eedb436a0c7 0 1434352848039 6 connected
e5337aa24d19a177a6e90bcb1e144eedb436a0c7 192.168.1.246:6379 master - 0 1434352847638 2 connected 5461-10922
e1fc77808551cccf5efe6bcbb5816b624bff9633 192.168.1.245:6379 myself,master - 0 0 1 connected 0-5460
721e4986d1cc2dbf87ac11054242233ac3d67ede 172.17.42.1:6380 slave 5042f162cd5e057df377dd473359972f1bb7e287 0 1434352847139 4 connected
5042f162cd5e057df377dd473359972f1bb7e287 192.168.1.247:6379 master - 0 1434352847839 3 connected 10923-16383
192.168.1.245 / 172.17.42.1 # redis-cli -h 192.168.1.245 -p 6380 cluster nodes
e1fc77808551cccf5efe6bcbb5816b624bff9633 172.17.42.1:6379 master - 0 1434352872207 1 connected 0-5460
8bd0598380fc05aa374ef22a74da1a691963cfbb 192.168.1.246:6380 slave e1fc77808551cccf5efe6bcbb5816b624bff9633 0 1434352872509 5 connected
5042f162cd5e057df377dd473359972f1bb7e287 192.168.1.247:6379 master - 0 1434352872308 3 connected 10923-16383
e5337aa24d19a177a6e90bcb1e144eedb436a0c7 192.168.1.246:6379 master - 0 1434352872108 2 connected 5461-10922
5bd7fbc8a4dfe96667dfd668746c100f5829a0e3 192.168.1.247:6380 slave e5337aa24d19a177a6e90bcb1e144eedb436a0c7 0 1434352872509 6 connected
721e4986d1cc2dbf87ac11054242233ac3d67ede 192.168.1.245:6380 myself,slave 5042f162cd5e057df377dd473359972f1bb7e287 0 1434101988241 4 connected

however i've noticed they all act the same -- the ones on the 192.168.1.246 IP behave exactly the same (for their own IP). Relevant bits as seen by 192.168.1.246:6379:

e5337aa24d19a177a6e90bcb1e144eedb436a0c7 192.168.1.246:6379 myself,master - 0 1434101985463 2 connected 5461-10922
8bd0598380fc05aa374ef22a74da1a691963cfbb 172.17.42.1:6380 slave e1fc77808551cccf5efe6bcbb5816b624bff9633 0 1434375798687 5 connected

So they each have a (machine local) 172.17.* net that is not available from outside the machine

@drshade
Copy link

@drshade drshade commented Jul 7, 2015

+1 please

@Ducatel
Copy link

@Ducatel Ducatel commented Jul 8, 2015

If have the same problem. It isn't possible to create a redis cluster on different servers with docker.
Redis send the container IP instead of the host IP.
So for this Redis should add the support of announce-ip like the sentinel.

+1 please

@bjackson
Copy link

@bjackson bjackson commented Jul 9, 2015

+1
This would be amazing.

@antirez
Copy link
Contributor

@antirez antirez commented Jul 9, 2015

Prioritized in order to be shipped with the next release (3.2). Thanks for the feedbacks.

@ychrysler
Copy link

@ychrysler ychrysler commented Jul 11, 2015

👍 i there a proposed work-around available for this issue?

@ZhuoRoger
Copy link

@ZhuoRoger ZhuoRoger commented Jul 16, 2015

+1

@mistypig
Copy link

@mistypig mistypig commented Aug 24, 2015

+1

3 similar comments
@devaos
Copy link

@devaos devaos commented Sep 12, 2015

+1

@turbospaces
Copy link

@turbospaces turbospaces commented Sep 29, 2015

+1

@sst1xx
Copy link

@sst1xx sst1xx commented Sep 29, 2015

+1

@fab-io
Copy link

@fab-io fab-io commented Sep 30, 2015

As a workaround, if possible in your env, run the containers with the option --net=host (see https://docs.docker.com/articles/networking/#container-networking)

@rpannell
Copy link

@rpannell rpannell commented Oct 6, 2015

+1

@antirez
Copy link
Contributor

@antirez antirez commented Oct 15, 2015

Scheduled for Redis 3.2 that will go in RC in a matter of weeks (~4 hopefully).

@antirez antirez added this to the Redis >= 3.2 milestone Oct 15, 2015
@antirez
Copy link
Contributor

@antirez antirez commented Oct 15, 2015

p.s. this is going to be backported into 3.0.x anyway...

@himulawang
Copy link

@himulawang himulawang commented Nov 19, 2015

+1

@fatduo
Copy link

@fatduo fatduo commented Dec 9, 2015

Hi @antirez @harley84
Do you have some work-around solutions?
I think you can get the hostname instead of ip address. In docker container, I can pass hostname env to the container.

@MrMMorris
Copy link

@MrMMorris MrMMorris commented Dec 9, 2015

@fatduo the workaround is using --net=host as mentioned here: #2527 (comment)

@srinikalyanaraman
Copy link

@srinikalyanaraman srinikalyanaraman commented Dec 16, 2015

Hi @antirez @harley84
I have put in a fix in a fork and opened a PR (as per the CONTRIBUTING document). I have tested them in kubernetes on openstack. Please review. Thanks!!

@antirez
Copy link
Contributor

@antirez antirez commented Aug 25, 2016

@ovanes this is a very useful info, thanks!

@mostolog
Copy link

@mostolog mostolog commented Aug 29, 2016

Hi.

Probably this comment doesn't match exactly the topic, but I hope is enough close-related.

AFAIK, redis requires different node.conf cluster files and ports.
For that reason, when using docker, entrypoint scripts, variable substitution...are required.

This seems quite incompatible with new docker swarm mode and I was wondering if a gossip protocol, multicast or something else instead of static configurations could be used to create/handle the cluster on an easier way. (ie: the way elasticsearch does).
Is this what you have planned for Redis 4.0?

If I understood correctly, this is an example of such trickyness https://jamespedwards42.github.io/2016/06/21/docker-swarm-redis-cluster/

Regards

@ovanes
Copy link

@ovanes ovanes commented Aug 29, 2016

@mostolog Can you explain how elastic search does this and what is incompatible with swarm mode? Swarm Mode is smth. Kubernetes does for a long time :) As soon as ip announcement works everything is fine, cluster nodes can be found by contacting an infrastructure service which routes the request to the appropriate node.

@mostolog
Copy link

@mostolog mostolog commented Aug 31, 2016

Hi

Consider the following docker command:

docker service create --name my-redis --replicas 3 redis:3.2.1 redis-server /etc/redis/redis.conf

Instead of creating a 3 node redis cluster, it creates a 3 node redis array (3 standalone instances of redis).

It would be great to have an option to configure a cluster name, multicast group, SSDP...to have a "PnP" cluster, if they shared the same password..
In other words: https://en.wikipedia.org/wiki/Zero-configuration_networking

Elasticsearch had multicast support enabled by default on previous versions, but they just moved to unicast, and we are experiencing the same issues: "the need of address to be able to form a cluster"
IIRC, docker is considering something like apple bonjour mDNS-SD

Regards

PS: Even more: it would be great allowing redis listen on a ip/mask (eg: 172.16.0.2/16) instead of fixed IP. this would enable dynamic IP configuration while preserving the redis-network isolation

@ovanes
Copy link

@ovanes ovanes commented Aug 31, 2016

I don't think multicast is an option :( It'd mean a lot of redesign in Redis, afaik Redis is TCP/IP only, on top this answer might be interesting:

https://groups.google.com/forum/#!topic/coreos-user/tbh1op__qNM

Multicast is very hard to implement with cloud providers and from all available networking layers for docker only weave supports multicast.

@mostolog
Copy link

@mostolog mostolog commented Aug 31, 2016

Ok about multicast...but what about the other options?

Have you considered an easier way of deploying a cluster not involving redis-trib/utilities?
For example: a unicast list of hostname:port in the config file, sharing the same password and creating a cluster+slots...working just out of the box

If you consider other proposals (bonjour or listening on network/mask) are interesting I can create those issues

Regards

@mostolog
Copy link

@mostolog mostolog commented Sep 26, 2016

@vvanholl I didn't mention those, 'cause as a docker user, swarm mode is replacing external registry needs (IMHO: that's the way to take)

@yank1
Copy link

@yank1 yank1 commented Nov 22, 2016

Docker Swarm is Good to me

@bironran
Copy link

@bironran bironran commented Feb 17, 2017

An additional feedback based on my own try-and-fail attempt with Redis cluster and K8s:
We need to be able to distinguish between the internal Redis network and the external one. Specifically, we need to be able to have each Redis server identify by two IPs - one for it's fellow cluster members (local network) and that would be reported to clients using the "cluster nodes" and other cluster commands, as "moved" response and when using "asking".
Because some commands rely on IP rather than node IDs this is where it gets complicated. With the current implementation I see no easy way out for the general case. However, if we can relax the requirements and commit to no overlap in networks, we can simply use each additional IP as an alias to the original IP the cluster would use, then add alias resolution to commands and add all known IP aliases to each response. As this is breaking protocol, perhaps toggle this by flag.

zuxqoj pushed a commit to zuxqoj/kubernetes-redis-cluster that referenced this issue Jul 17, 2017
It supports cluster-announce-ip and cluster-announce-port
redis/redis#2527
@drnybble
Copy link

@drnybble drnybble commented Aug 17, 2017

Can someone summarize -- can redis be run under Docker Swarm in the master/slave/sentinel configuration? I have tried (redis 3.2) but the cluster seems to get befuddled as containers change IP addresses when they are restarted. It seems to get into a state where it will never elect a master even though there are two redis servers running + 3 sentinels.

@ovanes
Copy link

@ovanes ovanes commented Aug 17, 2017

@drnybble If I am not mistaken, this feature was first introduced in Redis 4.0.

Here is what I read from the release history log's section "Redis 4.0.0-RC1 Released Fri Dec 2 10:40:01 CEST 2016" located in (https://raw.githubusercontent.com/antirez/redis/4.0/00-RELEASENOTES):

  • Redis Cluster support for NAT / Docker. There are new functionalities in order to force cluster instances to announce specific sets of IP address, client and bus ports, to the rest of the cluster, regardless of the auto detected IP. This required a bus protocol change that will force users to mass-restart all the nodes of a Redis 3.2 installation in order to upgrade to 4.0.
@drnybble
Copy link

@drnybble drnybble commented Aug 17, 2017

Thanks -- I am talking master/slave/sentinel configuration (replication) though, not cluster.

I seem to have some success specifying this for each master/slave:

slave-announce-ip [my-virtual-ip]

I specified the hostname of the service -- there still seems to be some confusion here since the sentinels seem to recognize both the hostname and the ip address as separate entities:

1:X 17 Aug 18:04:37.371 # -sdown slave redis1:6379 10.0.0.3 6379 @ redis-cluster 10.0.0.7 6379
1:X 17 Aug 18:04:37.372 # -sdown slave 10.0.0.3:6379 10.0.0.3 6379 @ redis-cluster 10.0.0.7 6379
1:X 17 Aug 18:06:05.395 * +reboot slave redis2:6379 10.0.0.5 6379 @ redis-cluster 10.0.0.7 6379
1:X 17 Aug 18:06:06.255 * +reboot slave 10.0.0.5:6379 10.0.0.5 6379 @ redis-cluster 10.0.0.7 6379

I can always resolve the hostname before writing it to the .conf file if it is a problem.

I was able to force update services for masters and slaves (docker stack update --force redis_redis1 for instance) and so far it seems to be working as expected.

@ovanes
Copy link

@ovanes ovanes commented Aug 18, 2017

@drnybble I am a bit mislead: I think this is wrong issue to discuss about announce-ip in HA deployment, as this issue explicitly deals with Redis Cluster.

@drnybble
Copy link

@drnybble drnybble commented Aug 18, 2017

@ovanes You are correct. However, I think this information is also relevant for running Redis Cluster in Docker Swarm. The announce-ip should announce the Virtual IP of the service so it is stable over container updates.

@ovanes
Copy link

@ovanes ovanes commented Aug 18, 2017

@drnybble Yes, you are right. But than it should be tested with the version where it was introduced than, i.e. 4.0 and not 3.2. And redis release docs explicitly state, that there were bus protocol updates which are incompatible with version 3.2, etc. IMO, all the findings presented for 3.2 are pointless :( As one can't make any kind of conclusion to the version 4.0 because of severe changes.

@DrGunjah
Copy link

@DrGunjah DrGunjah commented Oct 6, 2017

So is this supposed to work in redis 4.0.2? There are cluster-announce directives now though I still can't get it to work.
I have a docker container which contains three redis cluster nodes. I've set up the container in bridge mode and exposed the redis server and bus ports. All ports can be reached from the host machine and from within the container, however if I try to set up the cluster with redis-trib.rb the cluster won't join.
I get this message in the redis.log:
Unable to connect to Cluster Node [1.2.3.4]:12345 -> Name or service not known
(dummy ip and port)
However, the address is reachable from host and container. I'm only a bit skeptical towards the [] brackets in the error message. It seems a bit like at some stage these are added and the dns then fails because of them. I could be totally wrong though.

Cheers

@wizzu
Copy link

@wizzu wizzu commented Oct 6, 2017

I'm only a bit skeptical towards the [] brackets in the error message. It seems a bit like at some stage these are added and the dns then fails because of them. I could be totally wrong though.

The square brackets are part of the error message formatting, so the address is fine.
See https://github.com/antirez/redis/blob/d9565379da0adf97caec12e9b88c17da51900e40/src/cluster.c#L3275 (I think this is the spot, but could be wrong).

As for the actual problem, sorry but don't have any idea about that.

@DrGunjah
Copy link

@DrGunjah DrGunjah commented Oct 9, 2017

@wizzu thank you for pointing this out. So if the ip address is correct I have no idea what else could be wrong at this point.

@ianmobbs
Copy link

@ianmobbs ianmobbs commented Mar 29, 2019

+1

@wojnosystems
Copy link

@wojnosystems wojnosystems commented Oct 7, 2019

Hey all. I ran into this as well.

I wrote a proxy that appears to work for simple set/gets on small clusters. This is a work in progress and was created only for testing a RedisCluster running in a docker-swarm, but it should work for K8s, too.

This proxy runs on the cluster-side and alters the IP addresses from the Redis Cluster to lie to clients so that they can route to a single proxy end-point. The proxy end-point listens on a single port for each node in the cluster and passes the traffic onto the actual nodes.

https://github.com/wojnosystems/redis_cluster_proxy

Please feel free to contribute for anything that's broken. This code has some limitations, but this enables me to test my cluster-integration tests.

Hope it helps you as well.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
You can’t perform that action at this time.