New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Start Redis cluster #79

Open
sparachi opened this Issue Oct 18, 2016 · 21 comments

Comments

Projects
None yet
@sparachi

sparachi commented Oct 18, 2016

I tried to use this Dockerfile and create a cluster. Have created the cluster instances based on this image. But I need to run "./redis-trib.rb create ... " command to setup the cluster configuration and I need to have ruby installed in the image. I tried couple of things but nothing seems to be working. Can you also update the dockerfile to include ruby installation.

@yosifkit

This comment has been minimized.

Member

yosifkit commented Oct 19, 2016

Since ruby is not necessary to run redis-server I would be hesitant in adding it. It should be simple to create a Dockerfile similar to the following:

FROM redis:3.2
RUN apt-get update && apt-get install -y ruby && rm -rf /var/lib/apt/lists/*

The it could be built by the automated build system of the Docker Hub or via your own deployment processes.

@dimthe

This comment has been minimized.

dimthe commented Nov 6, 2016

@sparachi can you please explain/show you set up the cluster with this dockerfile please

@mmzhou

This comment has been minimized.

mmzhou commented Dec 21, 2016

I have the same problem too. @yosifkit now that there is no other efficient way to start a redis cluster, why we can't include ruby in this image? I feel it's necessary. Thanks!

@mmzhou

This comment has been minimized.

mmzhou commented Dec 21, 2016

Plus, why there is no file: redis-trib.rb in the image? Thanks!

@tianon

This comment has been minimized.

Member

tianon commented Dec 21, 2016

The image only includes what's included in a standard Redis install -- if that file isn't included in the image, it's because it's not included by upstream during make install.

If that file were included, then having Ruby in the image by default might make more sense, but I'd rather not do so unless upstream officially recommends it (especially since adding Ruby is going to have a fairly significant effect on our image size, I'd imagine).

@tianon

This comment has been minimized.

Member

tianon commented Dec 21, 2016

For those interested, that file appears to be available at http://download.redis.io/redis-stable/src/redis-trib.rb.

@yosifkit

This comment has been minimized.

Member

yosifkit commented Dec 22, 2016

You don't need ruby in the redis nodes, just ruby to bootstrap the cluster:

$ # create config files for the nodes
$ cat redis/redis.conf
port 6375
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly yes
$ # create a network for them to talk on
$ docker network create redis-net
$ # start 6 nodes (you would probably want to bake the config in an image instead of the bind mount)
$ for red in 1 2 3 4 5 6; do \
  docker run -d -v "$PWD"/redis/:/usr/local/etc/redis/ \
  --name "redis-$red" --net redis-net \
  redis redis-server /usr/local/etc/redis/redis.conf; \
done
$ # run the redis-trib.rb script  (the docker inspect runs on the host and the echo output is passed the along to the ruby container)
$ docker run -it --rm --net redis-net ruby sh -c '\
  gem install redis \
  && wget http://download.redis.io/redis-stable/src/redis-trib.rb \
  && ruby redis-trib.rb create --replicas 1 \
  '"$(for red in 1 2 3 4 5 6; do \
    echo -n "$(docker inspect --format '{{ (index .NetworkSettings.Networks "redis-net").IPAddress }}' "redis-$red")"':6375 '; \
  done)"

>>> Creating cluster
>>> Performing hash slots allocation on 6 nodes...
Using 3 masters:
172.20.0.2:6375
172.20.0.3:6375
172.20.0.4:6375
Adding replica 172.20.0.5:6375 to 172.20.0.2:6375
Adding replica 172.20.0.6:6375 to 172.20.0.3:6375
Adding replica 172.20.0.7:6375 to 172.20.0.4:6375
M: bdd9851c4da290e0b61794afee0f13825e00e2e0 172.20.0.2:6375
   slots:0-5460 (5461 slots) master
M: ce9bf375d937453fd201844879d53aa04ec0c31f 172.20.0.3:6375
   slots:5461-10922 (5462 slots) master
M: 7a63d49e7f751fcf0dc5bed8b69701efbd40e4de 172.20.0.4:6375
   slots:10923-16383 (5461 slots) master
S: cd1e320aef2b2f6a105e08fc6b90db71327a5354 172.20.0.5:6375
   replicates bdd9851c4da290e0b61794afee0f13825e00e2e0
S: 4edf33bd5f2f22672fede9dcc7bf2626c638a8a6 172.20.0.6:6375
   replicates ce9bf375d937453fd201844879d53aa04ec0c31f
S: 051ecb30f6099e62edbaedef95003e4dfa91befe 172.20.0.7:6375
   replicates 7a63d49e7f751fcf0dc5bed8b69701efbd40e4de
Can I set the above configuration? (type 'yes' to accept): yes
>>> Nodes configuration updated
>>> Assign a different config epoch to each node
>>> Sending CLUSTER MEET messages to join the cluster
Waiting for the cluster to join..
>>> Performing Cluster Check (using node 172.20.0.2:6375)
M: bdd9851c4da290e0b61794afee0f13825e00e2e0 172.20.0.2:6375
   slots:0-5460 (5461 slots) master
   1 additional replica(s)
S: cd1e320aef2b2f6a105e08fc6b90db71327a5354 172.20.0.5:6375
   slots: (0 slots) slave
   replicates bdd9851c4da290e0b61794afee0f13825e00e2e0
M: 7a63d49e7f751fcf0dc5bed8b69701efbd40e4de 172.20.0.4:6375
   slots:10923-16383 (5461 slots) master
   1 additional replica(s)
S: 051ecb30f6099e62edbaedef95003e4dfa91befe 172.20.0.7:6375
   slots: (0 slots) slave
   replicates 7a63d49e7f751fcf0dc5bed8b69701efbd40e4de
M: ce9bf375d937453fd201844879d53aa04ec0c31f 172.20.0.3:6375
   slots:5461-10922 (5462 slots) master
   1 additional replica(s)
S: 4edf33bd5f2f22672fede9dcc7bf2626c638a8a6 172.20.0.6:6375
   slots: (0 slots) slave
   replicates ce9bf375d937453fd201844879d53aa04ec0c31f
[OK] All nodes agree about slots configuration.
>>> Check for open slots...
>>> Check slots coverage...
[OK] All 16384 slots covered.

No, you cannot use hostnames unless this merges: antirez/redis#2323.

@mmzhou

This comment has been minimized.

mmzhou commented Dec 22, 2016

@tianon redis-trib.rb was located in redis source code, it doesn't appear to this image just because it has been deleted after make. So I think it makes sense to integrate all necessary tools to run cluster. Isolating configuration is a main reason why we are using docker.

@mmzhou

This comment has been minimized.

mmzhou commented Dec 22, 2016

$ docker run -it --rm --net redis-net ruby sh -c '
gem install redis
&& wget http://download.redis.io/redis-stable/src/redis-trib.rb
&& ruby redis-trib.rb create --replicas 1
'"$(for red in 1 2 3 4 5 6; do
echo -n "$(docker inspect --format '{{ (index .NetworkSettings.Networks "redis-net").IPAddress }}' "redis-$red")"':6375 ';
done)"

@yosifkit I feel not good each time I have to download and install new things when I create the cluster. Besides, It's also not good to assuming the hosting environment has python installed already.

@yosifkit

This comment has been minimized.

Member

yosifkit commented Dec 29, 2016

There is no python in the example I provided; that is just bash on the host, and sh and ruby in the ruby container. The only new things to download are the ruby image, the redis gem, and the redis-trib.rb file from redis.io (all of which could be bundled in a custom image via a Dockerfile). What I was pointing out is that these items do not belong in the redis image since they are not required to run redis, but could be part of your own orchestration to create and manage a redis cluster.

@racc-costa

This comment has been minimized.

racc-costa commented Feb 18, 2017

@mmzhou please try this image https://hub.docker.com/r/racccosta/redis/.
I already include the dependencies needed to run redis-trib.rb.

@boosh

This comment has been minimized.

boosh commented May 23, 2017

@yosifkit A good compromise would be to copy redis-trib to /usr/local/bin from the tarball. Then if anyone wants to run a cluster they could extend yours and just install ruby.

@boosh

This comment has been minimized.

boosh commented May 25, 2017

...Or provide an official redis-trib image like https://hub.docker.com/r/zvelo/redis-trib/

@akkizz

This comment has been minimized.

akkizz commented Jun 27, 2017

I want to create a Redis cluster instance using docker file . I tried modifying the redis.conf and add below parameters :

port 6379
cluster-enabled yes
cluster-config-file nodes.conf
cluster-node-timeout 5000
appendonly no

But when i run the image with below command the instance runs as a Standalone.

sudo docker run -v /myredis/conf/redis.conf:/myredis/conf/redis.conf --name myredis redis redis-server /myredis/conf/redis.conf instance.

Any pointer for this would be helpful.
Thanks in advance.

@yu2003w

This comment has been minimized.

yu2003w commented Aug 31, 2017

@yosifkit Is there any plan to release redis cluster orchestration solution for k8s, mesos?

@yosifkit

This comment has been minimized.

Member

yosifkit commented Aug 31, 2017

@yu2003w, no plans for an auto-clustering image that I know of; we've been keeping the image as close to how upstream releases the software as possible.

@cihanb and @ushachar from upstream are now collaborating on the official-image, so maybe they have an opinion here?

@thesobercoder

This comment has been minimized.

thesobercoder commented Apr 24, 2018

This is a really super needed feature. I wasted almost two days trying to figure this out.

@wglambert wglambert added the question label Apr 24, 2018

@shankarshastri

This comment has been minimized.

shankarshastri commented May 29, 2018

@yosifkit, Will redis-trib connect to the cluster without password?

@yosifkit

This comment has been minimized.

Member

yosifkit commented May 29, 2018

@shankarshastri, I assume it does. In my example above, it doesn't look like I used a password.

@shankarshastri

This comment has been minimized.

shankarshastri commented May 29, 2018

@yosifkit, can you help me with the command?

@boosh

This comment has been minimized.

boosh commented Jun 2, 2018

BTW I managed to set up a redis cluster on k8s using a stateful set last year. But in the end the cluster was really flaky when a pod was terminated. IIRC I think it was because redis (2 at least) uses IPs instead of hostnames, but YMMV. I ended up running a standalone redis for each of my pods that needed a cache. It was a suboptimal approach, but more stable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment