Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Redis cluster in input and output plugins #10

Open
jordansissel opened this issue May 18, 2015 · 8 comments

Comments

Projects
None yet
8 participants
@jordansissel
Copy link
Contributor

commented May 18, 2015

(This issue was originally filed by @liuxingyishi at elastic/logstash#2979)


When I set an output to a node in redis-cluster(Redis 3.0.0),it can not direct to other nodes in redis-cluster automatically.Port 7000 to 7005 are used by redis cluster.
logstash logstash-simple.conf:
input { stdin { } }
output {
redis {
host => ["localhost:7000"]
data_type => "list"
key => "key_count"}
stdout { codec => rubydebug }
}
logstash logs:exception=>#Redis::CommandError: MOVED 15454 127.0.0.1:7002
I supposed that it may be caused by logstash(redis-client role) was not started in cluster mode.

@ChastinaLi

This comment has been minimized.

Copy link

commented Jul 22, 2015

👍

@splitice

This comment has been minimized.

Copy link

commented Oct 17, 2015

Just thought I would share a little bit of an experiement with cluster support.

Heres the hacked together plugin (https://github.com/splitice/logstash-input-redis), do not even think of running this is in production. Its just some early exploration of what would be needed (and some fiddling in ruby as I have never touched the language before). After I get our log producer to support redis-cluster I'll do some more work.

Only the queue/list based method has been tested.

Configure like:

redis_cluster {
host => "127.0.0.1"
data_type => "list"
keys => ["rsyslog_1"]
}
@guyboertje

This comment has been minimized.

Copy link
Contributor

commented Aug 16, 2016

OK - All the bazillion plus ones have convinced me. I am also swayed by the fine experiment that @splitice has done with Jedis.

I will build a Jedis JRuby extension that supports pipelined lrange ltrim on clustered Redis.

@guyboertje guyboertje self-assigned this Aug 16, 2016

@suyograo suyograo changed the title Lack of settings for input and ouput for Redis cluster ? Support Redis cluster in input and output plugins Aug 16, 2016

@mp911de

This comment has been minimized.

Copy link

commented Aug 18, 2016

@guyboertje @splitice Did you test the redis input also with lettuce? Lettuce does not require connection pooling and pipelines commands by default.

@huseyincotuk

This comment has been minimized.

Copy link

commented Aug 26, 2016

@guyboertje @splitice Any progress on the development? I can test it as a volunteer right now. Please let me know when you finish.

@splitice

This comment has been minimized.

Copy link

commented Aug 26, 2016

We are now using a different intermediary queue, my work is complete / idle.

Imho redid cluster tooling still needs some work before it's ready for us.

@logstash-plugins logstash-plugins deleted a comment from aosmith Aug 21, 2017

@logstash-plugins logstash-plugins deleted a comment from shaharmor Aug 21, 2017

@logstash-plugins logstash-plugins deleted a comment from tdesrochers Aug 21, 2017

@logstash-plugins logstash-plugins deleted a comment from dzubchik Aug 21, 2017

@logstash-plugins logstash-plugins deleted a comment from splitice Aug 21, 2017

@logstash-plugins logstash-plugins deleted a comment from claygorman Aug 21, 2017

@logstash-plugins logstash-plugins deleted a comment from bline79 Aug 21, 2017

@logstash-plugins logstash-plugins deleted a comment from jacobluo Aug 21, 2017

@logstash-plugins logstash-plugins deleted a comment from jinxingf Aug 21, 2017

@logstash-plugins logstash-plugins deleted a comment from codezealer Aug 21, 2017

@logstash-plugins logstash-plugins deleted a comment from 523499159 Aug 21, 2017

@logstash-plugins logstash-plugins deleted a comment from yyt030 Aug 21, 2017

@guyboertje

This comment has been minimized.

Copy link
Contributor

commented Aug 21, 2017

Please do not keep adding +1 type comments. I (we, the Logstash team) are not yet able to code for Redis Clustering at the moment. We are in the process of defining the Java Plugin mechanism and selecting suitable plugins to move to Java - this plugin is one candidate - its just not a high priority one at the moment.

@berglh

This comment has been minimized.

Copy link

commented Jul 24, 2018

@guyboertje Redis-rb is looking like it's close to adding support for Redis Cluster. I'm not sure if this is directly portable to jruby for use in the logstash Redis plugins; but it sounded promising so I thought I'd add the ref here: redis/redis-rb#716

I've been blowing off the dust, getting this https://github.com/berglh/logstash-filter-redis plugin working for use as a dynamic look-up/write-out table as an alternative to the jdbc and translate filters. I've been testing this plugin against a Redis cluster only to find the Redis client didn't seem to like writes being redirected to other Redis server instances.

I'll keep an eye on that PR and figure out if it's possible to use it for logtash Redis plugins.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.