Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for REDIS ZSET #46

Open
wants to merge 19 commits into
base: main
Choose a base branch
from
Open

Conversation

amaury-d
Copy link

@amaury-d amaury-d commented Feb 7, 2017

This commit adds support for ZSET queues in REDIS with data_type sortedset.
Batch is supported.

Priority is based on a field of the message containing a number.
If the field is absent or not a number a default priority is then used.

It adds the following options to the configuration:

  • priority_field: string, default 'epoch', Priority field to
    use, if field doesn't exist, priority will be priority_default
  • priority_default: number, default '-1', Default priority when
    priority field is not found in the event

Usage example:

output {
   redis {
   	host => ["127.0.0.1"]
   	port => 6379
   	data_type => "sortedset"
   	key => "syslog"
   	priority_field => "myepochfield"
   }
}

=> The messages will be push to a sortedset with the key syslog using the content of the field myepochfield as score.

Thanks for contributing to Logstash! If you haven't already signed our CLA, here's a handy link: https://www.elastic.co/contributor-agreement/

This commit adds support for ZSET queues in REDIS with data_type 'list'.
Batch is supported.

Priority is based on a field of the message containing a number.
If the field is absent or not a number a default priority is then used.

It adds the following options to the configuration:
* **priority**: *boolean*, default false, enable priority mode
* **priority_field**: *string*, default '@timestamp', Priority field to
use, if field doesn't exist, priority will be priority_default
* **priority_default**: *number*, default '0', Default priority when
priority field is not found in the event

Usage example:
    output {
    	redis {
		host => ["127.0.0.1"]
		port => 6379
		data_type => "list"
		key => "syslog"
		priority => true
		priority_field => "epoch"
	}
    }
@amaury-d
Copy link
Author

amaury-d commented Feb 7, 2017

This pull request is linked to logstash-plugins/logstash-input-redis#56

@amaury-d amaury-d changed the title Support for REDIS priority queues Support for REDIS ZSET Feb 27, 2017
@untergeek
Copy link

@amaury-d Can you please add some rspec tests for these changes? This is a rather significant update and some tests are absolutely necessary.

@amaury-d
Copy link
Author

amaury-d commented Mar 2, 2017

@untergeek Sure, i'll work on it and update the pull request

Sometimes redis didn't have the time to ingest all messages and the
test fails randomly.
It appeared it was due to redis too short timeout.
As sorted sets are a native redis data type, it's best to support ZSET
through data_type option instead to boolean priority option.

A first group of integration tests (batched and unbatched) for
sorted sets is also added with this commit.
This commit adds support for ZSET queues in REDIS with data_type 'list'.
Batch is supported.

Priority is based on a field of the message containing a number.
If the field is absent or not a number a default priority is then used.

It adds the following options to the configuration:
* **priority**: *boolean*, default false, enable priority mode
* **priority_field**: *string*, default '@timestamp', Priority field to
use, if field doesn't exist, priority will be priority_default
* **priority_default**: *number*, default '0', Default priority when
priority field is not found in the event

Usage example:
    output {
    	redis {
		host => ["127.0.0.1"]
		port => 6379
		data_type => "list"
		key => "syslog"
		priority => true
		priority_field => "epoch"
	}
    }
Sometimes redis didn't have the time to ingest all messages and the
test fails randomly.
It appeared it was due to redis too short timeout.
As sorted sets are a native redis data type, it's best to support ZSET
through data_type option instead to boolean priority option.

A first group of integration tests (batched and unbatched) for
sorted sets is also added with this commit.
@amaury-d
Copy link
Author

Hi @untergeek @andrewvc ,

rspec tests and supplementary documentation have been added to pull requests (input and output).

@zivziv77
Copy link

Hello,

How can I download you changes? I need them also

Thanks

@amaury-d
Copy link
Author

@zivziv77 just go on my github or check the diff of that pull request

@zivziv77
Copy link

Thanks,

why not putting it in master and have it in the official release?

@zivziv77
Copy link

zivziv77 commented Jul 4, 2018 via email

@mudit1804
Copy link

mudit1804 commented May 29, 2019

@amaury-d Does it supports NX/XX options along with ZADD command ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants