Skip to content

Commit

Permalink
Increase the default batching to 100 events
Browse files Browse the repository at this point in the history 
wip

set delay to zero

use Constant for sleep + spec for no results behaviour + better perf bm

add contributor - me (try to trigger travis build)

add changelog entry

bump version

closes #25

Fixes #36
  • Loading branch information
@suyograo
suyograo authored and Guy Boertje committed Mar 8, 2016
1 parent 8c93857 commit 29fa6fa
Show file tree
Hide file tree
Showing 9 changed files with 362 additions and 45 deletions.
12 changes: 12 additions & 0 deletions .travis.yml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
services:
- redis-server
sudo: false
language: ruby
cache: bundler
rvm:
- jruby-1.7.24
jdk:
- oraclejdk8
script:
- bundle exec rspec spec --order rand
- bundle exec rspec spec --order rand -t redis
5 changes: 4 additions & 1 deletion CHANGELOG.md
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
## 2.0.3
- Changed default batch size to 125. Improve batch handling code. Add travis ci build with redis integration.

## 2.0.0
- Plugins were updated to follow the new shutdown semantic, this mainly allows Logstash to instruct input plugins to terminate gracefully,
- Plugins were updated to follow the new shutdown semantic, this mainly allows Logstash to instruct input plugins to terminate gracefully,
instead of using Thread.raise on the plugins' threads. Ref: https://github.com/elastic/logstash/pull/3895
- Dependency on logstash-core update to 2.0

Expand Down
1 change: 1 addition & 0 deletions CONTRIBUTORS
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ Contributors:
* Pier-Hugues Pellerin (ph)
* Richard Pijnenburg (electrical)
* piavlo
* Guy Boertje (guyboertje)

Note: If you've sent us patches, bug reports, or otherwise contributed to
Logstash, and you aren't on the list above and want to be, please let us know
Expand Down
6 changes: 5 additions & 1 deletion README.md
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
# Logstash Plugin

Travis Build
[![Travis Build Status](https://travis-ci.org/logstash-plugins/logstash-input-redis.svg)](https://travis-ci.org/logstash-plugins/logstash-input-redis)

Jenkins Build
[![Build
Status](http://build-eu-00.elastic.co/view/LS%20Plugins/view/LS%20Inputs/job/logstash-plugin-input-redis-unit/badge/icon)](http://build-eu-00.elastic.co/view/LS%20Plugins/view/LS%20Inputs/job/logstash-plugin-input-redis-unit/)

Expand Down Expand Up @@ -86,4 +90,4 @@ Programming is not a required skill. Whatever you've seen about open source and

It is more important to the community that you are able to contribute.

For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.
For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.
90 changes: 90 additions & 0 deletions batch_perf/perf_batch.rb
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
require "benchmark"
require "redis"
require "securerandom"

require "logstash/event"
require "logstash/pipeline"
require_relative "../lib/logstash/inputs/redis"

class BenchOptions
attr_reader :output_width, :redis
attr_reader :key, :event_count, :sizes

def initialize
@output_width = 70
@redis = Redis.new(:host => "localhost")
@event_count = 5000
@key = SecureRandom.hex
@sizes = [1, 10, 100, 125, 250, 1000, 1, 10, 100, 125, 250, 1000]
end

def cfg_batch(d)
<<-CONFIG
input {
redis {
type => "blah"
key => "#{key}"
data_type => "list"
batch_count => #{d}
}
}
CONFIG
end
end

bench_options = BenchOptions.new

def input(cfg, slow, &block)
pipeline = LogStash::Pipeline.new(cfg)
queue = Queue.new

pipeline.instance_eval do
# create closure to capture queue
@output_func = lambda do |event|
(0...slow).to_a.reduce(&:+) if slow > 0
queue << event
end

# output_func is now a method, call closure
def output_func(event)
@output_func.call(event)
end
end

pipeline_thread = Thread.new { pipeline.run }
sleep 0.1 while !pipeline.ready?

result = block.call(pipeline, queue)

pipeline.shutdown
pipeline_thread.join

result
end

def setup(bo, multiplier)
temp = []
(bo.event_count * multiplier).times do |value|
temp << LogStash::Event.new("sequence" => value).to_json
end
temp.each_cons(10) do |arr|
bo.redis.rpush(bo.key, arr)
end
end

def teardown(bo)
bo.redis.del(bo.key)
end

Benchmark.bm(bench_options.output_width) do |x|
setup(bench_options, bench_options.sizes.size + 2)
delay = 2000
bench_options.sizes.reverse.each do |batchs|
input(bench_options.cfg_batch(batchs), delay) do |pipeline, queue|
x.report("redis input batch: #{batchs}, size: #{bench_options.event_count}, slow: #{delay} delay") do
bench_options.event_count.times.map{queue.pop}
end
end
end
teardown(bench_options)
end
152 changes: 152 additions & 0 deletions batch_perf/results.txt
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,152 @@
On a 8 core 16MB - Apple MBP El Capitan

MK2 - All sizes run twice for warmup
larger latency
user system total real
redis input batch: 1, size: 5000, slow: 2000 delay 2.670000 0.330000 3.000000 ( 0.973000)
redis input batch: 10, size: 5000, slow: 2000 delay 1.150000 0.090000 1.240000 ( 0.289000)
redis input batch: 100, size: 5000, slow: 2000 delay 1.130000 0.060000 1.190000 ( 0.188000)
redis input batch: 125, size: 5000, slow: 2000 delay 1.060000 0.060000 1.120000 ( 0.203000)
redis input batch: 250, size: 5000, slow: 2000 delay 1.210000 0.050000 1.260000 ( 0.187000)
redis input batch: 1000, size: 5000, slow: 2000 delay 1.130000 0.050000 1.180000 ( 0.195000)

redis input batch: 1, size: 5000, slow: 2000 delay 2.350000 0.330000 2.680000 ( 0.935000)
redis input batch: 10, size: 5000, slow: 2000 delay 1.040000 0.090000 1.130000 ( 0.243000)
redis input batch: 100, size: 5000, slow: 2000 delay 1.100000 0.050000 1.150000 ( 0.175000)
redis input batch: 125, size: 5000, slow: 2000 delay 1.210000 0.070000 1.280000 ( 0.195000)
redis input batch: 250, size: 5000, slow: 2000 delay 1.160000 0.060000 1.220000 ( 0.178000)
redis input batch: 1000, size: 5000, slow: 2000 delay 1.180000 0.050000 1.230000 ( 0.182000)

redis input batch: 1, size: 5000, slow: 2000 delay 2.610000 0.330000 2.940000 ( 0.941000)
redis input batch: 10, size: 5000, slow: 2000 delay 1.430000 0.110000 1.540000 ( 0.315000)
redis input batch: 100, size: 5000, slow: 2000 delay 1.280000 0.050000 1.330000 ( 0.195000)
redis input batch: 125, size: 5000, slow: 2000 delay 1.190000 0.060000 1.250000 ( 0.190000)
redis input batch: 250, size: 5000, slow: 2000 delay 1.150000 0.050000 1.200000 ( 0.185000)
redis input batch: 1000, size: 5000, slow: 2000 delay 1.050000 0.060000 1.110000 ( 0.193000)

redis input batch: 1, size: 5000, slow: 2000 delay 1.900000 0.230000 2.130000 ( 0.648000)
redis input batch: 10, size: 5000, slow: 2000 delay 0.960000 0.080000 1.040000 ( 0.199000)
redis input batch: 100, size: 5000, slow: 2000 delay 1.130000 0.050000 1.180000 ( 0.169000)
redis input batch: 125, size: 5000, slow: 2000 delay 1.080000 0.050000 1.130000 ( 0.160000)
redis input batch: 250, size: 5000, slow: 2000 delay 1.080000 0.050000 1.130000 ( 0.159000)
redis input batch: 1000, size: 5000, slow: 2000 delay 1.030000 0.040000 1.070000 ( 0.154000)

redis input batch: 1, size: 5000, slow: 2000 delay 2.850000 0.300000 3.150000 ( 0.767000)
redis input batch: 10, size: 5000, slow: 2000 delay 0.970000 0.080000 1.050000 ( 0.204000)
redis input batch: 100, size: 5000, slow: 2000 delay 1.170000 0.050000 1.220000 ( 0.175000)
redis input batch: 125, size: 5000, slow: 2000 delay 1.310000 0.060000 1.370000 ( 0.191000)
redis input batch: 250, size: 5000, slow: 2000 delay 1.190000 0.050000 1.240000 ( 0.172000)
redis input batch: 1000, size: 5000, slow: 2000 delay 1.100000 0.050000 1.150000 ( 0.177000)


v small latency
user system total real
redis input batch: 1, size: 5000, slow: 20 delay 0.930000 0.170000 1.100000 ( 0.503000)
redis input batch: 10, size: 5000, slow: 20 delay 0.100000 0.020000 0.120000 ( 0.071000)
redis input batch: 100, size: 5000, slow: 20 delay 0.020000 0.000000 0.020000 ( 0.012000)
redis input batch: 125, size: 5000, slow: 20 delay 0.020000 0.000000 0.020000 ( 0.017000)
redis input batch: 250, size: 5000, slow: 20 delay 0.060000 0.000000 0.060000 ( 0.023000)
redis input batch: 1000, size: 5000, slow: 20 delay 0.010000 0.000000 0.010000 ( 0.005000)

redis input batch: 1, size: 5000, slow: 20 delay 1.060000 0.180000 1.240000 ( 0.533000)
redis input batch: 10, size: 5000, slow: 20 delay 0.100000 0.030000 0.130000 ( 0.072000)
redis input batch: 100, size: 5000, slow: 20 delay 0.020000 0.000000 0.020000 ( 0.013000)
redis input batch: 125, size: 5000, slow: 20 delay 0.030000 0.000000 0.030000 ( 0.020000)
redis input batch: 250, size: 5000, slow: 20 delay 0.020000 0.010000 0.030000 ( 0.019000)
redis input batch: 1000, size: 5000, slow: 20 delay 0.010000 0.000000 0.010000 ( 0.003000)

redis input batch: 1, size: 5000, slow: 20 delay 1.170000 0.190000 1.360000 ( 0.548000)
redis input batch: 10, size: 5000, slow: 20 delay 0.070000 0.020000 0.090000 ( 0.069000)
redis input batch: 100, size: 5000, slow: 20 delay 0.050000 0.000000 0.050000 ( 0.019000)
redis input batch: 125, size: 5000, slow: 20 delay 0.010000 0.000000 0.010000 ( 0.006000)
redis input batch: 250, size: 5000, slow: 20 delay 0.020000 0.010000 0.030000 ( 0.015000)
redis input batch: 1000, size: 5000, slow: 20 delay 0.010000 0.000000 0.010000 ( 0.012000)

redis input batch: 1, size: 5000, slow: 20 delay 0.710000 0.170000 0.880000 ( 0.511000)
redis input batch: 10, size: 5000, slow: 20 delay 0.110000 0.020000 0.130000 ( 0.082000)
redis input batch: 100, size: 5000, slow: 20 delay 0.050000 0.010000 0.060000 ( 0.025000)
redis input batch: 125, size: 5000, slow: 20 delay 0.030000 0.010000 0.040000 ( 0.024000)
redis input batch: 250, size: 5000, slow: 20 delay 0.030000 0.010000 0.040000 ( 0.020000)
redis input batch: 1000, size: 5000, slow: 20 delay 0.030000 0.010000 0.040000 ( 0.020000)

redis input batch: 1, size: 5000, slow: 20 delay 1.040000 0.180000 1.220000 ( 0.525000)
redis input batch: 10, size: 5000, slow: 20 delay 0.070000 0.020000 0.090000 ( 0.058000)
redis input batch: 100, size: 5000, slow: 20 delay 0.030000 0.010000 0.040000 ( 0.022000)
redis input batch: 125, size: 5000, slow: 20 delay 0.020000 0.000000 0.020000 ( 0.019000)
redis input batch: 250, size: 5000, slow: 20 delay 0.020000 0.000000 0.020000 ( 0.011000)
redis input batch: 1000, size: 5000, slow: 20 delay 0.020000 0.000000 0.020000 ( 0.014000)

redis input batch: 1, size: 5000, slow: 20 delay 1.040000 0.200000 1.240000 ( 0.511000)
redis input batch: 10, size: 5000, slow: 20 delay 0.090000 0.020000 0.110000 ( 0.068000)
redis input batch: 100, size: 5000, slow: 20 delay 0.010000 0.000000 0.010000 ( 0.007000)
redis input batch: 125, size: 5000, slow: 20 delay 0.020000 0.000000 0.020000 ( 0.014000)
redis input batch: 250, size: 5000, slow: 20 delay 0.020000 0.000000 0.020000 ( 0.016000)
redis input batch: 1000, size: 5000, slow: 20 delay 0.000000 0.000000 0.000000 ( 0.004000)

redis input batch: 1, size: 5000, slow: 20 delay 1.020000 0.190000 1.210000 ( 0.516000)
redis input batch: 10, size: 5000, slow: 20 delay 0.080000 0.020000 0.100000 ( 0.072000)
redis input batch: 100, size: 5000, slow: 20 delay 0.030000 0.010000 0.040000 ( 0.025000)
redis input batch: 125, size: 5000, slow: 20 delay 0.020000 0.000000 0.020000 ( 0.011000)
redis input batch: 250, size: 5000, slow: 20 delay 0.010000 0.000000 0.010000 ( 0.003000)
redis input batch: 1000, size: 5000, slow: 20 delay 0.020000 0.000000 0.020000 ( 0.014000)


Sizes reversed
user system total real
redis input batch: 1000, size: 5000, slow: 20 delay 0.080000 0.000000 0.080000 ( 0.026000)
redis input batch: 250, size: 5000, slow: 20 delay 0.020000 0.000000 0.020000 ( 0.011000)
redis input batch: 125, size: 5000, slow: 20 delay 0.040000 0.000000 0.040000 ( 0.023000)
redis input batch: 100, size: 5000, slow: 20 delay 0.040000 0.010000 0.050000 ( 0.023000)
redis input batch: 10, size: 5000, slow: 20 delay 0.060000 0.020000 0.080000 ( 0.060000)
redis input batch: 1, size: 5000, slow: 20 delay 0.680000 0.170000 0.850000 ( 0.484000)

redis input batch: 1000, size: 5000, slow: 20 delay 0.060000 0.010000 0.070000 ( 0.020000)
redis input batch: 250, size: 5000, slow: 20 delay 0.020000 0.010000 0.030000 ( 0.011000)
redis input batch: 125, size: 5000, slow: 20 delay 0.020000 0.000000 0.020000 ( 0.017000)
redis input batch: 100, size: 5000, slow: 20 delay 0.020000 0.010000 0.030000 ( 0.014000)
redis input batch: 10, size: 5000, slow: 20 delay 0.060000 0.010000 0.070000 ( 0.051000)
redis input batch: 1, size: 5000, slow: 20 delay 0.640000 0.170000 0.810000 ( 0.487000)

redis input batch: 1000, size: 5000, slow: 20 delay 0.060000 0.010000 0.070000 ( 0.020000)
redis input batch: 250, size: 5000, slow: 20 delay 0.030000 0.010000 0.040000 ( 0.014000)
redis input batch: 125, size: 5000, slow: 20 delay 0.030000 0.000000 0.030000 ( 0.023000)
redis input batch: 100, size: 5000, slow: 20 delay 0.030000 0.010000 0.040000 ( 0.030000)
redis input batch: 10, size: 5000, slow: 20 delay 0.070000 0.020000 0.090000 ( 0.060000)
redis input batch: 1, size: 5000, slow: 20 delay 0.690000 0.170000 0.860000 ( 0.492000)

redis input batch: 1000, size: 5000, slow: 20 delay 0.080000 0.010000 0.090000 ( 0.035000)
redis input batch: 250, size: 5000, slow: 20 delay 0.000000 0.000000 0.000000 ( 0.006000)
redis input batch: 125, size: 5000, slow: 20 delay 0.020000 0.000000 0.020000 ( 0.016000)
redis input batch: 100, size: 5000, slow: 20 delay 0.030000 0.010000 0.040000 ( 0.024000)
redis input batch: 10, size: 5000, slow: 20 delay 0.070000 0.020000 0.090000 ( 0.067000)
redis input batch: 1, size: 5000, slow: 20 delay 0.770000 0.190000 0.960000 ( 0.564000)

redis input batch: 1000, size: 5000, slow: 20 delay 0.020000 0.000000 0.020000 ( 0.016000)
redis input batch: 250, size: 5000, slow: 20 delay 0.020000 0.010000 0.030000 ( 0.016000)
redis input batch: 125, size: 5000, slow: 20 delay 0.080000 0.020000 0.100000 ( 0.061000)
redis input batch: 100, size: 5000, slow: 20 delay 0.100000 0.020000 0.120000 ( 0.045000)
redis input batch: 10, size: 5000, slow: 20 delay 0.060000 0.020000 0.080000 ( 0.063000)
redis input batch: 1, size: 5000, slow: 20 delay 0.790000 0.170000 0.960000 ( 0.525000)

Larger latency
user system total real
redis input batch: 1000, size: 5000, slow: 2000 delay 2.380000 0.230000 2.610000 ( 0.514000)
redis input batch: 250, size: 5000, slow: 2000 delay 1.760000 0.150000 1.910000 ( 0.357000)
redis input batch: 125, size: 5000, slow: 2000 delay 2.220000 0.200000 2.420000 ( 0.491000)
redis input batch: 100, size: 5000, slow: 2000 delay 2.600000 0.200000 2.800000 ( 0.581000)
redis input batch: 10, size: 5000, slow: 2000 delay 2.640000 0.280000 2.920000 ( 0.752000)
redis input batch: 1, size: 5000, slow: 2000 delay 2.880000 0.430000 3.310000 ( 1.263000)

redis input batch: 1000, size: 5000, slow: 2000 delay 1.150000 0.060000 1.210000 ( 0.191000)
redis input batch: 250, size: 5000, slow: 2000 delay 1.240000 0.060000 1.300000 ( 0.185000)
redis input batch: 125, size: 5000, slow: 2000 delay 1.270000 0.060000 1.330000 ( 0.207000)
redis input batch: 100, size: 5000, slow: 2000 delay 1.270000 0.060000 1.330000 ( 0.194000)
redis input batch: 10, size: 5000, slow: 2000 delay 1.120000 0.100000 1.220000 ( 0.267000)
redis input batch: 1, size: 5000, slow: 2000 delay 1.600000 0.250000 1.850000 ( 0.683000)

redis input batch: 1000, size: 5000, slow: 2000 delay 1.100000 0.050000 1.150000 ( 0.164000)
redis input batch: 250, size: 5000, slow: 2000 delay 1.080000 0.060000 1.140000 ( 0.179000)
redis input batch: 125, size: 5000, slow: 2000 delay 1.020000 0.050000 1.070000 ( 0.169000)
redis input batch: 100, size: 5000, slow: 2000 delay 1.050000 0.060000 1.110000 ( 0.170000)
redis input batch: 10, size: 5000, slow: 2000 delay 0.910000 0.070000 0.980000 ( 0.189000)
redis input batch: 1, size: 5000, slow: 2000 delay 1.470000 0.220000 1.690000 ( 0.618000)
Loading

0 comments on commit 29fa6fa

Please sign in to comment.