Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optionalize ensuring stream connection #35

Merged
merged 1 commit into from
Oct 30, 2015
Merged

Optionalize ensuring stream connection #35

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -119,6 +119,10 @@ It should be in form like "us-east-1", "us-west-2".
Refer to [Regions and Endpoints in AWS General Reference](http://docs.aws.amazon.com/general/latest/gr/rande.html#ak_region)
for supported regions.

### ensure_stream_connection

When enabled, the plugin checks and ensures a connection to the stream you are using by [DescribeStream](http://docs.aws.amazon.com/kinesis/latest/APIReference/API_DescribeStream.html) and throws exception if it fails. Enabled by default.

### http_proxy

Proxy server, if any.
Expand Down Expand Up @@ -196,7 +200,7 @@ Integer, default is 3. When **order_events** is false, the plugin will put multi
records to Amazon Kinesis in batches using PutRecords. A set of records in a batch
may fail for reasons documented in the Kinesis Service API Reference for PutRecords.
Failed records will be retried **retries_on_putrecords** times. If a record
fails all retries an error log will be emitted.
fails all retries an error log will be emitted.

### use_yajl

Expand Down
5 changes: 4 additions & 1 deletion lib/fluent/plugin/out_kinesis.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ class OutputFilter < Fluent::BufferedOutput
# The 'region' parameter is optional because
# it may be set as an environment variable.
config_param :region, :string, default: nil
config_param :ensure_stream_connection, :bool, default: true

config_param :profile, :string, :default => nil
config_param :credentials_path, :string, :default => nil
Expand Down Expand Up @@ -102,7 +103,9 @@ def start
detach_multi_process do
super
load_client
check_connection_to_stream
if @ensure_stream_connection
check_connection_to_stream
end
end
end

Expand Down
2 changes: 2 additions & 0 deletions test/plugin/test_out_kinesis.rb
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -143,6 +143,7 @@ def test_configure_with_more_options
conf = %[
stream_name test_stream
region us-east-1
ensure_stream_connection false
http_proxy http://proxy:3333/
partition_key test_partition_key
partition_key_expr record
Expand All @@ -154,6 +155,7 @@ def test_configure_with_more_options
d = create_driver(conf)
assert_equal 'test_stream', d.instance.stream_name
assert_equal 'us-east-1', d.instance.region
assert_equal false, d.instance.ensure_stream_connection
assert_equal 'http://proxy:3333/', d.instance.http_proxy
assert_equal 'test_partition_key', d.instance.partition_key
assert_equal 'Proc',
Expand Down