New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We鈥檒l occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Kinesis input plugin #10

Closed
wants to merge 1 commit into
base: master
from

Conversation

Projects
None yet
3 participants
@winebarrel
Contributor

winebarrel commented Dec 14, 2014

I've implemented the input plug-in.
Please add it if there is no problem. 馃檱

Exapmle configuration:

<source>
  type kinesis
  stream_name my-stream
  tag kinesis.data
  shard_iterator_type AFTER_SEQUENCE_NUMBER
  shards {"shardId-000000000000":"49545736698179024714137668171499632182578095385061883906"}
  db /var/tmp/kinesis.db
  #json_data true
</source>

Parse "data" attr as JSON if json_data is true:

put_record(data: '{"key1": "val1", "key2": "val2"}'
-> get_record:
{ ... "key1": "val1", "key2": "val2", ...}

see https://github.com/winebarrel/aws-fluent-plugin-kinesis/blob/7de7972ec569e22c7c59b4a3ba1ced55f21f37a1/test/plugin/test_in_kinesis.rb#L178


Use placeholder if tag is kinesis.${foo}

<source>
  tag kinesis.${foo}
  ...
</source>
put_record(data: '{"foo": "bar"}'
-> emit recoed tag: kinesis.bar 
@yuta-imai

This comment has been minimized.

Show comment
Hide comment
@yuta-imai

yuta-imai Dec 15, 2014

Contributor

Thank you! I will look into this. Please wait a while.

Contributor

yuta-imai commented Dec 15, 2014

Thank you! I will look into this. Please wait a while.

@yuta-imai yuta-imai self-assigned this Jan 9, 2015

@jmcle

This comment has been minimized.

Show comment
Hide comment
@jmcle

jmcle Mar 12, 2015

I've been using this input plugin for a couple weeks now and can confirm that it is working well. Would love to see this merged.

jmcle commented Mar 12, 2015

I've been using this input plugin for a couple weeks now and can confirm that it is working well. Would love to see this merged.

@yuta-imai

This comment has been minimized.

Show comment
Hide comment
@yuta-imai

yuta-imai Aug 14, 2015

Contributor

Hi @winebarrel , I apologize leaving this long time..
At this point of time, please let me close this PR.

I cannot be 100% sure that db approach could provide good experience for variety of use cases, especially in environment with shard # is frequently updated.

However, I think this works well in fixed shard # environment, and you already have bunch of users with fluent-plugin-kinesis-alt. I hope you go on develop in that project.

Sincerely.

Contributor

yuta-imai commented Aug 14, 2015

Hi @winebarrel , I apologize leaving this long time..
At this point of time, please let me close this PR.

I cannot be 100% sure that db approach could provide good experience for variety of use cases, especially in environment with shard # is frequently updated.

However, I think this works well in fixed shard # environment, and you already have bunch of users with fluent-plugin-kinesis-alt. I hope you go on develop in that project.

Sincerely.

@yuta-imai yuta-imai closed this Aug 14, 2015

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment