Optional flag to decompress file without having 'gz' extention

[gurupritmukhraiya]

We are using S3 input plugin to pull data from S3, we are using kinesis firehose to push data to S3, which does not adds 'gz' extension to the files, but data is in zipped format.

S3 input plugin is pulling data from S3 it can only decompress the data which has gz as extension since while code walk through we we notice there is a condition checking 'gz' as extension then only it decompress the file data.

We do not want to maintain the custom plugin at our end just for small change.

I just wanted to ask can we add a flag over there, which can be set as configuration and used with that condition, so that even files which use can set intentionally to decompress files does not have 'gz' extension or with any extension.

Can you provide such feature as same plugin or can we contribute to this input plugin, if yes how and what is the process for the same?

[gurupritmukhraiya]

Hi,

Just wanted to have an update on this,

1. Did you feel this issue is a candidate to be implemented in coming updates?
2. If yes, How long it would be available so that we can utilize this?

It would be great if this can be provided ASAP, since we would not need to implement patchy things at our end.
Thanks!!!

[blangenfeld]

Hey @gurupritmukhraiya , I've opened PR #168. Maybe it'll help you.

[gurupritmukhraiya]

Thank You Brian!!! May I know till when this feature would be available? - Guruprit

…

On Fri, Nov 30, 2018 at 2:58 AM Brian Langenfeld ***@***.***> wrote: Hey @gurupritmukhraiya <https://github.com/gurupritmukhraiya> , I've opened PR #168 <#168>. Maybe it'll help you. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#165 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AVr9HPhZ-OL6kBwLHRzRvY760c7LdBVPks5u0FFygaJpZM4W4q3O> .

-- Regards, Guruprit Mukhraiya 9890728332

[blangenfeld]

It's up to the maintainers to determine whether/when the PR gets merged.

[blangenfeld]

@jsvd , is there anything more that must be done to get the PR reviewed & merged in?

The build "failed", but it seems to be because Travis choked on something unrelated. Not sure how to proceed here...

[lenfree]

Any update on this? I am in need of this where Cloudwatch logs is already compressed before Firehose stream to S3. ATM, I am running my version of S3 input but as much as possible I would like not to maintain my own.