Add a general purpose gzip codec #1817

Open
cdenneen opened this Issue Oct 1, 2014 · 13 comments

Comments

Projects
None yet
10 participants
@cdenneen

cdenneen commented Oct 1, 2014

Add gzip codec or file input option for gzipped files

Edit: Add a general purpose gzip codec which can be used in inputs and outputs

@jordansissel

This comment has been minimized.

Show comment
Hide comment
@jordansissel

jordansissel Oct 2, 2014

Contributor

gzip codec is something we totally should have.

To make it work with file input, we'll have to fix how the file input is implemented. We need to do this improvement anyway, but it is a prerequisite for any gzip codec being usable on the file input.

<3 for the idea

Contributor

jordansissel commented Oct 2, 2014

gzip codec is something we totally should have.

To make it work with file input, we'll have to fix how the file input is implemented. We need to do this improvement anyway, but it is a prerequisite for any gzip codec being usable on the file input.

<3 for the idea

@cdenneen

This comment has been minimized.

Show comment
Hide comment
@cdenneen

cdenneen Oct 3, 2014

@jordansissel thanks! Another thought is possibly a complete flag of some sort.
Let's say I have a directory of logs and I point logstash file input at the glob. What would be cool in some cases is "do something when Logstash is done processing them" like

  • gzip them
  • move them to an archive directory
  • delete
  • send email
  • anything

Basically like a shell exec

Of course you'd have to know this directory of files is static and doesn't have open file handles but that's up to admin to determine.

cdenneen commented Oct 3, 2014

@jordansissel thanks! Another thought is possibly a complete flag of some sort.
Let's say I have a directory of logs and I point logstash file input at the glob. What would be cool in some cases is "do something when Logstash is done processing them" like

  • gzip them
  • move them to an archive directory
  • delete
  • send email
  • anything

Basically like a shell exec

Of course you'd have to know this directory of files is static and doesn't have open file handles but that's up to admin to determine.

@jordansissel

This comment has been minimized.

Show comment
Hide comment
@jordansissel

jordansissel Oct 3, 2014

Contributor

The file input currently has no concept of "done processing them". Files are assumed to be live streams that live forever, and as a result have no end. Reaching EOF on a log file generally means "wait a while and more data will show up".

Unfortunately, this 'files are live streams' means that folks doing archival or backfilling with old and "complete" logs will be caught without a way to inform Logstash about way to terminate.

Contributor

jordansissel commented Oct 3, 2014

The file input currently has no concept of "done processing them". Files are assumed to be live streams that live forever, and as a result have no end. Reaching EOF on a log file generally means "wait a while and more data will show up".

Unfortunately, this 'files are live streams' means that folks doing archival or backfilling with old and "complete" logs will be caught without a way to inform Logstash about way to terminate.

@suyograo suyograo changed the title from Gzip files to Add a general purpose gzip codec May 29, 2015

@suyograo suyograo added the new plugin label May 29, 2015

@suyograo

This comment has been minimized.

Show comment
Hide comment
@suyograo

suyograo May 29, 2015

Member

@khornberg gist here: #1895 (comment)

Gzip output:

Member

suyograo commented May 29, 2015

@khornberg gist here: #1895 (comment)

Gzip output:

@yukti-kaura

This comment has been minimized.

Show comment
Hide comment
@yukti-kaura

yukti-kaura Sep 25, 2015

Hello Everyone,

Has this been implemented?

Hello Everyone,

Has this been implemented?

@suyograo

This comment has been minimized.

Show comment
Hide comment
@suyograo

suyograo Sep 25, 2015

Member

@Yukti nope, not implemented. PR welcome :)

Member

suyograo commented Sep 25, 2015

@Yukti nope, not implemented. PR welcome :)

@tan-tan-kanarek

This comment has been minimized.

Show comment
Hide comment
@tan-tan-kanarek

tan-tan-kanarek Feb 18, 2016

This is a quick, not nicely implemented, working alternative:
https://github.com/tan-tan-kanarek/logstash-input-gzfile

This is a quick, not nicely implemented, working alternative:
https://github.com/tan-tan-kanarek/logstash-input-gzfile

@jordansissel

This comment has been minimized.

Show comment
Hide comment
@jordansissel

jordansissel Aug 18, 2016

Contributor

Everytime you comment +1 to this ticket, 25-75 emails are sent to out. Instead, please use Github's "reaction" feature to +1 this issue. It looks like this:

image

I will delete the +1 comments now to disuade this further. I appreciate y'alls eagerness for this feature.

Contributor

jordansissel commented Aug 18, 2016

Everytime you comment +1 to this ticket, 25-75 emails are sent to out. Instead, please use Github's "reaction" feature to +1 this issue. It looks like this:

image

I will delete the +1 comments now to disuade this further. I appreciate y'alls eagerness for this feature.

@jordansissel

This comment has been minimized.

Show comment
Hide comment
@jordansissel

jordansissel Aug 18, 2016

Contributor

I have deleted approximately 15 +1 comments.

Contributor

jordansissel commented Aug 18, 2016

I have deleted approximately 15 +1 comments.

@lmpampaletakis

This comment has been minimized.

Show comment
Hide comment
@lmpampaletakis

lmpampaletakis Nov 24, 2016

Do we have any news about this? Using PIPE which is another official alternative is probably inefficient.

Do we have any news about this? Using PIPE which is another official alternative is probably inefficient.

@honey07

This comment has been minimized.

Show comment
Hide comment
@honey07

honey07 Aug 25, 2017

Hi, how can we use gzip_lines plugin, logstash to read .gz files?

honey07 commented Aug 25, 2017

Hi, how can we use gzip_lines plugin, logstash to read .gz files?

@kunisen

This comment has been minimized.

Show comment
Hide comment
@kunisen

kunisen Mar 23, 2018

Hello Everyone,
Has this been implemented? 😄

As from this comment, not yet?
#1817 (comment)

kunisen commented Mar 23, 2018

Hello Everyone,
Has this been implemented? 😄

As from this comment, not yet?
#1817 (comment)

@dwdii

This comment has been minimized.

Show comment
Hide comment
@dwdii

dwdii May 19, 2018

It's not quite what you all are talking about, but a grassroots codec has recently popped up on RubyGems: https://rubygems.org/gems/logstash-codec-json_gz

It is specific to GZIP'd JSON, but the version I downloaded was working well for me.

dwdii commented May 19, 2018

It's not quite what you all are talking about, but a grassroots codec has recently popped up on RubyGems: https://rubygems.org/gems/logstash-codec-json_gz

It is specific to GZIP'd JSON, but the version I downloaded was working well for me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment