Join GitHub today
GitHub is home to over 50 million developers working together to host and review code, manage projects, and build software together.Sign up
gzip codec is something we totally should have.
To make it work with file input, we'll have to fix how the file input is implemented. We need to do this improvement anyway, but it is a prerequisite for any gzip codec being usable on the file input.
<3 for the idea
@jordansissel thanks! Another thought is possibly a complete flag of some sort.
Basically like a shell exec
Of course you'd have to know this directory of files is static and doesn't have open file handles but that's up to admin to determine.
The file input currently has no concept of "done processing them". Files are assumed to be live streams that live forever, and as a result have no end. Reaching EOF on a log file generally means "wait a while and more data will show up".
Unfortunately, this 'files are live streams' means that folks doing archival or backfilling with old and "complete" logs will be caught without a way to inform Logstash about way to terminate.