Skip to content

Commit

Permalink
Add a gzip force decompress flag to deal with Cloudwatch logs in S3
Browse files Browse the repository at this point in the history
bucket. This is a breaking change from S3 input plugin and meant to
be temporary until PR
logstash-plugins/logstash-input-s3#165
merge
  • Loading branch information
lenfree committed Jan 14, 2019
1 parent d7cbf77 commit 3ac2570
Show file tree
Hide file tree
Showing 8 changed files with 505 additions and 61 deletions.
1 change: 1 addition & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
*.gem
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,2 +1,4 @@
## 0.1.0
- Plugin created with the logstash plugin generator
- Forked from input S3 plugin but this have a flag force_gzip_decompress
to deal with Cloudwatch logs in S3. This is meant to be a temporary plugin
2 changes: 1 addition & 1 deletion DEVELOPER.md
Original file line number Diff line number Diff line change
@@ -1,2 +1,2 @@
# logstash-input-packetloop_mys3
# logstash-input-packetloop_s3
Example input plugin. This should help bootstrap your effort to write your own input plugin!
30 changes: 28 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,35 @@
# Logstash Plugin
# Logstash Plugin - packetloop_s3

This is a plugin for [Logstash](https://github.com/elastic/logstash).
This is a fork plugin of S3 input that only reads Cloudwatch logs from S3 for [Logstash](https://github.com/elastic/logstash).
Unfortunately, Cloudwatch logs is already compressed without .gz extension. With current S3 input plugin, it checks whether
file is gzip compressed or not by inspecting file extension and therefore umable able to read Cloudwatch logs streamed via
Firehose or by other means.

This plugin is supposed to be temporary workaround until PR https://github.com/logstash-plugins/logstash-input-s3/issues/165
is merged. Hence, this does not have tests.

It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.

## Usage:
```bash

input {
packetloop_s3 {
access_key_id => "xxx"
secret_access_key => "xxx"$
region => "us-east-1"
delete => true
force_gzipp_decompress => true
codec => cloudwatch_logs {
decompress => false
}
interval => 30
bucket => "bucket"
type => "cloudwatch"
}
}


## Documentation

Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/).
Expand Down
50 changes: 0 additions & 50 deletions lib/logstash/inputs/packetloop_mys3.rb

This file was deleted.

Loading

0 comments on commit 3ac2570

Please sign in to comment.