Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor CloudWatch related code and add logs input #43

Merged
merged 14 commits into from
Aug 16, 2017
Merged

Refactor CloudWatch related code and add logs input #43

merged 14 commits into from
Aug 16, 2017

Commits on Aug 9, 2017

  1. Refactor CloudWatch related code 

    - Create a generic Kinesis transport and reader that can be used by
  different inputs
- Create an abstract CloudWatch log data codec
- Rename some flow log classes to more generic names
    @bernd
    bernd committed Aug 9, 2017
    Configuration menu
    Copy the full SHA
    20b4e03 View commit details
    Browse the repository at this point in the history

  2. Add CloudWatch input and codec to read raw logs

    @bernd
    bernd committed Aug 9, 2017
    Configuration menu
    Copy the full SHA
    eef764b View commit details
    Browse the repository at this point in the history

Commits on Aug 10, 2017

  1. Use custom object mapper as we did in the old code

    @bernd
    bernd committed Aug 10, 2017
    Configuration menu
    Copy the full SHA
    494182d View commit details
    Browse the repository at this point in the history

  2. Add payload examples and ignore unknwon properties

    @bernd
    bernd committed Aug 10, 2017
    Configuration menu
    Copy the full SHA
    5c4c36a View commit details
    Browse the repository at this point in the history

  3. Make kinesis consumer less noisy

    @bernd
    bernd committed Aug 10, 2017
    Configuration menu
    Copy the full SHA
    dc02d17 View commit details
    Browse the repository at this point in the history

  4. Update to the latest AWS SDK and Kinesis library

    @bernd
    bernd committed Aug 10, 2017
    Configuration menu
    Copy the full SHA
    d421fb1 View commit details
    Browse the repository at this point in the history

  5. Use unique application name per Kinesis stream 

    The Kinesis client is using a DynamoDB table under the hood to track
shard offsets. When a consumer is using the same application name for
two different Kinesis streams, checkpointing will fail.
    @bernd
    bernd committed Aug 10, 2017
    Configuration menu
    Copy the full SHA
    b5318af View commit details
    Browse the repository at this point in the history

  6. Periodically checkpoint the Kinesis stream 

    This ensures that we are getting old records after a Graylog server
restart instead of starting with LATEST on each restart.
    @bernd
    bernd committed Aug 10, 2017
    Configuration menu
    Copy the full SHA
    f9c8e34 View commit details
    Browse the repository at this point in the history

  7. Add second constructor to AWSAuthProvider and simplify conditions

    @bernd
    bernd committed Aug 10, 2017
    Configuration menu
    Copy the full SHA
    f9b45ba View commit details
    Browse the repository at this point in the history

  8. Pass auth provider into InstanceLookupTable to allow different methods 

    Also use AmazonEC2Client builder instead of deprecated constructor.
    @bernd
    bernd committed Aug 10, 2017
    Configuration menu
    Copy the full SHA
    ddcd318 View commit details
    Browse the repository at this point in the history

  9. Inject InstanceLookupTable and bind it as singleton

    @bernd
    bernd committed Aug 10, 2017
    Configuration menu
    Copy the full SHA
    bcbda91 View commit details
    Browse the repository at this point in the history

Commits on Aug 14, 2017

  1. Switch tests to junit

    @bernd
    bernd committed Aug 14, 2017
    Configuration menu
    Copy the full SHA
    592c791 View commit details
    Browse the repository at this point in the history

  2. Use default config in instance name lookup processor 

    - Remove now unused "AWSPluginConfiguration#isComplete()"
- Log a warning if the lookup processor is enabled but no regions are
  configured
    @bernd
    bernd committed Aug 14, 2017
    Configuration menu
    Copy the full SHA
    f3bf4dd View commit details
    Browse the repository at this point in the history

Commits on Aug 16, 2017

  1. Rename inputs to remove "CloudWatch" from the names

    @bernd
    bernd committed Aug 16, 2017
    Configuration menu
    Copy the full SHA
    5e35da9 View commit details
    Browse the repository at this point in the history