Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add S3 input to retrieve logs from AWS S3 buckets #12640

Merged
merged 59 commits into from
Aug 2, 2019
Merged

Add S3 input to retrieve logs from AWS S3 buckets #12640

merged 59 commits into from
Aug 2, 2019

Commits on Aug 2, 2019

  1. Add S3 input to retrieve logs from AWS S3 buckets

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    772de35 View commit details
    Browse the repository at this point in the history

  2. run mage fmt update

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    a98e05c View commit details
    Browse the repository at this point in the history

  3. update doc

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    9b11cb0 View commit details
    Browse the repository at this point in the history

  4. Change bucketName to bucket_name

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    9d5a624 View commit details
    Browse the repository at this point in the history

  5. Run make update

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    252e2d7 View commit details
    Browse the repository at this point in the history

  6. remove receiveMessages function and put goroutines directly in Run

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    cf23865 View commit details
    Browse the repository at this point in the history

  7. Added several methods to read resp.body to string

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    b8d017a View commit details
    Browse the repository at this point in the history

  8. Use bufio.NewReader to read s3 GetObject response body

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    6ae5e46 View commit details
    Browse the repository at this point in the history

  9. Update changelog

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    0c692bd View commit details
    Browse the repository at this point in the history

  10. Change to push individual event when it's ready

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    1fdf6fe View commit details
    Browse the repository at this point in the history

  11. add getAWSCredentials function

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    0f44668 View commit details
    Browse the repository at this point in the history

  12. Add shared_config_profile option and read from ~/.aws/credentials

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    605e51c View commit details
    Browse the repository at this point in the history

  13. Update aws credentials with credential_profile_name

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    9c4aaee View commit details
    Browse the repository at this point in the history

  14. fix wg.wait and change log.offset to be byte size instead of line number

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    87b2307 View commit details
    Browse the repository at this point in the history

  15. ChangeMessageVisibility when process one message longer than 1/2 visi… 

    …bility timeout
    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    68f1bf7 View commit details
    Browse the repository at this point in the history

  16. Add changeMessageVisibility and check time spend on reading log file

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    0a16db3 View commit details
    Browse the repository at this point in the history

  17. rebase

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    6121b57 View commit details
    Browse the repository at this point in the history

  18. Move select into a separate go routine

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    aa70bd8 View commit details
    Browse the repository at this point in the history

  19. Remove bucket_names from config and add missing return

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    cbf7f60 View commit details
    Browse the repository at this point in the history

  20. Fix unit tests

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    561e6bc View commit details
    Browse the repository at this point in the history

  21. Add VisibilityTimeout into config with default to be 5min

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    5d52a84 View commit details
    Browse the repository at this point in the history

  22. Add channel to close in Stop()

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    339b2a5 View commit details
    Browse the repository at this point in the history

  23. Add id for each event

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    991c29d View commit details
    Browse the repository at this point in the history

  24. add sqsMessage struct to decode resp body from sqs

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    12ee91c View commit details
    Browse the repository at this point in the history

  25. Change event id to hash + offset

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    af21325 View commit details
    Browse the repository at this point in the history

  26. Add error channel to handleS3Objects

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    e841d36 View commit details
    Browse the repository at this point in the history

  27. Move deleteMessage after done

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    8744cbf View commit details
    Browse the repository at this point in the history

  28. Replace done channel with errC channel

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    9fb151e View commit details
    Browse the repository at this point in the history

  29. Update documentation

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    824689a View commit details
    Browse the repository at this point in the history

  30. Fix unit test

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    3f4bd22 View commit details
    Browse the repository at this point in the history

  31. Fix unit test

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    1314fa3 View commit details
    Browse the repository at this point in the history

  32. Move sendKeepAlive into separate function

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    ffe44ad View commit details
    Browse the repository at this point in the history

  33. separate processMessage into a function

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    bc480d2 View commit details
    Browse the repository at this point in the history

  34. Add bounded message queue

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    62a7b30 View commit details
    Browse the repository at this point in the history

  35. Uncomment meta in event

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    1cedb02 View commit details
    Browse the repository at this point in the history

  36. Revert "Uncomment meta in event" 

    This reverts commit 742c25a69dff2b434f338f40248d53c8858217d6.
    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    47d378b View commit details
    Browse the repository at this point in the history

  37. Revert "Add bounded message queue" 

    This reverts commit a53de38c7912b2bbdd3eea72b2316e330c560a75.
    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    dafbea7 View commit details
    Browse the repository at this point in the history

  38. add workerOnce to only start Run() once

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    566c82a View commit details
    Browse the repository at this point in the history

  39. Rebase to use x-pack libbeat common GetAWSCredentials

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    1c46b03 View commit details
    Browse the repository at this point in the history

  40. Change defaultConfig to be a function

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    5a81235 View commit details
    Browse the repository at this point in the history

  41. Move Validate into config.go

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    6b06d0f View commit details
    Browse the repository at this point in the history

  42. Add channelContext

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    efa648f View commit details
    Browse the repository at this point in the history

  43. Add s3Context

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    a271dc1 View commit details
    Browse the repository at this point in the history

  44. Remove parallel for handleS3Objects

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    da21e06 View commit details
    Browse the repository at this point in the history

  45. Uncomment Meta in event

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    d733b3a View commit details
    Browse the repository at this point in the history

  46. fix unit test

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    8943d13 View commit details
    Browse the repository at this point in the history

  47. Replace select case with p.context.Err() == nil

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    bce8452 View commit details
    Browse the repository at this point in the history

  48. rebase and update to use new connector

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    c916745 View commit details
    Browse the repository at this point in the history

  49. Add end to end ACK in connector

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    c3a26ae View commit details
    Browse the repository at this point in the history

  50. Ignore aws ErrCodeRequestCanceled

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    59bcd61 View commit details
    Browse the repository at this point in the history

  51. Run mage fmt update

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    e0e83d9 View commit details
    Browse the repository at this point in the history

  52. Change ref to start from 1

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    775c628 View commit details
    Browse the repository at this point in the history

  53. Change to newS3BucketReader

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    6558743 View commit details
    Browse the repository at this point in the history

  54. Add waitgroup

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    0ce8dc6 View commit details
    Browse the repository at this point in the history

  55. Add documentation for setting up sqs s3 and manual testing

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    610d854 View commit details
    Browse the repository at this point in the history

  56. move manual testing into a separate doc

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    837f16a View commit details
    Browse the repository at this point in the history

  57. Add s3 input doc into filebeat-options

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    13dd71a View commit details
    Browse the repository at this point in the history

  58. rebase and add filebeat-s3-input-config

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    87be1a9 View commit details
    Browse the repository at this point in the history

  59. Fix link in metricbeat doc

    @kaiyan-sheng
    kaiyan-sheng committed Aug 2, 2019
    Configuration menu
    Copy the full SHA
    e132c19 View commit details
    Browse the repository at this point in the history