Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat(new source): aws_s3 source #4779

Merged
merged 49 commits into from
Nov 10, 2020
Merged

feat(new source): aws_s3 source #4779

merged 49 commits into from
Nov 10, 2020

Commits on Oct 27, 2020

  1. Initial implemntation of aws_3 source 

    A number of outstanding TODOs

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Oct 27, 2020
    Configuration menu
    Copy the full SHA
    2cb29db View commit details
    Browse the repository at this point in the history

  2. Add initial integration test 

    Also brings up-to-date with master changes.

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Oct 27, 2020
    Configuration menu
    Copy the full SHA
    5b47846 View commit details
    Browse the repository at this point in the history

  3. Add initial compression support 

    I ended up dropping a few types that were not already supported
`async_compression`. I'll be creating separate issues to add support for
them.

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Oct 27, 2020
    Configuration menu
    Copy the full SHA
    bf534fc View commit details
    Browse the repository at this point in the history

  4. Add multiline config 

    Also propagate up some errors. I still need to emit them.

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Oct 27, 2020
    Configuration menu
    Copy the full SHA
    7aee881 View commit details
    Browse the repository at this point in the history

  5. Remove resolved comments 

    Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Oct 27, 2020
    Configuration menu
    Copy the full SHA
    1242459 View commit details
    Browse the repository at this point in the history

  6. Import rusoto from new location 

    Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Oct 27, 2020
    Configuration menu
    Copy the full SHA
    d8df611 View commit details
    Browse the repository at this point in the history

  7. Use Arc for credentials 

    Also resolve a couple of TODOs that are stale.

Just create the Resolver pending
#4685

Remove comment about FIFO handling as it seems to be the same from the
consumer side as a standard queue.

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Oct 27, 2020
    Configuration menu
    Copy the full SHA
    45a8525 View commit details
    Browse the repository at this point in the history

  8. Validate S3 event name 

    We only want to process ObjectCreated events

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Oct 27, 2020
    Configuration menu
    Copy the full SHA
    cbcbf6b View commit details
    Browse the repository at this point in the history

  9. Remove a HashMap clone 

    Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Oct 27, 2020
    Configuration menu
    Copy the full SHA
    2feaaf6 View commit details
    Browse the repository at this point in the history

  10. Update some TODOs 

    Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Oct 27, 2020
    Configuration menu
    Copy the full SHA
    9039929 View commit details
    Browse the repository at this point in the history

  11. Add descriptive comment for reading S3 object bodies 

    Also remove stale TODO

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Oct 27, 2020
    Configuration menu
    Copy the full SHA
    5f6bfb7 View commit details
    Browse the repository at this point in the history

  12. Handle overflow of u64 -> i64 

    Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Oct 27, 2020
    Configuration menu
    Copy the full SHA
    cf15d4b View commit details
    Browse the repository at this point in the history

  13. Validate visibility_timeout_secs 

    Ensure that it is a positive i64

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Oct 27, 2020
    Configuration menu
    Copy the full SHA
    de04df8 View commit details
    Browse the repository at this point in the history

  14. Validate that object is in the same region 

    This is a failsafe that I don't think will actually be hit in normal
cases

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Oct 27, 2020
    Configuration menu
    Copy the full SHA
    825224d View commit details
    Browse the repository at this point in the history

  15. Handle empty receipt_handles 

    Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Oct 27, 2020
    Configuration menu
    Copy the full SHA
    a846258 View commit details
    Browse the repository at this point in the history

  16. Raise error if we cannot parse SQS message 

    Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Oct 27, 2020
    Configuration menu
    Copy the full SHA
    6bb4023 View commit details
    Browse the repository at this point in the history

  17. Avoid redundant map 

    Also added comment to unwrap() use

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Oct 27, 2020
    Configuration menu
    Copy the full SHA
    3204b5f View commit details
    Browse the repository at this point in the history

  18. Add internal events 

    Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Oct 27, 2020
    Configuration menu
    Copy the full SHA
    eadf2a5 View commit details
    Browse the repository at this point in the history

  19. Add support for fetching queue owned by another AWS account 

    Also removed future work notes to attach to PR

Removed TODO for validating shutdown as it does appear to be structured
well

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Oct 27, 2020
    Configuration menu
    Copy the full SHA
    cab0b68 View commit details
    Browse the repository at this point in the history

  20. Cast usize in internal_events 

    Resolving remaining TODO. This appears to be the pattern in other
internal_events (specifically prometheus's).

Also fix spelling error

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Oct 27, 2020
    Configuration menu
    Copy the full SHA
    249e049 View commit details
    Browse the repository at this point in the history

  21. Move region configuration up 

    Matches other AWS components.

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Oct 27, 2020
    Configuration menu
    Copy the full SHA
    04e9803 View commit details
    Browse the repository at this point in the history

  22. Set timestamp to S3 object timestamp 

    Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Oct 27, 2020
    Configuration menu
    Copy the full SHA
    93ded11 View commit details
    Browse the repository at this point in the history

  23. Initial aws_s3 source documentation 

    Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Oct 27, 2020
    Configuration menu
    Copy the full SHA
    dbeec99 View commit details
    Browse the repository at this point in the history

  24. Fix docs 

    Will add examples and pull in AWS config

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Oct 27, 2020
    Configuration menu
    Copy the full SHA
    55a720d View commit details
    Browse the repository at this point in the history

  25. Move up cue AWS configuration 

    To apply it to both sinks and sources using AWS.

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Oct 27, 2020
    Configuration menu
    Copy the full SHA
    e0ad883 View commit details
    Browse the repository at this point in the history

  26. Add examples of parsing service logs and sqs url 

    Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Oct 27, 2020
    Configuration menu
    Copy the full SHA
    5b5d01a View commit details
    Browse the repository at this point in the history

  27. Add to CODEOWNERS 

    Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Oct 27, 2020
    Configuration menu
    Copy the full SHA
    7055679 View commit details
    Browse the repository at this point in the history

  28. Fix region configuration to allow for region or endpoint 

    Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Oct 27, 2020
    Configuration menu
    Copy the full SHA
    7cc983b View commit details
    Browse the repository at this point in the history

  29. Rename RegionParseError to RegionParse 

    Matches other errors which do not have an Error suffix.

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Oct 27, 2020
    Configuration menu
    Copy the full SHA
    fa56529 View commit details
    Browse the repository at this point in the history

  30. Move SQS bits to a separate file 

    Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Oct 27, 2020
    Configuration menu
    Copy the full SHA
    1083553 View commit details
    Browse the repository at this point in the history

  31. Move integration tests back 

    These seem to make more sense in aws_s3/mod.rs

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Oct 27, 2020
    Configuration menu
    Copy the full SHA
    a6d81e2 View commit details
    Browse the repository at this point in the history

  32. Clippy 

    Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Oct 27, 2020
    Configuration menu
    Copy the full SHA
    8256bf0 View commit details
    Browse the repository at this point in the history

  33. Scope internal events for aws_s3 source 

    So that sink can be built separately

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Oct 27, 2020
    Configuration menu
    Copy the full SHA
    ef31e3f View commit details
    Browse the repository at this point in the history

Commits on Oct 28, 2020

  1. Merge remote-tracking branch 'origin/master' into aws-s3-source

    @jszwedko
    jszwedko committed Oct 28, 2020
    Configuration menu
    Copy the full SHA
    dbc0cd1 View commit details
    Browse the repository at this point in the history

  2. Bruce PR feedback 

    Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Oct 28, 2020
    Configuration menu
    Copy the full SHA
    0431a51 View commit details
    Browse the repository at this point in the history

  3. Validate S3 event notification version 

    Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Oct 28, 2020
    Configuration menu
    Copy the full SHA
    03a7083 View commit details
    Browse the repository at this point in the history

Commits on Oct 29, 2020

  1. Take ownership of io read error 

    Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Oct 29, 2020
    Configuration menu
    Copy the full SHA
    d88928d View commit details
    Browse the repository at this point in the history

  2. Handle pipeline send failures 

    To avoid acking SQS

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Oct 29, 2020
    Configuration menu
    Copy the full SHA
    a9fdaef View commit details
    Browse the repository at this point in the history

Commits on Nov 3, 2020

  1. Merge remote-tracking branch 'origin/master' into aws-s3-source

    @jszwedko
    jszwedko committed Nov 3, 2020
    Configuration menu
    Copy the full SHA
    c90f865 View commit details
    Browse the repository at this point in the history

  2. PR feedback from Jean 

    Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Nov 3, 2020
    Configuration menu
    Copy the full SHA
    97eeef9 View commit details
    Browse the repository at this point in the history

  3. clippy 

    Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Nov 3, 2020
    Configuration menu
    Copy the full SHA
    bed6acd View commit details
    Browse the repository at this point in the history

Commits on Nov 6, 2020

  1. Readd serde defaults 

    I mistaken thought derivative default would handle this.

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Nov 6, 2020
    Configuration menu
    Copy the full SHA
    bb20901 View commit details
    Browse the repository at this point in the history

Commits on Nov 9, 2020

  1. Use custom version parsing 

    Since semver requires the patch version

Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Nov 9, 2020
    Configuration menu
    Copy the full SHA
    a1f6d05 View commit details
    Browse the repository at this point in the history

  2. Fix integration tests 

    Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Nov 9, 2020
    Configuration menu
    Copy the full SHA
    9b6d00f View commit details
    Browse the repository at this point in the history

Commits on Nov 10, 2020

  1. Merge remote-tracking branch 'origin/master' into aws-s3-source

    @jszwedko
    jszwedko committed Nov 10, 2020
    Configuration menu
    Copy the full SHA
    84d8953 View commit details
    Browse the repository at this point in the history

  2. Use queue_url for config following the pattern of #4675 

    Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Nov 10, 2020
    Configuration menu
    Copy the full SHA
    9a3900a View commit details
    Browse the repository at this point in the history

  3. Fix test error assertion 

    Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Nov 10, 2020
    Configuration menu
    Copy the full SHA
    16800f8 View commit details
    Browse the repository at this point in the history

  4. Fix AWS S3 integration tests 

    Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Nov 10, 2020
    Configuration menu
    Copy the full SHA
    33fdbdb View commit details
    Browse the repository at this point in the history

  5. lint internal event 

    Signed-off-by: Jesse Szwedko <jesse@szwedko.me>
    @jszwedko
    jszwedko committed Nov 10, 2020
    Configuration menu
    Copy the full SHA
    c5b4345 View commit details
    Browse the repository at this point in the history