Skip to content

CAMEL-16594 Support more than one active shard in DynamoDB stream.#6151

Closed
PyvesB wants to merge 2 commits into
apache:mainfrom
PyvesB:ddb-streams-tree
Closed

CAMEL-16594 Support more than one active shard in DynamoDB stream.#6151
PyvesB wants to merge 2 commits into
apache:mainfrom
PyvesB:ddb-streams-tree

Conversation

@PyvesB
Copy link
Copy Markdown
Contributor

@PyvesB PyvesB commented Sep 23, 2021

  • Make sure there is a JIRA issue filed for the change (usually before you start working on it). Trivial changes like typos do not require a JIRA issue. Your pull request should address just this issue, without pulling in other changes.
  • Each commit in the pull request should have a meaningful subject line and body.
  • If you're unsure, you can format the pull request title like [CAMEL-XXX] Fixes bug in camel-file component, where you replace CAMEL-XXX with the appropriate JIRA issue.
  • Write a pull request description that is detailed enough to understand what the pull request does, how, and why.
  • Run mvn clean install -Psourcecheck in your module with source check enabled to make sure basic checks pass and there are no checkstyle violations. A more thorough check will be performed on your pull request automatically.
    Below are the contribution guidelines:
    https://github.com/apache/camel/blob/main/CONTRIBUTING.md

This pull request achieves the following:

  • drops support for the iteratorType and sequenceNumberProvider parameters. My reasoning as to why these don't make much sense can be found in CAMEL-16594.
  • adds support for a new streamIteratorType parameter which can have two values, FROM_LATEST and FROM_START.
  • models DynamoDB stream shards as a tree rather than a list and adds some unit tests to this shard data structure.
  • reworks ShardIteratorHandler to operate on a tree structure and adds unit tests to it.
  • reworks Ddb2StreamConsumer to operate on multiple returned shard iterators.

Copy link
Copy Markdown
Contributor

@oscerd oscerd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Thanks a lot for the work. @davsclaus can you have a look too? Thanks

@davsclaus
Copy link
Copy Markdown
Contributor

Can you add a note to the 3.12 upgrade guide about the option renamed
https://github.com/apache/camel/blob/main/docs/user-manual/modules/ROOT/pages/camel-3x-upgrade-guide-3_12.adoc

@oscerd
Copy link
Copy Markdown
Contributor

oscerd commented Sep 23, 2021

I'm going to merge this today. I think it would be really nice to have a blog post about how to use this and maybe a bit more love to this component.

@oscerd
Copy link
Copy Markdown
Contributor

oscerd commented Sep 23, 2021

Merged. Thanks!

@oscerd oscerd closed this Sep 23, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants