Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Kamelet aws-ddb-sink #819

Merged
merged 2 commits into from
Mar 16, 2022
Merged

Conversation

christophd
Copy link
Contributor

chore: Add Kamelet aws-ddb-sink

Add sink Kamelet for AWS DynamoDB
Use YAKS version 0.9.0-202203140033
Add integration test for aws-ddb-sink Kamelet
Introduce unit tests on camel-kamelets-utils module

The Kamelet automatically converts Json body to proper AWS DynamoDB AttributeValue model. The Kamelet uses new utility class org.apache.camel.kamelets.utils.transform.aws.ddb.JsonToDdbModelConverter for this to happen. The PR tests are expected to fail because of this new utility class that is not available in main-SNAPSHOT JitPack repository yet.

Please have a look at the tests running successful on my fork branch where I was using a separate JitPack repository:
christophd#3

chore: Refactor YAKS tests

Also had to refactor YAKS tests in order to stabilize CI workflows:

  • Use proper yaks-config conventions
  • Use YAKS dump functionality in case of failure
  • Move Camel K installation steps to workflow
  • Apply all current Kamelets at once in workflow prior to running the tests
  • Keep pre/post test scripts as simple as possible
  • Use Camel K 1.8.2 (fixes earthquake-source CI test)

Copy link
Contributor

@oscerd oscerd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot, just minor findings, otherwise this is really good.


This Kamelet expects a JSON as body. The mapping between the JSON fields and table attribute values is done by key, so if you have the input:

'{"username":"oscerd", "city":"Rome"}'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for having me as example :-)

kamelets/aws-ddb-sink.kamelet.yaml Outdated Show resolved Hide resolved
kamelets/aws-ddb-streams-source.kamelet.yaml Outdated Show resolved Hide resolved
kamelets/aws-ddb-streams-source.kamelet.yaml Outdated Show resolved Hide resolved
* In case key and item attribute value maps are identical you can omit the special top level properties completely.
* The converter will map the whole Json body as is then and use it as source for the attribute value map.
*/
public class JsonToDdbModelConverter {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks really nice, thanks for this. I think we should move this kind of utility in the component at some point and recognize when we have a json payload.

@oscerd oscerd added this to the 0.8.0 milestone Mar 15, 2022
- Add sink Kamelet for AWS DynamoDB
- Use YAKS version 0.9.0-202203140033
- Add integration test for aws-ddb-sink Kamelet
- Introduce unit tests on camel-kamelets-utils module
- Use proper yaks-config conventions
- Use YAKS dump functionality in case of failure
- Move Camel K installation steps to workflow
- Apply all current Kamelets at once in workflow prior to running the tests
- Keep pre/post test scripts as simple as possible
- Use Camel K 1.8.2 (fixes earthquake-source CI test)
@oscerd oscerd merged commit 33316ec into apache:main Mar 16, 2022
@oscerd
Copy link
Contributor

oscerd commented Mar 16, 2022

Fixes #792

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

2 participants