Migrate all Kinesis ITs to embedded-tests#19046
Migrate all Kinesis ITs to embedded-tests#19046kfaraz wants to merge 12 commits intoapache:masterfrom
Conversation
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| schema -> schema | ||
| .withTimestamp(new TimestampSpec("timestamp", null, null)) | ||
| .withParserMap(parserMap) |
Check notice
Code scanning / CodeQL
Deprecated method or constructor invocation
| DataSchema.builder(baseSpec.getSpec().getDataSchema()) | ||
| .withParserMap(parserMap) |
Check notice
Code scanning / CodeQL
Deprecated method or constructor invocation
kfaraz
changed the title
KinesisDataFormatsTest to replace ITKinesisDataFormatsTest
|
Yes, @jtuglu1 , unblocking your PR was the primary motivation behind migrating these tests now. 🙂 Could you help review this PR so that we can merge it off first and then use the tests to verify the changes in your PR? Meanwhile, you could try running the tests in your PR to catch any bugs. |
jtuglu1
left a comment
jtuglu1
left a comment
There was a problem hiding this comment.
LGTM few nits on creating builders for spec/configs like we do in KafkaTuningConfigBuilder, KafkaSupervisorSpecBuilder, and KafkaIOConfigBuilder
| protected SupervisorSpec createSupervisorWithParser(String dataSource, String topic, Map<String, Object> parserMap) | ||
| { | ||
| final KinesisSupervisorSpec baseSpec = createKinesisSupervisorSpec(dataSource, topic, null); | ||
| return new KinesisSupervisorSpec( |
There was a problem hiding this comment.
Can we create KinesisSupervisorSpecBuilder like KafkaSupervisorSpecBuilder?
There was a problem hiding this comment.
Thanks for the suggestion! Let me try including that.
services/src/test/java/org/apache/druid/testing/embedded/TestcontainerResource.java
Show resolved
Hide resolved
2 participants
Changes
StreamIngestResourcewhich is implemented byKafkaResourceandKinesisResourceKinesisResourcethat uses LocalstackKafkaDataFormatsTesttoStreamIndexDataFormatsTestto allow testing different stream typesKinesisDataFormatsTestandKafkaIndexDataFormatsTestITKinesisIndexingServiceSerializedTestwithKinesisFaultToleranceTestSchemaRegistryResourceto allow use with Kinesis as well as KafkaITKinesisIndexingServiceParallelizedTestsince Kinesis does not support multi-supervisor yetThis PR has: