Add KafkaPartition and KafkaPartitionKey annotations #387
Conversation
|
|
| if (partitionKey != null) { | ||
| byte[] partitionKeyBytes = Optional.ofNullable(serdeRegistry.pickSerializer(argument)) | ||
| .orElseGet(ByteArraySerializer::new) | ||
| .serialize(finalTopic, parameterValues[i]); |
There was a problem hiding this comment.
I'm not sure how expensive it might be to pick a serializer from the registry on every single call.
If it's deemed expensive then we can/should keep it in a map, but I'm not entirely sure what it should be keyed by. I suppose for the current implementation it would make the most sense to key by Class, but in the end that's more or less what I'd expect the registry implementations to do.
| * <p>When used in consumers, it is populated with the partition that the record was received from.</p> | ||
| * | ||
| * <p>Note that while using {@link KafkaPartition} in the same method as {@link KafkaPartitionKey} | ||
| * will not throw an exception, the outcome of doing so is left unspecified.</p> |
There was a problem hiding this comment.
I'm not sure if this is good behaviour. I'm happy to implement a different one with some advice/guidance
graemerocher
added this to Review in progress
in Micronaut Developers Work Coordination
|
Thanks for the contribution, this looks good |
|
Could you address the compilation errors? |
|
I believe I fixed the Java 8 compilation issue. I'm not entirely sure whether GraalVM's build will be fixed though, I'm not sure why it failed |
|
Looks like one test is flaky, perhaps some concurrency issue and/or misuse of index-based access to queues, but I'm not sure since I don't have a strong grasp of the language. I'll look more into it but any input would be appreciated. |
|
yeah getting async tests right can be challenging it might be best to wait first until the size of the collection is what you expect and then do any index based queries |
Obviously the messages aren't necessarily received in the same order that they are sent, since in these tests they travel in different partitions
|
It was fairly obvious, actually. Should be good now. I appreciate the patience :-) |
|
Thanks for the contribution! |
Hi all,
Although I haven't faced this need in micronaut projects yet, I have found it potentially useful to control partitioning separately from record keys.
This is useful when Keys have specific functional requirement (deduplication, log compaction, etc), but where partitioning of a topic needs to be made on a separate field (often having a one to many relationship with record keys, although that's not a requirement).
This PR brings in the ability to do just that with two new additional annotations:
@KafkaPartitionand@KafkaPartitionKey.