GOJ-87311 Add producing support in Ziggurat via Kafka Producer #50
Conversation
There was a problem hiding this comment.
@mjayprateek Right now, Ziggurat can read from multiple Kafka topics and from different Kafka servers.
Can we have something producing support as well? Like publishing to different Kafka servers.
Currently, you are mounting the Kafka producer with one Kafka server.
|
Also, Raise the PR against 2.x branch. |
| :enabled [true :bool]}}}}} | ||
| :producer {:default {:bootstrap-servers "localhost:9092" |
There was a problem hiding this comment.
@kartik7153 default just refers to the default KafkaProducer config. Let me know if this needs change.
src/ziggurat/streams.clj
Outdated
| @@ -113,7 +113,7 @@ | |||
| topic-pattern (Pattern/compile origin-topic)] | |||
| (.addStateStore builder (store-supplier-builder)) | |||
| (->> (.stream builder topic-pattern) | |||
| (transform-values topic-entity-name oldest-processed-message-in-s) | |||
| (transform-values topic-entity-name 1) | |||
There was a problem hiding this comment.
By mistake. Removed.
| (deftest send-data-with-topic-and-value-test | ||
| (let [topic "test-topic" | ||
| data "Hello World!!!"] | ||
| (let [future (producer/send topic data)] |
There was a problem hiding this comment.
can this be in the last let only? like:
(let [ topic "test-topic"
data "Hello World!!!"
future (producer/send topic data)]also, what happens if the topic is nil? 🤔
There was a problem hiding this comment.
@riteeks You're right. But, I think these two lets logically divide the code in two parts:
- First let setups the data for calling send()
- Second let simply acts on the result of calling send()
Mixing these two, IMHO, will lead to a less readable code.
send in ziggurat is just a wrapper around KafkaProducer#send. Thus, whatever arguments are passed will just be passed "as they are" to KafkaProducer. That class will be responsible for error handling and exceptions. Ziggurat's code is just a thin layer in-front
of KafkaProducer.
There was a problem hiding this comment.
Mixing these two, IMHO, will lead to a less readable code.
make sense.
send in ziggurat is just a wrapper around KafkaProducer#send. Thus, whatever arguments are passed will just be passed "as they are" to KafkaProducer. That class will be responsible for error handling and exceptions. Ziggurat's code is just a thin layer in-front
of KafkaProducer.
for this, I was just thinking of having a test for bad-path also for the ziggurat's send function. But I am not very sure how much value will this add.
@kartik7153 I agree with you on this. The only glitch is that this will increase the scope of this story. Prima facie, supporting multiple producers needs more thought and some discussion. I'll create a GitHub issue for that and we can take that forward in the next sprint. Let me know if you agree on this. |
Removing unintentional changes from streams.clj
|
Closing this PR as the the changes were meant to be raised against branch 2.x. |
No description provided.