[APEXMALHAR-2086] Kafka output operator: 0.9.0 #298
Conversation
|
Please comment on the overall approach |
| import org.apache.kafka.clients.producer.ProducerRecord; | ||
| import com.datatorrent.api.DefaultInputPort; | ||
|
|
||
| public class KafkaSinglePortOutputOperator<K,V> extends AbstractKafkaOutputOperator |
There was a problem hiding this comment.
Can you make this evolving?
sandeshh
changed the title
| } catch (IOException e) { | ||
| throw new RuntimeException(e); | ||
| } catch (InterruptedException e) { | ||
| e.printStackTrace(); |
There was a problem hiding this comment.
Don't swallow exception here
|
Why do we need this operator, how is it different from the existing operator? |
|
@tweise There is no 0.9 output operator. |
sandeshh
changed the title
sandeshh
changed the title
|
@siyuanh I have addressed the review comments that we discussed. |
sandeshh
changed the title
|
@siyuanh that does not answer my question. If added a comment to the JIRA. |
| /** | ||
| * Kafka output operator with exactly once processing semantics under certain conditions., | ||
| * | ||
| * This operator uses *Key* to distinguish the messages written by particular instance of the Output operator. |
There was a problem hiding this comment.
I think this needs more explanation. What makes it exactly once, are there other restrictions or assumptions?
|
@sandeshh Test is failing, can you change the log level and run it again so it won't fail because of too much logs |
|
@siyuanh Travis build is failing as it reached log limit of 4mb. Here is the log file, What do you recommend? Disable the Kafka unit tests? |
|
@sandeshh Looks like the travis doesn't pick up the log4j settings. Can you try to run the travis build on your local machine? |
| @@ -47,3 +47,4 @@ log4j.logger.org.apache.apex=INFO | |||
| log4j.logger.org.apacke.kafka=WARN | |||
There was a problem hiding this comment.
Typo here, this is the reason why log4j doesn't pick up lines after this
1. Abstract Base class
2. Kafka Output operator
3. Exactly Once output operator
Key in the Kafka message is used by the operator to track the tuples written by it.
Kafka output exactly once operator and the regular output operator.