Add support for kafka transactions

[Yexo]

Is your feature request related to a problem? Please describe

I'd like to publish multiple message to Kafka in a single transaction. This makes it much easier to synchronise that action with writing multiple entries to a database and mitigates the problem of some messages being send but others failing.

Describe the solution

Add support for kafka transactions. See https://cwiki.apache.org/confluence/display/KAFKA/Transactional+Messaging+in+Kafka for details.

[mantzas]

@Yexo Hey, AFAIK sarama does not support Transactions. Has this changed?

[Yexo]

I'm not sure, it definitely isn't very well documented: IBM/sarama#1299

At the same time some documentation does hint to it being supported: "TxnOffsetCommit sends a request to commit transaction offsets and returns a response or error".

[mantzas]

@Yexo we have probably to take a deeper look into the support in sarama to determine if it is supported but AFAIK it is not...
There is a function in the producer that allows sending multiple messages but I do not believe it is transactional...

      // SendMessages produces a given set of messages, and returns only when all
	// messages in the set have either succeeded or failed. Note that messages
	// can succeed and fail individually; if some succeed and some fail,
	// SendMessages will return an error.
	SendMessages(msgs []*ProducerMessage) error

[Yexo]

I noticed that function too, but "Note that messages can succeed and fail individually" lead me to believe they're not send in a transaction.

[mantzas]

I agree.
Even if we cannot to transactions due to sarama limitation we can always send the same message multiple times assuming that the consumers are idempotent (and they should be if you ask me).
If the topic is log compacted it will also work since the last message is the surviving one (eventually).
There was also the notion of de-duplication on the producer side but I am not sure about this topic...

[mantzas]

@Yexo hey, i was wondering about this issue. Sice we cannot support transactions with sarama should we close this one?

[mantzas]

Due to inactivity we are closing this for now.