allow Writer to publish to any topic

[savaki]

Now that Message supports Topic, it seems like there's no reason not to allow WriteMessages to publish to more than one topic. Until the next major release, we could leave the public api alone and allow topics passed directly on the message to override the default topic.

Thoughts?

[achille-roussel]

I'm not against this change, my only concern is this will require quite a bit of a rework of the internals of Writer which would increase the complexity of the code, since we'd have to maintain a connection pool for each topic. There will be limitations tho since a Writer still has only one Balancer attached, it means the balancing logic will have to be the same for every topics that the writer publishes to.

Also if we take this route, should we also allow direct addressing of partitions when writing to a topic (effectively bypassing the balancer)?

In practice it seems to me that programs rarely need to publish to multiple topics, and if they do a map[string]*kafka.Writer may be more practical.

Feel free to chime in if you have examples of use cases that would clearly benefit from having the Writer support publishing to multiple topics.

[savaki]

While I like the idea of maintaining a connection pool per broker, a smaller step might be just to expand writeProduceRequestV2 to publish to more than one topic. The underlying kafka protocol supports it, but it looks like for simplicity, it's been limited to just 1.

https://github.com/segmentio/kafka-go/blob/master/write.go#L216

As for only a single balancer, I think it's quite common for organizations to use a just a single balancer. However, if multiple balancers are needed, multiple writers could be created.

[savaki]

Also if we take this route, should we also allow direct addressing of partitions when writing to a topic (effectively bypassing the balancer)?

We don't have any use cases where we'd ever directly assign to a partition without a balancer. Most cases that I can think of that would desire direct routing, like QoS and priority routing, seems like they'd be better served by being put in a balancer anyway.

[savaki]

In practice it seems to me that programs rarely need to publish to multiple topics, and if they do a map[string]*kafka.Writer may be more practical.

We have a number of services that publish to multiple topics. Some examples include:

• database change data capture where each table is published to a separate topic
• data extraction. a service might extract N types of data from a single provider e.g. Salesforce. Each data type would be published onto a different topic.
• event sourcing - a single service might be responsible for more than one aggregate. each aggregate would have its own topic.

While it's possible to use map[string]*kafka.Writer for such use cases, it feels like it pushes complexity down to the users for common use cases for something that seems like a reasonably small change to the writer.

[achille-roussel]

Sounds good, I just wanted to be sure we were not making te change “because we can”, seems like it would serve you guys well to have this as a feature.

One more question i have, should we allow it only if no topic has been set on the Writer when it was created? Or should we use the configured topic as a default? I feel like the former is less error prone, let me know what you think.

[savaki]

I like the former as well. The topic should either be set at initializing or per message, but not both.

[jonasrmichel]

I'm interested in this feature as well. Is there still momentum?

[bwinata]

I second this feature request.

Use case:
We are currently interested in using this Go library for our purpose to route data from MQTT into our Kafka cluster. In short - we have a variety of IoT devices which are publishing data on various MQTT topics which are mapped to associated Kafka topics. For this, I'm writing a router using said API. However, because the underlying connection dictates the topic at instantiation, I've had to create a variation of map[string]*kafka.Writer on the app layer to support this. It would be nice for this library to have this feature natively built in.

[larryzhao]

I second this feature request as well.

We send a couple kinds of events to kafka ( encoded in avro ), and we need to send each kind to a specific topic.

[achille-roussel]

Seems like this is a popular feature request, anybody willing to contribute a change to support it would be very welcome, but until we have a need for this at Segment we probably won't have time to dedicate to building it.

[achille-roussel]

@stevevls is going to take a look at this.

[jainnidhi703]

I second this feature request.

Use case:
We are currently interested in using this Go library for our purpose to route data from different services to different topics.

[dominicbarnes]

This has been added to the 0.4 branch. Please open a new issue if you have any trouble using the new functionality.