Should have native .NET output binding support

[ryancrawcour]

Must be able to write output to a Kafka topic using output bindings

• Output binding
• Expose important configuration options for producers
• Add unit tests

[brandonh-msft]

To help in somebody picking this up, can we define "important" in this issue?

[fbeltrao]

My suggestions would be for you to take a look at the librdkafka and Confluent.Kafka configuration docs and propose what settings should be exposed in the function.

I would start with settings that affect performance and throughput. Settings about max message size could be important too, in case we need to support topics with large messages.

At the end we should add the newly exposed options in our documentation section.

Does it help @brandonh-msft ?

[brandonh-msft]

perfect, thanks

[ryancrawcour]

@brandonh-msft is this* something you're going to tackle?

• take a look at the librdkafka and Confluent.Kafka configuration docs and propose what settings should be exposed in the function.
  At the end we should add the newly exposed options in our documentation section.

[brandonh-msft]

you can put this on my backlog but i'm tackling #44 first

[ryancrawcour]

Ok, assigned to you

[brandonh-msft]

K so right now, looking at Global configuration properties in librdkafka docs I'm thinking:

property	proposed name	description
message.max.bytes	MaxMessageBytes	Maximum transmit message size.
enable.auto.commit	AutoCommit	Automatically and periodically commit offsets in the background.
auto.commit.interval.ms	AutoCommitIntervalMs	The frequency in milliseconds that the consumer offsets are commited (written) to offset storage. (0 = disable)
batch.num.messages	BatchSize	Maximum number of messages batched in one MessageSet.

It's not clear to me if some of those are implementations we'd have to code up (autocommit, for instance) or if it's something we specify on the msg that's sent and the protocol/receiver takes care of it. Would be good to get some clarity there if you guys know off the top of your head.

Nothing else is standing out as either applicable to sending messages, or particularly useful.

[fbeltrao]

Are those consumer or producer related settings? It is unclear to me.
Regarding commit configuration: we are doing manual commits, therefore enable.auto.commit and auto.commit.interval.ms will not be used/exposed.

[brandonh-msft]

yeah TBH i'm not sure either; i agree it's not super clear in the docs whether it's producer related or not. I just kinda guessed.

So now considering just:

property	proposed name	description
message.max.bytes	MaxMessageBytes	Maximum transmit message size.
batch.num.messages	BatchSize	Maximum number of messages batched in one MessageSet.

[ryancrawcour]

Ok. let's start with these. and when people request others or want extra features that are unblocked by exposing others, then we can go add them.

[fbeltrao]

Not sure if we should expose:
enable.idempotence

source: https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md

[ryancrawcour]

Can someone please update the docs and send a PR for bullet point 2 then. sounds like it's done.

[brandonh-msft]

After some more digging found these are all the knobs even possible to set w/ the Kafka .Net SDK - do we just expose them all?

[fbeltrao]

We already have so many, adding just for the sake of having all might in turn cause the configuration to be over complicated.

[brandonh-msft]

fair enough - it doesn't look like, though, i can tweak the pieces I called out earlier after looking at what the .Net SDK exposes in ProducerConfig - which of the above do you think we'd want to expose, then? Perhaps just the EnableIdempotence? I can see value in the Timeouts, BatchNumMessages, and MessageSendMaxRetries, though.

[fbeltrao]

I like your proposal. For Idempotence we need to check how we handle errors in our producer

[ryancrawcour]

Yeah, go for the simplest config that enables the majority of scenarios. We can start exposing more config options as developers using this start requesting additional features. Too many knobs and dials get confusing

[ryancrawcour]

@brandonh-msft What's the status of this? In progress? Or complete?