Need to Support Kafka Streams APIs

[liu-hongwei]

Description

Start from version 0.10.0.X, Kafka itself supports the Streams APIs, however, seems except the Java version clients support this feature, the .net client still does not support Kafka Streams APIs.

enabling this feature in .net client will be really helpfully when we use Kafka on .net platform , especially for real time data processing.

How to reproduce

not bug, but feature implementation.

Checklist

Please provide the following information:

• Confluent.Kafka nuget version:
• Apache Kafka version:
• Client configuration:
• Operating system:
• Provide logs (with "debug" : "..." as necessary in configuration)
• Provide broker log excerpts
• Critical issue

[mhowlett]

Unfortunately, we don't have near term plans to implement a Kafka Streams API in .NET (it's a very large amount of work) though we're happy to facilitate other efforts to do so.

Confluent have recently launched KSQL, which effectively allows you to use the Streams API without Java and has a REST API that you can call from .NET. It's more limited, but perhaps it satisfies your use case.

https://www.confluent.io/blog/ksql-open-source-streaming-sql-for-apache-kafka/

[pavel-agarkov]

That contradicts with what is written earlier:

Supporting languages beyond Java

One of the nice things about this approach is that the amount of code is quite small and we think it should be possible to build out a really nice implementation in a number of major languages and still have them feel like natives of their ecosystem. We’re focusing on getting the normal clients in shape there first, but we’ll pick up working on stream processing support shortly after.

[alessandroabba]

Still no intention to publish a client for the kafka streams api for .net?

[mhowlett]

not at this stage. it's a lot of work and we need to implement more foundational things like the AdminClient and exactly-once-semantics first.

[raskolnikoov]

Any update on this nowdays? Would be really cool feature to use in .Net-land :)

[mhowlett]

we're still focussed on getting to feature parity with the java producer/consumer (admin client currently in progress, and work on exactly once about to begin).

at this stage, take a look at KSQL (which is production ready in Confluent Platform 4.1, just released), to see if it meets your needs.

[smitty-codes]

@mhowlett Just making sure I understand your response - Confluent is focused on making sure the existing .net client library (which supports producer/consumer only) is equivalent to the Java version, then Kafka Streams will come? Or you guys are already working on Kafka Streams for .net? Thanks

[mhowlett]

(as much as i would like to be working on this) we don't currently have a timeframe for any non-java streams implementation.

[smitty-codes]

Bummer. Thanks for the follow up.

[ghost]

@mhowlett Any news/update about this? :)

[mhowlett]

keep the up-votes coming and I'll periodically ping the relevant decision makers with this thread :-)

[srinathrangaramanujam]

Any Updates ?

[RTodorov]

Hi @mhowlett ! A few months have passed, just checking if there have been any changes to this topic. Thank you!

[mhowlett]

We don't have any definite plans for streams in .net., but 72 upvotes ... I'll make sure product are aware of this.

It's worth noting that KSQL now has UDFs and UDAFs. I believe we are going to have a good story here for non-java languages relatively soon, which would cover-off many of the use cases of native streams in other languages. KSQL is worth checking out.

https://www.confluent.io/blog/build-udf-udaf-ksql-5-0

[almazik]

I'm eagerly waiting for .NET implementation of KStreams.

I tried to use KSQL for simple data processing/filtering, but I constantly hit roadblocks:
Cannot escape reserved words as column names confluentinc/ksql#677 (or ttps://github.com/confluentinc/ksql/issues/1213, same issue)
Cannot parse ISO-formatted timestamp confluentinc/ksql#1439 (fixed, waiting for release)
Cannot map values confluentinc/ksql#620
Cannot (could not find how to) delete rows from a table based on stream data (e.g. "remove a row if the field "status = 'deleted'") - just created an issue confluentinc/ksql#1945

That's just from our last attempt to add KSQL to our technology stack

[dengia]

Can we push/help in any way for .net kafka-streams? Kafka usage would definitely expand in .net world with streams implemented ...

[SquaLLpt]

Any news about this?

[caryhocker]

New year, new hopes, any news to report on this request?

[smitty-codes]

In the confluent 5.2 release, I see this:

"Confluent Platform 5.2 proudly introduces librdkafka 1.0. This is a big milestone, because it brings this popular client library closer to parity with the Java client for Kafka."

I assume when we're past close and librdkafka is in parity with Java implementation, then we'll see streams for .NET client soon after?

[fred070]

Any updates on Streaming library? It has 122 votes as of this writing, reckon this feature can be added in the near future?

[budibong]

I am learning KAFKA for my green field project and very excited about streaming processing with KAFKA STREAM API, but when i know it only for java, I am confused if I must continue to adopt KAFKA or not :(

[k-tomak]

@mhowlett any news?
144 votes

[mhowlett]

the basic non-java clients are still playing catch-up - achieving feature parity here is our priority (and a prerequisite for a streams implementation). in the meantime, KSQL development is being pushed very heavily as a language agnostic alternative for many use cases.

related, i've been playing with some ideas for stream processing in my spare time independent of confluent, however this is just a start, not for production use, no guarantees etc. etc.

https://www.matthowlett.com/2019-06-02-stream-processing-pt1-what-and-why.html
https://github.com/mhowlett/howlett-kafka-extensions

[ghost]

@mhowlett Thanks for your preliminary efforts on this. Perhaps a new issue should be opened to actually track Streams API support, rather than encouraging users who still want this feature to keep necroing this 2-yr old issue?

[ahernandez-allegion]

Is there a reason this issue was closed or was it moved to another issue? Any updates on this?

[mhowlett]

it was closed 2 years ago, i'll open it again as a place to accumulate votes.

this is unlikely to be prioritized in the short term - LOE is too high to match the java streams api and there are too many other pressing things.

Worth mentioning that when transactions are out, I think it will be feasible to roll your own stream processing framework with exactly once semantics that gets you 80% of the way there (some commentary on that in previously referenced links).

[mhowlett]

interesting project: https://github.com/LGouellec/kafka-stream-net

[lemkepf]

@mhowlett We have a large .net project and my very first use case for Kakfa was dynamic streams.... alas my hopes are dashed. Any update on where this might be at in the priority list?

[LGouellec]

@lemkepf Hi, my project https://github.com/LGouellec/kafka-stream-net aims to offer a stream processing framework around kafka like Kafka Stream.
The goal is to provide the same functionality. if you want contribute it, i’m Ok ;)

[iam-gopi]

Do we have any update on this?
I've a .net framework 4.7.2 REST API Project where I'm trying to implement confluent Kafka there. But, unfortunately it's not working. but, the same implementation works fine with SOAP API. Can I get any guidance on this please @mhowlett ?

[smitty-codes]

I keep up with Confluent's blog posts - I know they are marketing in disguise but they do offer alot of good info as well. Yesterday's blog post titled Kafka Streams Interactive Queries Go Prime Time has the first sentence "What is stopping you from using Kafka Streams as your data layer for building applications?"

Answer: C# support :-)

We're using the Microsoft stack and we're likely going to use Azure EventHub, despite its limitations, instead of Confluent Cloud solely because Kafka Streams is not supported in C#. It doesn't make sense for me to retrain and retool my team to use NetBeans/IntelliJ/Eclipse and learning Java when our team is already skilled in C# and Visual Studio and all of our code and processes are built around that.

I'm really hoping you consider C# support.

As some kind of middle ground, I raised on one of Confluent's webinars invoking KSQL like we would using a regular SqlConnection and SqlCommand classes/objects to query Kafka but that was not supported. Maybe this is the path Confluent is pursuing rather than porting Kafka Streams for every language?

Thanks

[mhowlett]

we have a big focus on making ksqldb fully featured and powerful and i think use cases for streams itself will become increasingly niche. on the client side, we're committed to feature parity of the basic clients with java, including adding a lot of things that make streams powerful, like exactly once semantics (including kip-447) and incremental rebalancing. in summary, it's very feasible to write a lot of applications with just the basic clients, and a lot of the applications you'd use the higher level streams api for are the types of things you can easily express in ksql. we'd like to support kafka streams in different languages, but it's a relatively large amount of work, and my perception is the payoff probably isn't there - we're better off putting the focus into ksqldb (disclaimer, i'm not on the product team!)

[smitty-codes]

@mhowlett Thanks Matt. I'll revisit KSQL with my team and see if we can blend it with the regular .NET Kafka client.

I think I'll keep an eye on confluentinc/ksql#734 which appears to be closer to that middle ground I was describing.

[ahernandez-allegion]

@smitty-codes We're also largely an Azure/C# shop. We're currently using Event Hubs with Connect as a stop gap measure and we're doing our stream processing strictly through KSQL. So far the basic C# client support as of 1.4+ (with the updated Schema Registry serdes support) has been pretty good against Event Hubs.

[devopg]

vote C# kafka streams

[amitla]

we are also looking forward for KStreams in .net

[dariengit]

Vote Kafka Streaming for .Net Core

[tolgabalci]

In the absence of Kafka Streaming API support in .Net It would be helpful if a guide and examples could be provided to how best use KSQL in .Net.

[jasoncouture]

+1 here too.

[suadev]

Knock knock for the .Net core 🤓

[kvansaders]

Can the subscriber listen to a ksql stream by name (as it does a topic)?

[almazik]

Can the subscriber listen to a ksql stream by name (as it does a topic)?

KSQL stream IS a topic. Each KSQL stream and table has a topic behind it.

[efeozyer]

Happy new year -.-

[Asif-Mahmood]

any update on stream for .NET core?

[LGouellec]

Hi @Asif-Mahmood, community package exist to support Kafka Streams in .NET Standard 2.1. It's begin in written. All features are not available yet (but soon I hope).
You can check here for more details.

[Asif-Mahmood]

that's great and thanks for quick reply.. i hope that it works on .NET core as well because its mentioned on the repo

[LGouellec]

Yeah of course. This library is .net standard 2.1 compliant. So .net core >= 3 works !

[zlarson]

Hi @Asif-Mahmood, community package exist to support Kafka Streams in .NET Standard 2.1. It's begin in written. All features are not available yet (but soon I hope).
You can check here for more details.

is there a timeframe for the rocksdb support?

[LGouellec]

Hi @Asif-Mahmood, community package exist to support Kafka Streams in .NET Standard 2.1. It's begin in written. All features are not available yet (but soon I hope).

You can check here for more details.

is there a timeframe for the rocksdb support?

Hi, I am small late about this feature.
Rocksdb support, changelog topic & répartition topic will be available in next release (current April if all Ok)

[undecided2013]

I am bumping this up. As a .net dev for many years and a kafka user for the past 4 years, I believe making kafka streams available to the .net is important. I see this issue has been around for 2 years now. Are there any roadblocks to starting this work? Is it anywhere on the roadmap? Thx

[tomasfabian]

Hi, I created an alternative approach to Kafka Streams via ksqldb for .NET. Please check out my repo ksqlDB.RestApi.Client-DotNet.

ksqldb LINQ provider is an officially unofficial .NET client for ksqldb.

[DatStorm]

It is now 2023 and this issue is still open and upvotes has reached 279! How high does it need to go before work will start?

[bewarusman]

We still need this.

[In-Wake]

6 years?

[jangarach]

Why not negotiate with @LGouellec about adding his works to the official support?