Change ISubscription and adding EOF messages into stream #42
Conversation
AndreSteenbergen
changed the title
|
From what I've seen, there's no notion of EOF in JVM - instead they start from getting end offset at the beginning and after reaching it emitting appropriate event type. See akka.streams.eventsourcing for more inspiration. |
…'s an EOF message. Splitted classes into separate files.
|
@Horusiath I couldn't find an endOffset in the metadata. I guess the java client is a little richer then the c# client? |
This way a user can await all messages being send.
original CompletionSource is well suited for this.
The code had a few async callbacks. These callbacks made sure the stage never completed. Changed calls to use callback
It isn't always important to get deliveryreports. The async retrieval can make the flow slow, it needs to wait for the ACKs for the messages sent based on number of threads.
|
Anything holding this PR back ? |
|
Actually... I have a local version using the beta 2 instead of the experimental Kafka connector. This PR would work as is, no problem. But I can update this PR with a newer kafka client and later Akka.Net version. I think I can have a fresh PR done by tomorrow 1300 CET. That is better I guess. |
|
An update would be great. 👍 Drop a comment here once you pushed your changes, and ill take a look. |
|
Done; I am using this code in production without issues so far. Only thing I have doubts about are the failstage parts. Kafka has a connection reaper, every 10 minutes the connection will be dropped. Raising an error, but the the producer is still able to push messages to kafka without restarting. Don't know what to do really. |
|
We fixed build issues. Needs rebase |
|
@AndreSteenbergen would you mind also updating the version prefix in the csproj for kafka ? |
|
will do. |
|
I am in the process of updating all nuget packages for kafka (again) kafka 1.0.0beta3 has quite a number of differences and breaking changes. wrt beta2, 1.0.0 is supposed to be out in march. If there is a huge demand for beta3 I can make a push, but I need to alter the tests as well. |
|
@Arkatufus is this PR still necessary? |
|
nope; there is a better nuget package already in Akka Streams |
|
Ah! I didn't realize this was for Kafka - good deal |
No description provided.