WIP: KafkaProducer #333
Closed
WIP: KafkaProducer #333
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 6 commits
March 7, 2015 21:46
…: Noop, String, Json, Msgpack
The interface of this producer class is modeled after the new upstream java client in version 0.8.2: http://kafka.apache.org/documentation.html#producerapi KafkaProducer is configured via keyword arguments to the __init__() constructor. The configuration keywords generally follow the upstream java client settings. Messages are produced with KafkaProducer.send(), which places the messages on an internal FIFO queue. Messages are consumed from the queue asynchronously by a background worker thread.
added 5 commits
March 8, 2015 17:40
…ining each exception we handle
|
@dpkp is there anything I can do to get help with this PR? |
|
I put it aside for a while in favor of improving the existing producer implementations. If you think it is useful in current form, I will take a look and maybe merge after 0.9.4 release. Have you tried it out? |
|
No :-) Just trying to get the current state of async-producers (or batching message sending) and stumbled above this PR. |
dpkp
force-pushed
the
master
branch
2 times, most recently
from
b5e3a14
to
4955582
Compare
|
closing in favor of #515 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR implements a new asynchronous producer with an interface similar to the new java upstream client.
It currently uses a single background thread and does not support batching.
TODO: support batching; use event loop or multiple threads; use futures instead of callbacks
(this PR is based on #332, will rebase to master if/when that lands separately)