-
Notifications
You must be signed in to change notification settings - Fork 62
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Question : How can I consume my kafka topic as fast as possible ? #37
Comments
In other words, when producing a constant flux of message in my topic, my receiver is only receiving message every second. |
I am having a similar issue. Did you manage to find a solution to yours? Thanks :) |
When you commit every 10 milliseconds you are writing to zookeeper every 10 milliseconds. Your commit interval should be much, much larger. try 10 seconds? |
and @pocman sorry for the huge lag, gmail started putting my github notifications in spam. |
Thanks for the reply @sclasen. I seem to have about a 1 to 3 second delay before I receive a message to my receiver actor. val props = AkkaConsumerProps.forSystemWithFilter(
system = actorSystem,
zkConnect = config.zookeeperHosts,
topicFilter = filter,
group = subscriberSet.map(_.name) getOrElse new UUID().toString,
streams = config.numStreams,
keyDecoder = new StringDecoder(),
msgDecoder = new KafkaJsonDecoder[T](mapper),
receiver = actorSystem.actorOf(Props(classOf[ConsumerActor[T]], consumer))
)
val akkaConsumer = new AkkaConsumer(props)
akkaConsumer.start()
consumerRegistry.put(topicPattern.pattern, akkaConsumer) And then the ConsumerActor class ConsumerActor[T](consumer: Consumer[T]) extends Actor with Logging {
def receive = {
case event: PublishedEvent[T] =>
logger.trace(s"Received Event [$event]")
try {
consumer(event)
logger.trace(s"Event Processing complete for [$event]")
} catch {
case ex: Exception =>
logger.error(s"Error Processing Event(id=${event.eventId})", ex)
}
sender ! StreamFSM.Processed
}
} Any idea where that would be occurring? Thanks for your work :) |
@coreyauger can you set the log level to DEBUG and send me the output? |
Hi @sclasen .. thanks again for your help in this. The log explodes fast as akka is logging "heartbeats" for the cluster and other things that turn my local log file into a mess. I hope there is something in there that helps. There are a lot of msgs similar to this I suspect that this is more what you are looking for? Let me know if I can provide anything else :) |
This is helpful. Things appear to be working as expected to me. when you see things like this
it means that the underlying kafka consumer iterator What is your use case here? akka-kafka is designed more for high-throughput while still giving some processing/commit guarantees. Commit is a blocking operation in that when a commit happens we wait until all in-flight messages are processsed. Latencies for low-throughput topics are not expected to be very low. |
related #23 |
There are a few use cases.. but one of which is IM or (chat) based messaging.. so low latency is certainly required for this. In this case am I better to adapt a new kind of consumer or are you saying that I should not use kafka for this? I found this which lead me to believe that new version of kafka can achieve this low latency. Thanks again :) |
Well not sure about kafka being suitable for low latency, quote from the highest ranked answer on the link provided... I suppose it depends on how you design and tune things, but generally I dont think akka-kafka is a good fit for a low-latency, low-throughput chat application. How do the topics and partitions work? is there a consumer for each user? I think you will have much better chances of success for this particular use case if you just use the kafka consumer directly (this library uses it under the covers), with auto commit enabled, it will keep polling on the iterator during commits, and doesnt have any notion of an unused state. You may have the odd lost message during hard crashes but seems totally fine for a chat app imo. |
I will take a look at this. Would there be any interest if I submit a pr for another type of low latency consumer for this lib? Or is this use case best left for my own lib? Let me know Either way. Your help is much appreciate :) Thanks again ! |
Sure, happy to take a look at a PR, please do! On Wed, Feb 3, 2016 at 2:52 PM, Corey Auger notifications@github.com
|
Hi @sclasen , I have built a initial actor that is working well for my use case. It has been modelled from the Kafka HighLevel Consumer. Here is the code I have so far. Please take a look and if it is something that is you think we should have. I will clean it up and add it to the documentation. I will also branch the code and fold the commits into a single commit before I do a PR. Thanks :) |
Thanks! I think you can get rid of the hasNext stuff, since it is only used Otherwise I fear the code will spin on hasNext and burn cpu On Sunday, February 7, 2016, Corey Auger notifications@github.com wrote:
|
That was indeed my original implementation.. but I continue to get the time-out exception. There is most likely a setting that I can pass in for an infinite wait... I will have a look for that. :) |
Yeah, you will still have to handle timeout excetion in the next() call. On Mon, Feb 8, 2016 at 11:18 AM, Corey Auger notifications@github.com
|
adding this as a ref
Config
The blocking in my case will happen on |
Since timeout and auto commit are required for the HighLevel Consumer to work.. would it be best to append these arguments when the actor is creating the config? That way we can leave the settings intact for the other types of actors? I plan to use the system with services that support both actors with different goals.. |
Want to open a PR? easier to review/discuss. On Mon, Feb 8, 2016 at 1:16 PM, Corey Auger notifications@github.com
|
Sounds good :) |
Hi,
I'm using Kafka between ruby and scala and I noticed that my ruby HermannConsumer is consuming message faster than my AkkaConsumer.
I can set commitConfig = CommitConfig(commitInterval = Some(10 millis)) in kafkaProps but I'm not sure that it's the right thing to do.
How can I consume my kafka topic as fast as possible ?
The text was updated successfully, but these errors were encountered: