StatsD using NodeJS Cluster Module #250

Closed
idanmashaal opened this Issue Feb 19, 2013 · 2 comments

Comments

Projects
None yet
2 participants

Hi,

Currently StatsD isn't using the Cluster module limiting it to a single core on multi-core servers. In order to spread load one can do persistent hash based load-balancing from the client side to multiple StatsD instances running on the same server, from my testing a good value is (NUM_OF_CORE-1) instances of StatsD per server in case StatsD goes crazy due to a surge in incoming packets.

Load-Balanceing via a UDP load-balancer isn't a viable option since there aren't many load-balancers that can handle UDP traffic.

I think that if StatsD used the Cluster module via dynamic forking worker processes until NUM_OF_CORE-1 would make StatsD handle the computation power of powerful servers.

What is the limitation in using NodeJS Cluster with StatsD - is it something in the design of StatsD ?

Thanks,
Idan.

Contributor

draco2003 commented Mar 5, 2013

Hey Idan,

This is definitely on our radar, and something we are investigating. We are playing with both setups currently, using node's cluster module for local-only loadbalancing as well as UDP loadbalancing with a statsd-proxy of sorts to be able to handle local and remote load balancing.

Though before we put to much time into that we are currently focusing on optimizing and cleaning up the current code, as well as profiling to see where exactly the bottlenecks are.

Contributor

draco2003 commented Apr 30, 2013

@idanmashaal we are going to go the route of the StatsD Cluster Proxy as a way to scale up StatsD for now while keeping the core as simple as possible for most use cases. More details for the Statsd Cluster Proxy can be found in this pr #284

draco2003 closed this Apr 30, 2013

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment