Statsd - UDP - Data Loss -

[RajanBhatt]

Hello,

I would be inclined to know data loss observed in data center using Statsd which is a UDP based. What decision rationale was used leveraging UDP instead of TCP ?
Has anybody observed significant data loss in data center using Statsd ?
Any suggestion and thought on these issue is highly appreciated,

Thanks
Rajan

[mrtazz]

UDP was chosen because of its fire and forget characteristic. Chances are that you are sending enough data that losing a small amount of packets doesn't make a real difference. At least not enough to justify the overhead of connection init and closing for a small data payload. And packet loss hasn't really been a problem for us so far.

Do you see a lot of packet loss? Can you share any numbers about the system?

[RajanBhatt]

Hello Daniel,

Thanks for reply.
I concur with your argument. I have yet to quantify packet loss as our system is
not yet in full swing, What are your observations regarding packet loss or
estimate within data center ?

Regards
Rajan Bhatt

---

From: Daniel Schauenberg notifications@github.com
To: etsy/statsd statsd@noreply.github.com
Cc: Raj rajanbhatt@sbcglobal.net
Sent: Sat, February 9, 2013 12:41:19 PM
Subject: Re: [statsd] Statsd - UDP - Data Loss - (#245)

UDP was chosen because of its fire and forget characteristic. Chances are that
you are sending enough data that losing a small amount of packets doesn't make a
real difference. At least not enough to justify the overhead of connection init
and closing for a small data payload. And packet loss hasn't really been a
problem for us so far.

Do you see a lot of packet loss? Can you share any numbers about the system?
—
Reply to this email directly or view it on GitHub..

[draco2003]

Hey @RajanBhatt
Packet loss numbers will vary between data centers and even within different network segments within data centers. We haven't had any issues with packet loss affecting our metrics, but like @mrtazz stated above, even a small amount of udp packet loss is fine for a busy statsd machine. You may run into problems if you have low frequency metrics and high packet loss, in which case switching to sending those metrics via TCP straight to graphite might be a better idea.

Best of luck.

[jeff-minard-ck]

Because this comes up in searches, I thought I might chime in here -- we're definitely seeing packet loss happen. This isn't statsd's fault, but just the quirk of how UDP and a very busy network box operates, but it's something people do need to keep an eye out for:

The top graph has three reported stats: the green value from across our entire cluster, and the other two as part of a background job. When we check via database metric lookup, these numbers line up. In graphite, they do not. The reason is that the second two metrics are only generated from 2 background worker machines which get swamped with other network traffic during the blue line peaks (heavy background job time period). When those heavy background jobs are running, they are generating a ton of network requests and, I believe, drowning out the UDP for the two metrics from the top graph.

[hit9]

cc https://github.com/hit9/statsd-proxy

[RajanBhatt]

Thanks Chao. Appreciate it.Let me check it out Regards Rajan Bhatt

 On Thursday, September 10, 2015 7:19 AM, Chao Wang <notifications@github.com> wrote:

cc https://github.com/hit9/statsd-proxy—
Reply to this email directly or view it on GitHub.

[hit9]

@RajanBhatt 😄 We meet the same problem, and we are now using this statsd-proxy.

Maybe you also want to checkout the following projects:

• http://githubengineering.com/brubeck/
• https://github.com/uber/statsrelay

[RajanBhatt]

Your are kind...Let me check out and thanks again for brubeck link...Keep in touch Regards Rajan Bhatt

 On Thursday, September 10, 2015 7:58 AM, Chao Wang <notifications@github.com> wrote:

@RajanBhatt We meet the same problem, and we are now using this statsd-proxy.Maybe you also want to checkout the following projects:

• http://githubengineering.com/brubeck/
  —
  Reply to this email directly or view it on GitHub.