Different values receiver by graphite when activating repeater

[sveronese]

I have a setup where applications are sending metrics to 2 different statsd. Both Statsd are sending to a graphite backend.
I decide to test repeater backend on one of 2 statsd in order to copy all metrics to statsd-exporter and expose some metrics in prometheus format. The scenario is the following using statsd version v0.8.5, STATSD1 and STATSD2 are running on different servers. Everything is running in a docker.

                 |--UDP--> STATSD1 -UDP-> GRAPHITE-BACKEND-1
                 |
APPLICATION -----|
                 |					 
                 |                  |-TCP--> REPEATER-BACKEND --> STATSD-EXPORTER
                 |--UDP---> STATSD2-|
                                    |-UDP--> GRAPHITE-BACKEND-2

What is happening here is that, when I activate "REPEATER-BACKEND" on STATSD2, the data sent to GRAPHITE-BACKEND-2 are different from the data sent to GRAPHITE-BACKEND-1 and also seems some data are lost. If I disable REPEATER-BACKEND, GRAPHITE-BACKEND-1 and GRAPHITE-BACKEND-2 receive same data and everything is working as expected.
Here the configuration that I used on STATSD2:

STATSD2

{
  port: 8125,
  mgmt_port: 8126,

  percentThreshold: [ 50, 75, 90, 95, 98, 99, 99.9, 99.99, 99.999],

  graphitePort: 2003,
  graphiteHost: "relay",
  flushInterval: 10000,
  deleteIdleStats: true,
  debug: false,

  backends: [ "./backends/repeater", "./backends/graphite" ],
  repeater: [ { host: 'statsd-exporter', port: 9125 } ],
  repeaterProtocol: "tcp",

  graphite: {
    legacyNamespace: false
  }
}

I tried also with another scenario as following:

                |--UDP--> STATSD1 -UDP-> GRAPHITE-BACKEND-1
                |
APPLICATION ----|
                |
                | 
                |                  |-TCP--> REPEATER-BACKEND-1 --> STATSD-EXPORTER  
                |--UDP--> STATSD2 -| 
                                   |-TCP--> REPEATER-BACKEND-2 --> STATSD-GRAPHITE-1 -UDP-> GRAPHITE-BACKEND-2

With following configurations:

STATSD2

{
  port: 8125,
  mgmt_port: 8126,

  percentThreshold: [ 50, 75, 90, 95, 98, 99, 99.9, 99.99, 99.999],

  flushInterval: 10000,
  deleteIdleStats: true,
  debug: false,

  backends: [ "./backends/repeater" ],
  repeater: [ { host: 'statsd-exporter', port: 9125 }, { host: 'statsd-graphite', port: 10125 } ],
  repeaterProtocol: "tcp",

  graphite: {
    legacyNamespace: false
  }
}

STATSD-GRAPHITE-1

{
  port: "10125",
  mgmt_port: 10126,

  percentThreshold: [ 50, 75, 90, 95, 98, 99, 99.9, 99.99, 99.999],

  graphitePort: 2003,
  graphiteHost: "relay",
  flushInterval: 10000,
  deleteIdleStats: true,
  debug: false,
  
  server: "./servers/tcp",

  backends: [ "./backends/graphite" ],

  graphite: {
    legacyNamespace: false
  }
}

But the values sent are different. Here you can find screenshot of data read from

GRAPHITE-BACKEND-1

GRAPHITE-BACKEND-2

Do you have any idea why this is happening or have any suggestions on ways to troubleshoot that ?

Thank you.

[BlueHatbRit]

Hm, I've not used the repeater backend much so I'm not sure how much I can help. Maybe @mrtazz might have time to take a look at this and provide his expertise?

[mrtazz]

There are a couple of things to note here. First, in this scenario

                |--UDP--> STATSD1 -UDP-> GRAPHITE-BACKEND-1
                |
APPLICATION ----|
                |
                | 
                |                  |-TCP--> REPEATER-BACKEND-1 --> STATSD-EXPORTER  
                |--UDP--> STATSD2 -| 
                                   |-TCP--> REPEATER-BACKEND-2 --> STATSD-GRAPHITE-1 -UDP-> GRAPHITE-BACKEND-2

You effectively aggregate data twice (in STATSD2 and STATSD-GRAPHITE-1) if I understand your setup correctly. So I would expect the data to be very different here. In the first scenario, given that you mention some data is also lost, are you running anywhere close to max CPU usage on your setup? Given the nature of how nodejs works, once you're close to max CPU packet (and thus data) drops are fairly common.

The best way to debug scenarios like that is in my experience to tcpdump/wireshark on the different connection points and make sure the data you see is what you'd expect to pin down where the deviation is happening.

[BlueHatbRit]

@gagado please don't comment on mega old commits or PR's unless it's relevant to them. If your question can be answered here in an active thread it'd be best to do it here. mrtazz has provided some information above which is worth looking at and trying.