Error sending a metric: channel full

[olksdr]

This happened few times, our statsd metrics could not be sent.

Sentry Issue: POP-RELAY-2NV

Error sending a metric: channel full
  caused by: channel full, maximum capacity: 100000

Error is coming from

relay/relay-statsd/src/lib.rs

Line 117 in 576466b

"Error sending a metric: {}, maximum capacity: {}",

[iker-barriocanal]

Investigation doc (internal).

[jan-auer]

We have concluded the only viable short-term option is to reduce the amount of metrics. We are going to analyze which metrics are creating too much traffic and will reduce them. Note that this is NOT a cardinality concern, it's that our individual increments of metrics are too much.

Other considered options:

• Increase channel size: This option doesn’t really address the problem, and it only postpones it. The cost is the increased CPU and memory usage, which currently may be around 20% of a relay instance’s cost.
• Increase number of channels or clients: Same as above.
• “Optimize” the channel: To “optimize” the channel for higher throughput, it is recommended to move from UDP to Unix Domain Socket (UDS). UDS is another, strem-oriented protocol (supports UDP). However, there aren’t any guarantees of increased performance (we need to run benchmarks for that), comes with its own set of challenges, and requires infrastructure changes. The main challenge is based on how UDS fundamentally works, based on a special file.