-
Notifications
You must be signed in to change notification settings - Fork 10
Sample successful delivery and connections stats at 10% #217
base: dev
Are you sure you want to change the base?
Conversation
93ae105
to
04ea63a
Compare
04ea63a
to
a3cf377
Compare
8c85f6a
to
603c84d
Compare
@@ -426,7 +426,7 @@ func (b *EtcdBalancer) Fetch() (peers *EtcdPeers, err error) { | |||
b.metrics.Increment("balancer.fetch.error") | |||
return nil, err | |||
} | |||
b.metrics.Increment("balancer.fetch.success") | |||
b.metrics.IncrementByRate("balancer.fetch.success", 1, 0.1) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure if this needs a low sample rate...we're only polling etcd every 10 seconds by default, and updating every minute. Even with both rates reduced to a second, that seems like a trickle compared to the storm of delivery metrics.
* Ensure `IncrementByRate()` passes the absolute value of the delta to `StatsClient.Dec()`. * Replace `TestMetrics` with a `StatsClient` interface for `Metrics`. * Lazily initialize `Metrics` snapshots. * Remove remaining logged metrics. * Add tests. Closes #177.
`{IncrementBy, Timer}Rate()` and `GaugeDelta()` no longer call the underlying `StatsClient` methods for zero values.
603c84d
to
74fdb69
Compare
Looks fine, except the sampling rate shouldn't be hard-coded, it should be a config value so we can determine it when we deploy. |
@bbangert Sounds good. Do you think a single config value for all metrics should be sufficient? Does the sampling logic look sound? |
@kitcambridge I think a single value for the heaviest usage ones is fine. Sampling logic looks sound. I am curious about how much faster the stats might be if they stopped doing all that inefficient string concat. |
Metrics.IncrementByRate()
andTimerByRate()
for specifying the sampling rate.IncrementByRate()
would callStatsClient.Dec()
with a negative value.TestMetrics
with aStatsClient
interface.Closes #177.