[POC] Client side aggregation.

[hush-hush]

First, early try of client side aggregation.

For now this looks promising. We're testing both best case scenarion (1 metric sampled a lot) and worst case scenario (a lot of metrics sampled only once).

goos: linux
goarch: amd64
pkg: github.com/DataDog/datadog-go/statsd
BenchmarkStatsdUDPSameMetricBLocking-8                   	 5984832	       218 ns/op	         0 %_dropRate
BenchmarkStatsdUDPSameMetricDropping-8                   	13658887	        85.3 ns/op	        93.3 %_dropRate
BenchmarkStatsdUDPSameMetricBLockingAggregation-8        	13163277	        81.6 ns/op	         0 %_dropRate
BenchmarkStatsdUDPSameMetricDroppingAggregation-8        	15646678	        74.6 ns/op	         0 %_dropRate
BenchmarkStatsdUDPDifferentMetricBlocking-8              	17076426	        64.3 ns/op	         0 %_dropRate
BenchmarkStatsdUDPDifferentMetricDropping-8              	17353012	        66.9 ns/op	        83.4 %_dropRate
BenchmarkStatsdUDPDifferentMetricBlockingAggregation-8   	14357695	        81.4 ns/op	         0 %_dropRate
BenchmarkStatsdUDPDifferentMetricDroppingAggregation-8   	17213880	        68.9 ns/op	         0 %_dropRate
BenchmarkStatsdUDSSameMetricBLocking-8                   	 6019815	       214 ns/op	         0 %_dropRate
BenchmarkStatsdUDSSameMetricDropping-8                   	16993280	        81.6 ns/op	        93.4 %_dropRate
BenchmarkStatsdUDSSameMetricBLockingAggregation-8        	16397464	        85.0 ns/op	         0 %_dropRate
BenchmarkStatsdUDSSameMetricDroppingAggregation-8        	15898207	        68.9 ns/op	         0 %_dropRate
BenchmarkStatsdUDPSifferentMetricBlocking-8              	15929932	        64.1 ns/op	         0 %_dropRate
BenchmarkStatsdUDSDifferentMetricDropping-8              	15504646	        64.6 ns/op	        82.0 %_dropRate
BenchmarkStatsdUDPSifferentMetricBlockingAggregation-8   	14955110	        76.3 ns/op	         0 %_dropRate
BenchmarkStatsdUDSDifferentMetricDroppingAggregation-8   	16314648	        69.6 ns/op	         0 %_dropRate

[ogaca-dd]

Great POC! A few nits.

[ogaca-dd]

Consider thread 1 after executing the instruction at line 102 and thread 2 at line 79

Thread 1 takes the lock and run until line 105, execute a.countsM.RUnlock() and stop (before executing count.sample(value))

As lock was released, thread 2 can run until line 97.

Thread 1 execute count.sample(value). It is increment the value in the map counts (counts := a.counts) but as values were already flushed (metrics = append(metrics, c.flushUnsafe())) the increment count.sample(value) is not take into account.

Maybe this is the reason of the name flushUnsafe

[hush-hush]

Indeed, we should not release the read lock until we sampled the metric. This should be fine as we won't slow down the sampling.

[ogaca-dd]

LGTM

[ogaca-dd]

As sample is used inside a block a.countsM.RLock() / a.countsM.RUnlock(), s.Lock() is not required anymore.

[hush-hush]

I think it is to avoid concurrent write to the map, no ?

[remeh]

Well-written PR!