metrics reporting is dominating our CPU usage.

[stevej]

While profiling linkerd-tcp using linux perf, I uncovered that a large chunk of our CPU usage is stats reporting.

CPU usage without stats reporting is 36% at 10k qps.
CPU usage with stats reporting is 150% at 10k qps.

Attached is a zipfile with the SVG flame graph. Use a Mac App like Gapplin to view it.
flame.zip

When we turned off stats, the exact patch we wrote against tacho master was this

diff --git a/src/recorder.rs b/src/recorder.rs
index 123850f..7d361a6 100644
--- a/src/recorder.rs
+++ b/src/recorder.rs
@@ -36,7 +36,7 @@ impl Recorder {
             *curr += n;
             return;
         }
-        self.sample.counters.insert(k.clone(), n);
+        //self.sample.counters.insert(k.clone(), n);
     }

     pub fn set(&mut self, k: &GaugeKey, n: u64) {
@@ -44,7 +44,7 @@ impl Recorder {
             *curr = n;
             return;
         }
-        self.sample.gauges.insert(k.clone(), n);
+        //self.sample.gauges.insert(k.clone(), n);
     }

     pub fn add(&mut self, k: &StatKey, n: u64) {
@@ -55,17 +55,17 @@ impl Recorder {

         let mut vals = VecDeque::new();
         vals.push_back(n);
-        self.sample.stats.insert(k.clone(), vals);
+        //self.sample.stats.insert(k.clone(), vals);
     }
 }
 impl Drop for Recorder {
     fn drop(&mut self) {
         // Steal the sample from the recorder so we can give it to the channel without
         // copying.
-        let sample = mem::replace(&mut self.sample, Sample::default());
+        /*let sample = mem::replace(&mut self.sample, Sample::default());
         if mpsc::UnboundedSender::send(&self.tx, sample).is_err() {
             info!("dropping metrics");
-        }
+        }*/
     }
 }

[stevej]

Closing this due to the tacho fixes we made.