Determine why metric reported 4 quadrillion ms

[keith-turner]

Saw the following while testing 2.0.0-SNAPSHOT. Need to determine why the commit prep time metric was so high.

[Manno15]

@keith-turner What tests were you running when getting these results? I want to see if I can reproduce these so I can get to the bottom of it.

[keith-turner]

I am not sure, but probably continuous ingest. Possibly with tablet server agitation.

[ctubbsii]

The TServer adds metrics by taking the size of an .entrySet() (which is an int) to a double, so it can be the divisor for the total time taken in millis (which is a long), presumably so it can keep some precision in the division, before it casts to a long to return the result.

I see two possibilities:

1. The entry set was empty, and casting it to a 0 to a double resulted in something very large when placing this "very close to, but not quite zero" value in the denominator, or
2. The system clock was updated and caused a bigger than expected time difference in millis when using System.currentTimeMillis().

I think option 1 is more likely. We should probably guard against the set being empty before adding a metrics.

There are two methods that do this:

accumulo/server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java

Lines 1156 to 1162 in ebca8e1

	private void updateAvgCommitTime(long time, int size) {
	updateMetrics.addCommitTime((long) (time / (double) size));
	}
	private void updateAvgPrepTime(long time, int size) {
	updateMetrics.addCommitPrep((long) (time / (double) size));
	}

This is all just rounding issues, but if I'm right, and the size was zero, what probably happened is that no mutations were queued on line:

accumulo/server/tserver/src/main/java/org/apache/accumulo/tserver/TabletServer.java

Line 1091 in ebca8e1

updateAvgPrepTime(pt2 - pt1, us.queuedMutations.size());

[ctubbsii]

This should be a simple fix for 2.0. We just return in the above methods if (size < 1).

[Manno15]

I'll add those changes and do some quick testing, doubt I can reproduce this bug consistently enough to guarantee it being fixed