CounterMap

We herewith begin a saga about monitoring with this series entitled "Timestamped: a trilogy in a few parts," this being the first part, where we introduce a map to count key things, and ensure we can order it by its integer values.

We will be analysing logs in this unwinding series. Ultimately we gonna hook up a remote Log4j appender to digest our logs in order to gather stats, and make some judgement calls as to the rapidly changing status of our app.

As you can imagine we will need to count so many key things like the number of successes and failures of one thing or another. For this purpose, we introduce the following so-called "counter map."

public class IntegerCounterMap<K> extends TreeMap<K, Integer> {
    private int totalCount = 0;

    public int getInt(K key, int defaultValue) {
        if (!containsKey(key)) {
            return defaultValue;
        } else {
            return get(key);
        }
    }

    public int getInt(K key) {
        return getInt(key, 0);
    }
    
    public void add(K key, int augend) {
        totalCount += augend;
        put(key, new Integer(getInt(key) + augend));
    }
    
    public void increment(K key) {
        add(key, 1);
    }

    public int getTotalCount() {
        return totalCount;
    }

    public int calculateTotalCount() {
        int total = 0;
        for (K key : keySet()) {
            total += getInt(key);
        }
        return total;
    }
    
    public LinkedList<K> descendingValueKeys() {
        return Maps.descendingValueKeys(this);
    }
}

where since we are often counting things one by one e.g. as we digest log messages, we provide that increment() convenience method.

Our typical key type is String e.g. the name of something we are counting e.g. "ERROR".

    @Test
    public void testCounterMap() {
        IntegerCounterMap<String> counterMap = new IntegerCounterMap();
        counterMap.increment("INFO");
        counterMap.increment("ERROR");
        counterMap.increment("ERROR");
        Assert.assertEquals(100*counterMap.getInt("ERROR")/counterMap.getTotalCount(), 66);
        Assert.assertEquals(counterMap.getInt("WARN"), 0);
        Assert.assertEquals(counterMap.getInt("INFO"), 1);
        Assert.assertEquals(counterMap.getInt("ERROR"), 2);
        Assert.assertEquals(counterMap.size(), 2);
        Assert.assertEquals(counterMap.getTotalCount(), 3);
        Assert.assertEquals(counterMap.getTotalCount(), counterMap.calculateTotalCount());
   }

where getTotalCount() is used to get a percentage of the total e.g. a 66% error rate, D'oh!

Note that since WARN is not incremented like the others, it's not put into the map, and so the size of map is only the two keys, namely INFO and ERROR.

We have a descendingValueKeys() method for when we wanna display counters in descending order, to see the biggest numbers and/or worst culprits. This delegates to the following util class.

public class Maps {  

    public static <K, V extends Comparable> LinkedList<K> descendingValueKeys(Map<K, V> map) {
        return keyLinkedList(descendingValueEntrySet(map));
    }
    
    public static <K, V extends Comparable> NavigableSet<Entry<K, V>> 
            descendingValueEntrySet(Map<K, V> map) {
        TreeSet set = new TreeSet(new Comparator<Entry<K, V>>() {

            @Override
            public int compare(Entry<K, V> o1, Entry<K, V> o2) {
                return o2.getValue().compareTo(o1.getValue());
            }
        });
        set.addAll(map.entrySet());
        return set;
    }

    public static <K, V> LinkedList<K> keyLinkedList(NavigableSet<Entry<K, V>> entrySet) {
        LinkedList<K> keyList = new LinkedList();
        for (Map.Entry<K, V> entry : entrySet) {
            keyList.add(entry.getKey());
        }
        return keyList;
    }

    public static <K, V extends Comparable> V getMinimumValue(Map<K, V> map) {
        return getMinimumValueEntry(map).getValue();
    }
    ...
}

where courtesy of a TreeMap, we sort the map's entries by value, and put the thus ordered keys into a LinkedList to iterate over e.g. in a for each loop.

See also stackoverflow.com which helped me with that.

Let's test this ordering.

    @Test
    public void testDescendingMap() {
        IntegerCounterMap<String> counterMap = new IntegerCounterMap();
        counterMap.add("BWARN", 0);
        counterMap.add("DEBUG", 1000000);
        counterMap.add("ERROR", 5000);
        counterMap.add("INFO", 1);
        Assert.assertEquals(counterMap.size(), 4);
        Assert.assertEquals(counterMap.descendingValueKeys().getFirst(), "DEBUG");
        Assert.assertEquals(counterMap.descendingValueKeys().getLast(), "BWARN");
        Assert.assertEquals(Maps.getMinimumValue(counterMap).intValue(), 0);
        Assert.assertEquals(Maps.getMaximumValue(counterMap).intValue(), 1000000);
        for (String key : Maps.descendingValueKeys(counterMap)) {
            System.out.printf("%d %s\n", counterMap.get(key), key);
        }
     }

where we use BWARN instead of WARN to be sure we aren't picking up the natural ordering. (Yes, that was hiding a bug that bit me in the bum!)

After some DEBUG'ing (with the help of a lot logging), we eventually get what we would have expected...

1000000 DEBUG
1001 ERROR
1 INFO
0 BWARN

As so often seems to be the case, we have a million DEBUG messages, a thousand and one ERRORs, with very little INFO to go on, and no bloody warning! "Put that in your internet, put that in your twitter right there."

Resources

https://github.com/evanx/vellum/wiki