[feature request] Make _internal's queryExecutor more reliable

[rossmcdonald]

Feature Request

Proposal:

Make the _internal databases queryExecutor more reliable for capturing the queriesActive metric.

Current behavior:

Currently, a query is not recorded as being "active" unless it is running when the stat snapshot occurs. This means that there can be multiple queries at any given time, and none of them will be recorded as "active". For example, the output of my local machine:

> select max(queriesActive) from queryExecutor where time > now() - 3m group by time(30s)
name,time,max
queryExecutor,1470434100000000000,0
queryExecutor,1470434130000000000,0
queryExecutor,1470434160000000000,0
queryExecutor,1470434190000000000,0
queryExecutor,1470434220000000000,0
queryExecutor,1470434250000000000,0
queryExecutor,1470434280000000000,0

Even though several queries have been active (one of them occurring when I captured this output).

Desired behavior:

I'd like to see the queriesActive field be exact. I think it would make sense to have it be a counter, instead of a snapshot. That way the counter is incremented for every query execution and the queriesActive field will show an exact (or close to exact) reflection of the number of queries run on the system.

Use case:

The current functionality is inaccurate, and will only display queries that happen to be running during the stats snapshot or long-lived.

[jsternberg]

Part of the issue is that SHOW STATS uses this exact same information so we can sometimes obtain statistics at a time when the monitor is not recording them.

There are two possible solutions. I think one is correct, but the other is much easier.

The first is I can add a new statistic for the number of queries executed. I can also add two different statistics: one for queries started and one for queries finished. That would get you roughly what you need and you could use difference() to figure some stuff out about them (although that currently has some limitations for the types of queries you can execute). This one is relatively straightforward and easy.

The second is making some way where SHOW STATS will look at the stats and the monitor services can retrieve the stats and reset them. This might be a little too complicated though and we would want to think about it more before doing it.

Thoughts?