[20170116 nightly] Throttling errors

[kostasb]

Version 1.3.0~n201701160800, branch master, commit bb029b54447fc2806af00fe1cddd44146476d3f1

This is a deployment where InfluxDB v1.1 sustains uptime in the same environment (but slowly allocates all ram until oom in 2 hours due to #7832) whereas this nightly build returns throttling errors just a few minutes in to the test.

The query endpoint also becomes unresponsive.

In this excerpt errors start after a compaction kicks in (might be unrelated).

[I] 2017-01-16T16:41:34Z compacted level 1 group (0) into /data/datastore/monitoring/influxdb/data/historical/events.10d/232/000000344-000000002.tsm.tmp (#0) engine=tsm1
[I] 2017-01-16T16:41:34Z compacted level 1 2 files into 1 files in 7.087589285s engine=tsm1
[I] 2017-01-16T16:41:34Z beginning level 2 compaction of group 0, 2 TSM files engine=tsm1
[I] 2017-01-16T16:41:34Z compacting level 2 group (0) /data/datastore/monitoring/influxdb/data/historical/events.10d/232/000000342-000000002.tsm (#0) engine=tsm1
[I] 2017-01-16T16:41:34Z compacting level 2 group (0) /data/datastore/monitoring/influxdb/data/historical/events.10d/232/000000344-000000002.tsm (#1) engine=tsm1
...
[httpd] 10.243.64.125 - os [16/Jan/2017:16:41:43 +0000] "POST /write?consistency=any&db=historical&precision=ns&rp=events.10d HTTP/1.1" 500 44 "-" "InfluxDBClient" a983ebfe-dc0a-11e6-9eef-000000000000 10025189

Block, routine, heap profiles when 500's start:
influx_output.tar.gz