Prometheus cannot send > ~25k/s to Influxdb

[dan-cleinmark]

Following up from IRC -

I have a 4 node influx cluster behind an ELB. When I configure a single instance of Prometheus to send to this cluster, I'm able to send a max of around 25k events per-second to the cluster before the remote storage queue blocks and events start being dropped. Increasing 'maxSamplesPerSend' (10 -> 50) and 'maxConcurrentSends' (100 -> 1000) in storage/remote/queue_manager.go didn't seem to help (master...dan-cleinmark:increase_remote_maxSamplesPerSend)

The bottleneck does not seem to be on the Influx side as I'm able to send 75k messages per-second to the same Influx cluster (15k/s from 5x prometheus instances). Looking at the ELB stats while running 0086d48, I'm seeing ~2300 HTTP requests/minute through the ELB with an average latency of ~75ms.

Are there other parameters that could be tweaked to improve performance when sending to Influx from a single prometheus instance?

[dan-cleinmark]

Running without the changes to storage/remote/queue_manager.go (revision: e91d85b) has the same result (a single prometheus instance is bottlenecked at ~25k/s but 5x prometheus instances can send to the same influx cluster at ~75k/s aggregate). The ELB in front of the influx cluster reports ~40k HTTP requests/minute with latency of ~70ms.

[fabxc]

I think it's fair to say that we won't pursue improving the existing InfluxDB integration any further – especially in light of clustering being closed source now.
The future focus will be on a generic write path. Likely someone will provide an integration via that one then that will hopefully be performant.

[brian-brazil]

I note that I managed to get 100k/s out of the generic write path, so it's unclear that there's a problem here on the Prometheus end.

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.