proposal: increase default value for `max_samples_per_send`

[valyala]

Proposal

Default value for max_samples_per_send - 100 - is too low for any non-idle Prometheus setup with remote_write enabled. It results in too frequent requests to remote storage if Prometheus scrapes more than a few hundred metrics per second. High requests' frequency wastes resources on both Prometheus and remote storage sides, so users have to increase max_samples_per_send after the first attempt to write metrics to remote storage.

It would be great if default value for max_samples_per_send is increased from 100 to 1000 or even 10K. This would simplify remote_write configuration for the majority of users.

[cstyan]

Default value for max_samples_per_send - 100 - is too low for any non-idle Prometheus setup with remote_write enabled. It results in too frequent requests to remote storage if Prometheus scrapes more than a few hundred metrics per second.

Can you elaborate? I don't think we're seeing any issues like this with any of our Prometheus instances.

[valyala]

• Github users usually use high values for max_samples_per_send. This suggests that the default value is too low.
• See this and this issue. They contain suggestions for increasing max_samples_per_send to 1000 in order to fix performance issues.

[juliusv]

Yeah, 100 seems a bit low.

[bboreham]

Another datapoint: Weaveworks customer config is set to 1000.

@valyala a lot of those configs on GitHub also have max_shards: 10000 which suggests they haven't thought this through...

[valyala]

Yeah, default max_shards should be lowered to appropriate value when increasing default max_samples_per_send value

[beorn-]

The system CPU usage was quite high after adding a significant amount of metrics.

We were about to scale the platform and we noticed the CPU system usage was high. After profiling the system we've noticed that the kernel was spending a very significant amount of time handling lookups in established TCP connections kernel hashtable.

After checking things i've seen that in our setup we were pushing 150k datapoints/s which meant 1500 tcp connections/s : Hence a serious amount of TIME_WAIT connections (perfectly normal).

Load was through the roof (more than 30 on a 12 cores server) and rules evaluation time exploded( 30s instead of the usual milliseconds)

We have ended up with

    queue_config:
      capacity: 300000
      max_shards: 100
      max_samples_per_send: 10000

It now works fine with 2-3 load and very good rules evaluations time

One problem still remains. it eats up more memory, if i'm not mistaken and the prometheus code is not really meant for big max_samples_per_send values. Maybe fixing keep-alive through http 1.1 might be a quick win too ?

[beorn-]

About keep-alived connections @elwinar and I came up with elwinar@a153ee9

It seems to fix the issue.

[bboreham]

FYI “http pipelining” means something different to keep-alive, and is not relevant here. See https://en.m.wikipedia.org/wiki/HTTP_pipelining

This doesn’t impact your suggestion; I just like to keep the terminology clear.

[beorn-]

I stand corrected. To avoid any unneeded confusion i have edited my past comments. Thanks @bboreham .