Develop suite of standard Prometheus benchmarks

[juliusv]

We should develop and maintain a suite of easy-to-run Prometheus benchmarks that make it easy for us to compare the performance of different Prometheus versions and configurations.

Possible variables to vary in the benchmark:

• value of GOMAXPROCS
• general fulness of metrics database
• number of timeseries
• number of samples per timeseries
• ratio of samples in memory vs. disk
• query resolutions
• parallelism of queries
• Prometheus version

What to benchmark:

• point / range query times
• scraping / decoding / appending samples times

[brian-brazil]

http://prometheus.io/docs/introduction/faq/#why-does-prometheus-use-a-custom-storage-backend-rather-than-%5Bsome-other-storage-method%5D?-isn't-the-%22one-file-per-time-series%22-approach-killing-performance? answers this I'd say

[beorn7]

Not quite. Those benchmarks were executed "manually", by running a Prometheus server hotter and hotter until it broke. It hardly qualifies as "easy-to-run". There is https://github.com/prometheus/prometheus/blob/master/storage/local/storage_test.go#L711-L721 , which covers quite a lot in the storage layer, but has no knobs to turn and is not very specific.

[brian-brazil]

Can we close this now with prombench?

[fabxc]

Well, it could be better but all future issues should probably filed there.

…

On Fri, Jul 7, 2017 at 12:48 PM Brian Brazil ***@***.***> wrote: Can we close this now with prombench? — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#168 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AEuA8vn4cF2FWqNnfucG4ghquOPXOIX-ks5sLg0FgaJpZM4AmIKD> .

[gouthamve]

Hmm, I don't think so. We still don't have something to evaluate against. I mean, do we have a clear system to recognise regressions?

[fabxc]

Yes, there's stuff missing. But I think we should close here and file them in prombench.

[brian-brazil]

I agree with Fabian.

[juliusv]

+1

…

On Jul 7, 2017 14:06, "Brian Brazil" ***@***.***> wrote: I agree with Fabian. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#168 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAg1mPTy_x9PiWJxagI9AQxL06z7CQsQks5sLh9CgaJpZM4AmIKD> .

[lock]

This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.