Reduce size of pods_selector by using replica prefixes

[lujiajing1126]

Closes #60, #62, #66

This PR intends to reduce the size of pod_selector, by picking the <deployment>-<pod-template-hash> as the prefix.

This could significantly reduce the size of the querystring in the case that there are hundreds of pods for a single deployment.

300 may be a magic number here: pod-template-hash normally contains 8 bytes, with 5 chars pod hash. The default max http header that nginx can accept is ~4K, so with >300 pods it must exceed the threshold.

[CLAassistant]

All committers have signed the CLA.

[LeaveMyYard]

Hello and sorry for long time waiting on response :)

We've been working a lot with performance issues and also trying your solution.
One of the solutions we tried is to make pod gathering and metric gathering in one query:
https://github.com/robusta-dev/krr/blob/rework-promql-queries/robusta_krr/core/integrations/prometheus/metrics/cpu_metric.py

It resulted in using even more memory load on prometheus to execute that query, as well as longer loading time.
Currently we thing that the correct solution will be to use prometheus with POST except of GET to avoid the limitation of request URI being too large (#66)
You have mentioned that in #60, but still we can use POST by just subclassing the lib we are using for prometheus

Additionally, I've been thinging on fully rewriting regex to fully utilize the format for pod names, but there is a concern that there might be special cases where it fails.

In any case thank you so much for the PR, I will keep it for reference and will close it as soon as we have a solution that satisfies everyone

[LeaveMyYard]

I am closing this PR as #97 should have fixed the issue

Still, reopen it if you feel like it improves anything else, but I will keep this possibility in mind in any case

Thank you very much for participating in a project ❤️

[lujiajing1126]

Hello and sorry for long time waiting on response :)

We've been working a lot with performance issues and also trying your solution. One of the solutions we tried is to make pod gathering and metric gathering in one query: https://github.com/robusta-dev/krr/blob/rework-promql-queries/robusta_krr/core/integrations/prometheus/metrics/cpu_metric.py

It resulted in using even more memory load on prometheus to execute that query, as well as longer loading time.

I've tested this solution on VictoriaMetrics (a cluster w/ 4 vmstorage), but it failed to return data and throws timeout. Probably it is too complicated.

I don't know if anyone else can make another test.

Currently we thing that the correct solution will be to use prometheus with POST except of GET to avoid the limitation of request URI being too large (#66) You have mentioned that in #60, but still we can use POST by just subclassing the lib we are using for prometheus

Sure. I forgot to mention this. We've modified site-package to make use of POST. But besides that, I would like to remind (for VM users) that vmselect has an argument,

  -search.maxQueryLen size
     The maximum search query length in bytes
     Supports the following optional suffixes for size values: KB, MB, GB, TB, KiB, MiB, GiB, TiB (default 16384)

The maximum query length is 16KiB. So we still have a limit even for POST.