Revert "Improving Performance on the API Gzip Handler (#12363)"

[roidelapluie]

This reverts commit dfae954.

[roidelapluie]

/prombench v2.45.0-rc.0

[prombot]

⏱️ Welcome to Prometheus Benchmarking Tool. ⏱️

Compared versions: PR-12476 and v2.45.0-rc.0

After successful deployment, the benchmarking metrics can be viewed at:

• Prometheus Meta
• Prombench Dashboard
• Grafana Explorer, Loki logs

Other Commands:
To stop benchmark: /prombench cancel
To restart benchmark: /prombench restart v2.45.0-rc.0

[jesusvazquez]

This took a while! But we finally found it!!

http://prombench.prometheus.io/grafana/d/7gmLoNDmz/prombench?orgId=1&var-pr-number=12476&fullscreen&panelId=9

[bboreham]

I don't understand the mechanism (and I spent several hours trying to), but I agree with the results.

[roidelapluie]

/prombench restart v2.44.0

[prombot]

⏱️ Welcome to Prometheus Benchmarking Tool. ⏱️

Compared versions: PR-12476 and v2.44.0

After successful deployment, the benchmarking metrics can be viewed at:

• Prometheus Meta
• Prombench Dashboard
• Grafana Exlorer, Loki logs

Other Commands:
To stop benchmark: /prombench cancel
To restart benchmark: /prombench restart v2.44.0

[jesusvazquez]

Latest benchmark shows that now the branch has better performance than 2.44

So we're going to cut 2.45.0-rc.1 out of the current state of the release branch

cc @alanprot We remove this change from the release due to the performance implications. Letting you know in case you want to invest more time finding out if we can use that new lib without the performance impact. 🙏

[jesusvazquez]

/prombench cancel

[prombot]

Benchmark cancel is in progress.

[alanprot]

Yeah... indeed seems to be using more CPU. But (if im reading this correctly) also seems also that at some point we have a better latency and more req/second using the new handler? Do we know what happens around 6 that the req/sec start to decrease and de latency to increase (and is where we see a better latency with the new handler)? Maybe at this time the response become bigger and is where the new gzip is better?

[bboreham]

Some of the queries are from -2h to -1h, so they will start to see data at that point.

I think you have pointed out one part I didn't get - the requests/sec drops because queries are taking longer than the load-generator interval, and so the faster one (rc-0) is doing more work.
However rc-0 is slower before that point and using more CPU so I think it is worth looking into how we can improve that.

[alanprot]

However rc-0 is slower before that point and using more CPU so I think it is worth looking into how we can improve that.

Yeah, I agree... rc-0 is always using more CPU and we need to understand why. I was just pointing out that seems the rc-0 is faster after some point. (I wonder if the new lib is less efficient with smaller payload but more efficient with big ones or something like that - just speculation for now). I will see if i have some time this week to look into it!

[roidelapluie]

Thanks @alanprot :-)