Metric correlations are often slow

[manos-saratsis]

Describe the bug
When running metric correlations sometimes it takes a very long time to get the results. Some other times requests time out.

To Reproduce
Steps to reproduce the behavior: Go to a single node view -> Click "Metric Correlations" -> Select an area of interest in a chart -> Click "Find Correlations". Not always reproducible

Expected behavior
Metric correlations should show results in less than 15 seconds.

Cc @dim08

[papazach]

Took some time to check out the logic and take some measurements of various parts of the code.
The tests took place in Staging environment and more specifically in composite-charts space and 100NotesTest room.

So in a nutshell at the moment three sequential operations take place as depicted in the gantt chart below (every "day" in the chart equals to 100ms) :

First, We fetch the highlight node metrics and then the baseline node metrics. Those two operations are practically the same function call but with different time parameters.
Each one of them first fetches the node's chart IDs (if they are not provided in the request body) and then concurrently for each chart ID performs a timeseries data request.

Low hanging fruits here:

• The biggest one is to perform the fetch highlight and baseline metrics operations concurrently since there is no coupling between those two. In staging each of them takes about 3 sec. so we ll pay only the time cost of one of them
• The chart_ids can be fetched only once (or even better they could be provided by the front-end in the request body since the front-end should have them in order to render the page). The gain then would be much smaller here as approximately each /charts call took about 90ms in staging.
• TODO Additional metrics should be obtained regarding the time of each individual data request and decoding. Because since those operations are performed concurrently, the total time is effectively the time of the slowest one. As mentioned, we have the total time of all those concurrent calls only, due to an incompatibility of the metrics library we use and another one that groups and executes these concurrent data requests.

Then we send the highlight and baseline metrics over to the ML service via http. This http call needs approximately 1.5 sec to return. Response decoding times here are negligible.

Some additional notes:

• Currently in staging http/2 over clear text is enabled in the ADC so the we keep http open connections to a minimum and gain time reducing handshakes etc. on cases like this one that includes a lot of http requests to the ADC.

[andrewm4894]

Great @papazach thanks for looking at this. 100% think we should do the fetch metrics at the same time, that should help a lot.

We had also talked about batching each fetch request into a subset of charts in each request to maybe also see if breaking up the fetch requests into smaller groups could also help. Might then need to just be a bit of coordination then as to if we need to wait and then send all the data to the ML service, or if we just send each batch, once we have the baseline and highlight for that batch to the ML service and then just need to have some additional ranking logic in the browser to handle potentially multiple responses from the ML service.

In this case the ML service just needs a chart data for the baseline and highlight and then gives a score per dim on the chart. So the ML service does not need to see all metrics in a single request so we could play around with this too.

[papazach]

In this case the ML service just needs a chart data for the baseline and highlight and then gives a score per dim on the chart. So the ML service does not need to see all metrics in a single request so we could play around with this too.

Nice that is also an interesting approach to keep in mind 👍

I surmise that the TODO item also yields another low to medium hanging fruit.
It is not expected from an agent to respond to a data request in 2.5-3 sec. It is at least 10 times more than we would expect.
Upon addressing those initial fixes we will also reason about this and perform some enhancements.

I think we can start addressing the low hanging fruits and then re-evaluate based on the new response times (which we expect to be from 4 to 5 sec. in total without addressing the TODO.)

[cakrit]

Note that all these requests go to the same agent, which can explain why it takes a few sec to answer all of them. If we parallelize more, the agent will be loaded even more. We may still get a better overall performance, but the agent is clearly the bottleneck here (that includes the time it takes for the responses to go through the ACLK, not just the time it takes to generate the responses).
If the bound is the ACLK throughput, then we won't get much more from it. If it's the preparation of the responses, we may get an improvement, though most likely not 2x.

[stelfrag]

In production, I noticed that the incoming queries cap at 25 requests per second. I will try in staging.

[stelfrag]

In production, I noticed that the incoming queries cap at 25 requests per second. I will try in staging.

The same in staging. The cloud does not send more than 25 requests per second.