Fix bug in filtering Rules' log history in stack management

[maryam-saeidi]

🐞 Summary

When checking Rules' logs in stack management, the filter does not work as expected (slack, QA environment)

Screen.Recording.2023-11-14.at.09.32.27.mov

Besides not filtering correctly, in the first search, the table had rows, but in the second one, it didn’t.

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[Zacqary]

Only able to reproduce this when there are more than 1000 logs to filter through, and even then only some filters fail and some succeed. It's somehow related to having a lot of data. Investigating further.

[Zacqary]

Welp, here's our problem:

kibana/x-pack/plugins/actions/server/lib/get_execution_log_aggregation.ts

Line 16 in 942a8b1

const DEFAULT_MAX_BUCKETS_LIMIT = 1000; // do not retrieve more than this number of executions

const DEFAULT_MAX_BUCKETS_LIMIT = 1000; // do not retrieve more than this number of executions
const DEFAULT_MAX_KPI_BUCKETS_LIMIT = 10000;

I'm not sure why the limits for log buckets and KPI buckets are different. That seems odd, and a quick change of the limit to 10000 doesn't seem to break anything when testing locally.

The first goal is to figure out a better failure mode for when we attempt to aggregate over more than 1000 logs, and then we should figure out if syncing this bucket limit with the KPI is a major issue or not.

[Zacqary]

Ok this is the problem. We have two different aggregations determining the total number of logs, and the actual logs themselves:

executionUuidCardinality: {
  filter: {
    size: DEFAULT_MAX_BUCKETS_LIMIT,
    bool: {
      ...(dslFilterQuery ? { filter: dslFilterQuery } : {}),
      must: [getProviderAndActionFilter('alerting', 'execute')],
    },
  },
  aggs: {
    executionUuidCardinality: {
      cardinality: {
        field: EXECUTION_UUID_FIELD,
      },
    },
  },
},
executionUuid: {
  // Bucket by execution UUID
  terms: {
    field: EXECUTION_UUID_FIELD,
    size: DEFAULT_MAX_BUCKETS_LIMIT,
    order: formatSortForTermSort(sort),
  },

executionUuidCardinality is used to determine the total number of buckets, but it isn't subject to the DEFAULT_MAX_BUCKETS_LIMIT. Therefore it's not an accurate count of the total number of logs actually available to executionUuid. There doesn't seem to be a way to apply a size prop to a cardinality metric, so I don't know how to get this to be an accurate total.

Anyway, the logs API sends back the cardinality as the data table's total, and then the logs as the actual data to insert in it. When the total exceeds the length of the data, the table puts in all these empty rows.

I recommend syncing up the DEFAULT_MAX_BUCKETS_LIMIT and DEFAULT_MAX_KPI_BUCKETS_LIMIT. The UI limits data display to 1000 all by itself, but we're already doing the same amount of work to fetch the KPIs, so it's not too much of a wasted effort.

We can then pull the data table total from the KPI aggregation instead of the cardinality.