[AO] Test scenarios for the new threshold rule

[maryam-saeidi]

📝 Summary

We want to have good test coverage for the new threshold rule. In this ticket, we cover scenarios that we want to implement for this new rule.
We need to cover AAD fields as much as possible for all scenarios. We can ignore checking all the fields.

Test location
https://github.com/elastic/kibana/tree/main/x-pack/test/alerting_api_integration/observability/custom_threshold_rule

We also need to consider testing permission. We can use different permissions for different scenarios and not add another scenario for it.

• one with a superuser
• one with write access for that feature
• one with read access for that feature

🧾 Scenario 1 - Conditions and fields types

• Multiple conditions with different types of fields (such as number, string, percent) without grouping (Can we have one condition for every condition type? 😄)
  -- Verifying if the reason was generated in the correct field format
  -- Make sure to at least have one for document count and one for max/min/average/... (When we add percentile and rate, we need to cover them separately too)

🧾 Scenario 2 - No data

• No data alert without grouping

🧾 Scenario 3 - Missing group

• Group alerts based on a field and make one of the group disappears to get the missing group alert

🧾 Scenario 4 - Grouping

• Group alerts based on one of the fields that have additional context (like host.name)
  -- Make sure that the context for the group is preserved correctly in the AAD (like host.ip)
  -- Possibly validate multiple contextual fields with different value types (like keyword, long, array, single value)
  -- Make sure the number of generated alerts matches the number of groups
  -- It would also be good to check the tags field, it should consist of source and rule tags

🧾 Scenario 5 - Filtering

• Filter data and make sure the generated alert is triggered correctly based on the filter

🧾 Scenario 6 - Equation

• Create a complicated equation with multiple variables to test the result of the evaluation

✅ Acceptance Criteria

• Implement the above-mentioned test scenarios as API integration tests

⚙️ Implementation tickets

• [AO] Add integration test to the Threshold Rule #160510
• [AO] Create Serverless integration tests for the Threshold rule  #161458
• [AO] Add API integration test scenarios to the Threshold rule  #162491
• [AO] Add action variables testing for the new threshold rule #162950

[elasticmachine]

Pinging @elastic/actionable-observability (Team: Actionable Observability)

[maryam-saeidi]

@fkanout @simianhacker Any idea how to distribute checking permissions for different scenarios? How many different permission levels should we check?

[simianhacker]

For Scenario 4 I would make sure you include indexing a "non-standard" group by field. It can be made up like processor.outcome or something that's not included in the standard ECS library.