[ResponseOps][Discuss] prevent errors like removed references in the future

[pmuellr]

We recently found an issue with one of our APIs which ended up in a data loss situation - #152961

This issue is to discuss ways of catching issues with this earlier. Anything goes!

This is a fairly complex issue, as it involved specific rule types, which we don't test with a lot, and APIs run against those rules. After the API ran, the rule SO was damaged and needs to be deleted (or at least we don't have an easy fix at the moment).

On the face of it, the exhaustive test of this would require testing all our rule types, with different types of parameters (so more than one rule of the same type), using all of our APIs that mutate the rule SO against the rule, and then make sure it's still running.

Which is a LOT of work.

In this particular case, the issue was with "references" (saved object references saved as part of the rule params) - so at least for the "references" concern, we can narrow the rule types to two - elasticsearch query and tracking containment.

Perhaps we can just start by augmenting some existing tests. For instance, in the tests here - tests/alerting/update_api_key.ts, we could:

• use real rule types (for instance, elasticsearch query) instead of the test-specific rule types, at least for some of the tests
• have the tests ALSO test the remaining fields in the rule - they should be unchanged since before the API was called; note, not sure if the rule APIs give us EVERYTHING we want - we may want to get the docs from ES instead.

[elasticmachine]

Pinging @elastic/response-ops (Team:ResponseOps)

[XavierM]

We will review @JiaweiWu 's PR when we are done with http versioning.