[Monitoring] Cluster state watch to Kibana alerting

[chrisronline]

Builds upon #54306
Relates to #42960

This PR migrates the next cluster alert, the cluster state one.

Most of the changes in the PR are due to the initial work not working that well for multiple alerts, so some refactoring was necessary.

In particular, the way in which we tell the client what to show in the alert in the UI changed. The original implementation didn't support what we needed to do with other alerts so that's been changed and hopefully makes sense. It's a bit of a "generic" approach and I am welcome to any better approaches here.

Testing

• Set KIBANA_ALERTING_ENABLED to true in both legacy/plugins/monitoring/common/constants.js and plugins/monitoring/common/constants.ts
• Navigate to the monitoring app and follow the instructions in the alerting callout at the top to enable Kibana alerting
• If you do not see an alert in the UI yet, create an index using dev tools and go back to the monitoring UI and you should see one about replica shards.

Screenshots

Creation

Post creation

TODO

• This cluster alert has a link to the indices page with an action label that indicates you can do something about it. Should we change this since you can't really do anything on that page? cc @sorantis

[elasticmachine]

Pinging @elastic/stack-monitoring (Team:Monitoring)

[chrisronline]

@elasticmachine merge upstream

[kibanamachine]

💚 Build Succeeded

• continuous-integration/kibana-ci/pull-request
• Commit: 27dcca4

History

• 💚 Build #37958 succeeded 0720c7a
• 💚 Build #37588 succeeded 0c8dda5
• 💔 Build #37521 failed 845ee7b
• 💔 Build #37263 failed 9a7746a
• 💚 Build #37146 succeeded f60cf3a

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

[igoristic]

nit: Should this maybe be if (alerts.total > 1) {? Since it can still be a different falsy value

[igoristic]

Looking pretty good 🥇

[spalger]

The i18n tests started failing once this was merged, so I've reverted it for now: https://kibana-ci.elastic.co/job/elastic+kibana+master/4238/execution/node/148/log/

13:54:13  [20:54:12] Validating Default Messages [completed]
13:54:13  [20:54:12] Compatibility Checks [started]
13:54:13  [20:54:12] Compatibility check with x-pack/plugins/translations/translations/zh-CN.json [started]
13:54:13  [20:54:13] Compatibility check with x-pack/plugins/translations/translations/zh-CN.json [failed]
13:54:13  [20:54:13] → 
13:54:13  Incompatible translation: "values" object contains unused properties ("xpack.monitoring.alerts.licenseExpiration.ui.firingMessage"): [linkText].
13:54:13  
13:54:13  [20:54:13] Compatibility Checks [failed]
13:54:13  [20:54:13] → 
13:54:13  Incompatible translation: "values" object contains unused properties ("xpack.monitoring.alerts.licenseExpiration.ui.firingMessage"): [linkText].
13:54:13  
13:54:13  ERROR Unhandled exception!
13:54:13  ERROR Error: 
13:54:13        Incompatible translation: "values" object contains unused properties ("xpack.monitoring.alerts.licenseExpiration.ui.firingMessage"): [linkText].
13:54:13  
13:54:13            at createFailError (/var/lib/jenkins/workspace/elastic+kibana+master/kibana/packages/kbn-dev-utils/target/run/fail.js:25:26)
13:54:13            at verifyMessages (/var/lib/jenkins/workspace/elastic+kibana+master/kibana/src/dev/i18n/integrate_locale_files.ts:109:11)
13:54:13            at integrateLocaleFiles (/var/lib/jenkins/workspace/elastic+kibana+master/kibana/src/dev/i18n/integrate_locale_files.ts:191:3)
13:54:13  Warning: non-zero exit code 1� Use --force to continue.
13:54:13