[Monitoring] Kibana Alerting #48464

chrisronline · 2019-10-16T20:29:37Z

Relates to #42960

This PR lays the ground work for migrating cluster alerts to Kibana alerting. Part of this involves migrating a single alert for now - xpack_license_expiration.

In addition to migrating the xpack_license_expiration alert, there is also a UI exposed in this PR that helps the user do their work for the migration. We discussed this and want to require the user to click a button to initiate the migration, which is in this PR.

Also, we need a way to disable watcher-based cluster alerts as well. This PR does not do anything for that effort, as the plan is to disable them all when all alerts are migrated. This will involve a PR to Elasticsearch at some point.

Now you may be wondering why would we merge this PR with these new UIs right now? Well, we want to progressively merge into master the work around this migration, so we added a new constant in the code that indicates if kibana alerting is enabled or not for monitoring. For devs and testers, you will need to change this to true for any of this code to work. Once we are done migrating all alerts, we can just remove that constant.

Here are some screenshots of the UI flow:

This is what users will see when they first come into the app (assuming the constant is true)

Once they enter setup mode, they will see this:

Clicking the button will prompt a modal indicating the user needs to configure an email action (I already had one but if the user didn't have one, they'd be taken to the next screenshot form modal right away)

Clicking edit will show them this kind of form

Once the email action is correctly applied, they will see a loading state

Then they will see this state:

Testing

You'll need to use a PR from Elasticsearch which blacklists the necessary cluster alerts. Ensure you have ES cloned (do so, if not) in a sibling directory to kibana (like /dev/repos/kibana and /dev/repos/elasticsearch), pull down this PR, then run yarn es source inside of kibana to use that custom ES build.
Add these to your kibana.dev.yml:

xpack.alerting.enabled: true
xpack.actions.enabled: true

Add this to your elasticsearch.yml:

xpack.security.authc.api_key.enabled: true

Flip the constant to true
Ensure you have a trial, gold, or platinum license active
Ensure internal monitoring is enabled (Fairly sure that MB monitoring does not have a way to initiate the cluster alerts, cc @cachedout to verify)
Go to dev tools and ensure you only see 5 watches: GET .watches/_search?filter_path=hits.total.value
Use the UI to create an email action and migrate the xpack license expiration alert
For creating the action, if you are using gmail, I'd recommend reading this and this
Simulate a license expiration to receive an email. One way is to:

Add an ingest pipeline:

PUT _ingest/pipeline/force_license_expiration
{
  "processors": [
    {
      "script": {
        "lang": "painless",
        "source": "ctx.license.expiry_date_in_millis = Instant.ofEpochMilli(new Date().getTime()).plusSeconds(60 * 60 * 24 * 3).getEpochSecond() * 1000;"
      }
    }
  ]
}

Disable the default pipeline:

PUT _cluster/settings
{
  "persistent": {
    "xpack.monitoring.exporters": { 
      "local": {
        "type": "local",
        "use_ingest": false
      }
    }
  }
}

Set this as the default pipeline for .monitoring-es-* indices:

PUT .monitoring-es-7-*/_settings
{
  "index.required_pipeline": "force_license_expiration"
}

This should cause the alert to fire. You can get it to the "resolved" state by simply removing the ingest pipeline from the index:

PUT .monitoring-es-7-*/_settings
{
  "index.required_pipeline": null
}

TODO

Maybe we should do some kind of verifying of the email credentials before allowing them to create the action?
~~Add debug logs to help understand why the alert isn't firing~~ -> 5734cc3

…ration

…after creating alerts

elasticmachine · 2019-10-16T21:51:34Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: faf4ad2

elasticmachine · 2019-10-17T19:05:34Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: de90fe4

elasticmachine · 2019-10-18T00:36:17Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: 9b681d8

elasticmachine · 2019-10-18T16:03:05Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: 3506bd1

…alerts

elasticmachine · 2019-10-18T21:40:03Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: 9ac59c1

elasticmachine · 2019-10-21T15:39:03Z

Pinging @elastic/stack-monitoring (Team:Monitoring)

…alerts

cachedout · 2019-10-22T13:42:29Z

Hi @chrisronline

When I start Kibana (after doing a fresh yarn kbn bootstrap, I get this error in the Kibana logs:

FATAL Error: Unmet requirement "alerting" for plugin "monitoring"

I'm using the ES build as described in the setup instructions along with a checkout of Kibana. Is there something else I need?

chrisronline · 2019-10-22T13:59:12Z

@cachedout My bad! I forgot to include the necessary config changes in the PR description. The testing section is now updated to include this.

cc @mikecote

The kibana.yml settings:

xpack.alerting.enabled: true
xpack.actions.enabled: true

and the elasticsearch.yml settings:

xpack.security.authc.api_key.enabled: true

Are there any plans to enable these by default? Or do we need to check against these in our code and tell the user they need to flip these for Kibana alerting to work?

cachedout · 2019-10-22T14:34:16Z

@chrisronline OK, I got it started and I clicked on the button to migrate alerts, but Kibana went to a white overlay over the monitoring screen and nothing happened after that. What is the expected behavior?

(Github won't let me upload this screenshot for some reason but I can show you via a screen share if you want)

chrisronline · 2019-10-22T14:34:53Z

@cachedout There is most likely a javascript error. Can you check the dev tools?

cachedout · 2019-10-22T14:36:47Z

@chrisronline My bad. Should have done that right away. Below are the errors show in the console:

The above error occurred in the <ManageEmailAction> component:
    in ManageEmailAction (created by CreateActionModal)
    in div (created by EuiModalBody)
    in div (created by EuiModalBody)
    in EuiModalBody (created by CreateActionModal)
    in div (created by EuiModal)
    in div (created by EuiModal)
    in div (created by FocusLock)
    in FocusLock (created by EuiFocusTrap)
    in EuiFocusTrap (created by EuiModal)
    in EuiModal (created by CreateActionModal)
    in EuiOverlayMask (created by CreateActionModal)
    in CreateActionModal (created by MigrationStatus)
    in div (created by EuiText)
    in EuiText (created by EuiCallOut)
    in div (created by EuiCallOut)
    in EuiCallOut (created by MigrationStatus)
    in div (created by EuiFlexItem)
    in EuiFlexItem (created by MigrationStatus)
    in div (created by EuiFlexGroup)
    in EuiFlexGroup (created by MigrationStatus)
    in MigrationStatus (created by Overview)
    in main (created by EuiPageBody)
    in EuiPageBody (created by Overview)
    in div (created by EuiPage)
    in EuiPage (created by Overview)
    in Overview (created by SetupModeRenderer)
    in SetupModeRenderer
    in EuiContext (created by I18nContext)
    in PseudoLocaleWrapper (created by I18nProvider)
    in IntlProvider (created by I18nProvider)
    in I18nProvider (created by I18nContext)
    in I18nContext

Consider adding an error boundary to your tree to customize error handling behavior.
Visit https://fb.me/react-error-boundaries to learn more about error boundaries. react-dom.development.js:16911:13
    React 13
    unstable_runWithPriority webpack://[name]/./node_modules/scheduler/cjs/scheduler.development.js?:255
    React 7
TypeError: action is null
manage_email_action.tsx:54
TypeError: action is null
manage_email_action.tsx:54
Warning: Can't perform a React state update on an unmounted component. This is a no-op, but it indicates a memory leak in your application. To fix, cancel all subscriptions and asynchronous tasks in the componentWillUnmount method.
    in EuiIcon (created by EuiButtonIcon)
    in button (created by EuiButtonIcon)
    in EuiButtonIcon (created by Context.Consumer)
    in EuiI18n (created by EuiModal)
    in div (created by EuiModal)
    in div (created by FocusLock)
    in FocusLock (created by EuiFocusTrap)
    in EuiFocusTrap (created by EuiModal)
    in EuiModal (created by CreateActionModal)
    in EuiOverlayMask (created by CreateActionModal)
    in CreateActionModal (created by MigrationStatus)
    in div (created by EuiText)
    in EuiText (created by EuiCallOut)
    in div (created by EuiCallOut)
    in EuiCallOut (created by MigrationStatus)
    in div (created by EuiFlexItem)
    in EuiFlexItem (created by MigrationStatus)
    in div (created by EuiFlexGroup)
    in EuiFlexGroup (created by MigrationStatus)
    in MigrationStatus (created by Overview)
    in main (created by EuiPageBody)
    in EuiPageBody (created by Overview)
    in div (created by EuiPage)
    in EuiPage (created by Overview)
    in Overview (created by SetupModeRenderer)
    in SetupModeRenderer
    in EuiContext (created by I18nContext)
    in PseudoLocaleWrapper (created by I18nProvider)
    in IntlProvider (created by I18nProvider)
    in I18nProvider (created by I18nContext)
    in I18nContext react-dom.development.js:506:32
    React 5
    EuiIcon webpack://[name]/./node_modules/@elastic/eui/lib/components/icon/icon.js?:464
Warning: Can't perform a React state update on an unmounted component. This is a no-op, but it indicates a memory leak in your application. To fix, cancel all subscriptions and asynchronous tasks in the componentWillUnmount method.
    in SetupModeRenderer
    in EuiContext (created by I18nContext)
    in PseudoLocaleWrapper (created by I18nProvider)
    in IntlProvider (created by I18nProvider)
    in I18nProvider (created by I18nContext)
    in I18nContext react-dom.development.js:506:32
Warning: Can't perform a React state update on an unmounted component. This is a no-op, but it indicates a memory leak in your application. To fix, cancel all subscriptions and asynchronous tasks in a useEffect cleanup function.
    in MigrationStatus (created by Overview)
    in main (created by EuiPageBody)
    in EuiPageBody (created by Overview)
    in div (created by EuiPage)
    in EuiPage (created by Overview)
    in Overview (created by SetupModeRenderer)
    in SetupModeRenderer
    in EuiContext (created by I18nContext)
    in PseudoLocaleWrapper (created by I18nProvider)
    in IntlProvider (created by I18nProvider)
    in I18nProvider (created by I18nContext)
    in I18nContext react-dom.development.js:506:32

```


^^ UPDATED PREVIOUS DUE TO BAD PASTE

chrisronline · 2019-10-22T14:37:58Z

@cachedout Thanks - that's actually just a warning. Are there any console logs that say Error or Exception?

cachedout · 2019-10-22T14:39:23Z

@chrisronline I initially only pasted part of the output. Please refresh. There is an error included now:


Content Security Policy: The page’s settings blocked the loading of a resource at inline (“script-src”). monitoring:1:1
Content Security Policy: The page’s settings blocked the loading of a resource at inline (“script-src”). monitoring:372:1
^ A single error about an inline script not firing due to content security policy is expected! bootstrap.js:9:20
Content Security Policy: The page’s settings blocked the loading of a resource at inline (“script-src”). monitoring:600:159
Download the React DevTools for a better development experience: https://fb.me/react-devtools react-dom.development.js:21052:17
Testbed plugin set up plugin.ts:43
Testbed plugin started plugin.ts:52
Download the React DevTools for a better development experience: https://fb.me/react-devtools react-dom.development.js:21258:17
INFO: 2019-10-22T14:35:59Z
  Adding connection to http://localhost:5601/tzm/elasticsearch

elasticsearch.angular.js:31587:16
The above error occurred in the <ManageEmailAction> component:
    in ManageEmailAction (created by CreateActionModal)
    in div (created by EuiModalBody)
    in div (created by EuiModalBody)
    in EuiModalBody (created by CreateActionModal)
    in div (created by EuiModal)
    in div (created by EuiModal)
    in div (created by FocusLock)
    in FocusLock (created by EuiFocusTrap)
    in EuiFocusTrap (created by EuiModal)
    in EuiModal (created by CreateActionModal)
    in EuiOverlayMask (created by CreateActionModal)
    in CreateActionModal (created by MigrationStatus)
    in div (created by EuiText)
    in EuiText (created by EuiCallOut)
    in div (created by EuiCallOut)
    in EuiCallOut (created by MigrationStatus)
    in div (created by EuiFlexItem)
    in EuiFlexItem (created by MigrationStatus)
    in div (created by EuiFlexGroup)
    in EuiFlexGroup (created by MigrationStatus)
    in MigrationStatus (created by Overview)
    in main (created by EuiPageBody)
    in EuiPageBody (created by Overview)
    in div (created by EuiPage)
    in EuiPage (created by Overview)
    in Overview (created by SetupModeRenderer)
    in SetupModeRenderer
    in EuiContext (created by I18nContext)
    in PseudoLocaleWrapper (created by I18nProvider)
    in IntlProvider (created by I18nProvider)
    in I18nProvider (created by I18nContext)
    in I18nContext

Consider adding an error boundary to your tree to customize error handling behavior.
Visit https://fb.me/react-error-boundaries to learn more about error boundaries. react-dom.development.js:16911:13
TypeError: action is null
manage_email_action.tsx:54
TypeError: action is null
manage_email_action.tsx:54
Warning: Can't perform a React state update on an unmounted component. This is a no-op, but it indicates a memory leak in your application. To fix, cancel all subscriptions and asynchronous tasks in the componentWillUnmount method.
    in EuiIcon (created by EuiButtonIcon)
    in button (created by EuiButtonIcon)
    in EuiButtonIcon (created by Context.Consumer)
    in EuiI18n (created by EuiModal)
    in div (created by EuiModal)
    in div (created by FocusLock)
    in FocusLock (created by EuiFocusTrap)
    in EuiFocusTrap (created by EuiModal)
    in EuiModal (created by CreateActionModal)
    in EuiOverlayMask (created by CreateActionModal)
    in CreateActionModal (created by MigrationStatus)
    in div (created by EuiText)
    in EuiText (created by EuiCallOut)
    in div (created by EuiCallOut)
    in EuiCallOut (created by MigrationStatus)
    in div (created by EuiFlexItem)
    in EuiFlexItem (created by MigrationStatus)
    in div (created by EuiFlexGroup)
    in EuiFlexGroup (created by MigrationStatus)
    in MigrationStatus (created by Overview)
    in main (created by EuiPageBody)
    in EuiPageBody (created by Overview)
    in div (created by EuiPage)
    in EuiPage (created by Overview)
    in Overview (created by SetupModeRenderer)
    in SetupModeRenderer
    in EuiContext (created by I18nContext)
    in PseudoLocaleWrapper (created by I18nProvider)
    in IntlProvider (created by I18nProvider)
    in I18nProvider (created by I18nContext)
    in I18nContext react-dom.development.js:506:32
    React 5
    EuiIcon webpack://[name]/./node_modules/@elastic/eui/lib/components/icon/icon.js?:464
Warning: Can't perform a React state update on an unmounted component. This is a no-op, but it indicates a memory leak in your application. To fix, cancel all subscriptions and asynchronous tasks in the componentWillUnmount method.
    in SetupModeRenderer
    in EuiContext (created by I18nContext)
    in PseudoLocaleWrapper (created by I18nProvider)
    in IntlProvider (created by I18nProvider)
    in I18nProvider (created by I18nContext)
    in I18nContext react-dom.development.js:506:32
Warning: Can't perform a React state update on an unmounted component. This is a no-op, but it indicates a memory leak in your application. To fix, cancel all subscriptions and asynchronous tasks in a useEffect cleanup function.
    in MigrationStatus (created by Overview)
    in main (created by EuiPageBody)
    in EuiPageBody (created by Overview)
    in div (created by EuiPage)
    in EuiPage (created by Overview)
    in Overview (created by SetupModeRenderer)
    in SetupModeRenderer
    in EuiContext (created by I18nContext)
    in PseudoLocaleWrapper (created by I18nProvider)
    in IntlProvider (created by I18nProvider)
    in I18nProvider (created by I18nContext)
    in I18nContext react-dom.development.js:506:32

chrisronline · 2019-10-22T14:41:01Z

@cachedout Thanks, I thought I fixed that before I put the PR up (since I saw the same thing), but I apparently didn't.

Pushed up now and ready to go!

mikecote · 2019-10-22T15:24:01Z

@chrisronline

Are there any plans to enable these by default? Or do we need to check against these in our code and tell the user they need to flip these for Kibana alerting to work?

We have a task to either throw an error, log warning or disable the plugin whenever security is enabled but not SSL. It should be developer / user friendly with a clear message. We'll resolve this with https://github.com/elastic/kibana/projects/26#card-25410223.

mikecote · 2019-10-22T15:26:55Z

@chrisronline also I think if your question was in regards to the actions and alerting plugins being enabled by default. They will be when we're ready for beta. But until then we're using them as a feature flag.

elasticmachine · 2019-10-22T15:31:01Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: fc2227a

cachedout · 2019-10-22T15:50:53Z

@chrisronline I went through the steps and I think I configured an alert as described, but when the alert fires I don't receive any email and the Kibana log shows the following:

server log [17:49:10.294] [warning][monitoring][plugins] Unable to send email for monitoring_alert_type_license_expiration because there is no email configured.

What's the best way to see if I migrated the alert correctly and what its current configuration is?

chrisronline · 2019-10-22T15:53:33Z

@cachedout The email in that log is xpack.monitoring.cluster_alerts.email_notifications.email_address in kibana.dev.yml. We discussed keeping this the same, but since we aren't doing a slow rollout, we can change this so it's a prompt in the UI.

elasticmachine · 2019-10-22T16:53:16Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: b2d1bfc

elasticmachine · 2019-10-22T18:39:37Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: 5734cc3

… storing the email to receive alerts

elasticmachine · 2019-10-23T21:31:23Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: 1f3d311

elasticmachine · 2019-10-24T01:57:31Z

💔 Build Failed

continuous-integration/kibana-ci/pull-request
Commit: 4e9179c

chrisronline · 2019-10-24T13:33:45Z

Going to open a new PR for this when I'm done making some changes

chrisronline · 2019-10-24T16:54:50Z

See #49219

chrisronline added 2 commits October 16, 2019 16:28

License expiration alert mostly working

0747336

More working for license_expiration alert, including UI flows for mig…

0a9048e

…ration

chrisronline mentioned this pull request Oct 16, 2019

[Stack Monitoring] Alerting Phase -1 #42960

Open

10 tasks

Ensure we add the alert to the list so we can render the right state …

faf4ad2

…after creating alerts

Refactor the UI code to be more separated

de90fe4

Move to Typescript

9b681d8

Cleanup

3506bd1

chrisronline added 5 commits October 18, 2019 15:11

Fix issues with moment usage

8a7b079

Really fix this moment issue

1da2f88

Add tests for alert

55494bd

Disable alerting for the PR

9ac59c1

Merge remote-tracking branch 'elastic/master' into monitoring/kibana_…

3e2c2b3

…alerts

chrisronline self-assigned this Oct 21, 2019

chrisronline marked this pull request as ready for review October 21, 2019 15:38

chrisronline requested review from cachedout and igoristic October 21, 2019 15:38

chrisronline added Feature:Stack Monitoring review Team:Monitoring Stack Monitoring team v7.5.0 v8.0.0 labels Oct 21, 2019

chrisronline added release_note:skip Skip the PR/issue when compiling release notes v7.6.0 and removed v7.5.0 labels Oct 21, 2019

Merge remote-tracking branch 'elastic/master' into monitoring/kibana_…

cfd0555

…alerts

Fix JS error

450080d

Fix type issues

fc2227a

Fix broken test

b2d1bfc

Add better logging

5734cc3

Refactor a bit, change the UI around and introduce a saved object for…

1f3d311

… storing the email to receive alerts

Fix log levels and broken tests

4e9179c

chrisronline closed this Oct 24, 2019

chrisronline deleted the monitoring/kibana_alerts branch October 24, 2019 13:34

chrisronline restored the monitoring/kibana_alerts branch October 24, 2019 13:34

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Monitoring] Kibana Alerting #48464

[Monitoring] Kibana Alerting #48464

chrisronline commented Oct 16, 2019 •

edited

elasticmachine commented Oct 16, 2019

elasticmachine commented Oct 17, 2019

elasticmachine commented Oct 18, 2019

elasticmachine commented Oct 18, 2019

elasticmachine commented Oct 18, 2019

elasticmachine commented Oct 21, 2019

cachedout commented Oct 22, 2019

chrisronline commented Oct 22, 2019

cachedout commented Oct 22, 2019

chrisronline commented Oct 22, 2019

cachedout commented Oct 22, 2019 •

edited

chrisronline commented Oct 22, 2019

cachedout commented Oct 22, 2019

chrisronline commented Oct 22, 2019

mikecote commented Oct 22, 2019

mikecote commented Oct 22, 2019 •

edited

elasticmachine commented Oct 22, 2019

cachedout commented Oct 22, 2019 •

edited

chrisronline commented Oct 22, 2019

elasticmachine commented Oct 22, 2019

elasticmachine commented Oct 22, 2019

elasticmachine commented Oct 23, 2019

elasticmachine commented Oct 24, 2019

chrisronline commented Oct 24, 2019

chrisronline commented Oct 24, 2019

[Monitoring] Kibana Alerting #48464

[Monitoring] Kibana Alerting #48464

Conversation

chrisronline commented Oct 16, 2019 • edited

Testing

TODO

elasticmachine commented Oct 16, 2019

💔 Build Failed

elasticmachine commented Oct 17, 2019

💔 Build Failed

elasticmachine commented Oct 18, 2019

💔 Build Failed

elasticmachine commented Oct 18, 2019

💔 Build Failed

elasticmachine commented Oct 18, 2019

💔 Build Failed

elasticmachine commented Oct 21, 2019

cachedout commented Oct 22, 2019

chrisronline commented Oct 22, 2019

cachedout commented Oct 22, 2019

chrisronline commented Oct 22, 2019

cachedout commented Oct 22, 2019 • edited

chrisronline commented Oct 22, 2019

cachedout commented Oct 22, 2019

chrisronline commented Oct 22, 2019

mikecote commented Oct 22, 2019

mikecote commented Oct 22, 2019 • edited

elasticmachine commented Oct 22, 2019

💔 Build Failed

cachedout commented Oct 22, 2019 • edited

chrisronline commented Oct 22, 2019

elasticmachine commented Oct 22, 2019

💔 Build Failed

elasticmachine commented Oct 22, 2019

💔 Build Failed

elasticmachine commented Oct 23, 2019

💔 Build Failed

elasticmachine commented Oct 24, 2019

💔 Build Failed

chrisronline commented Oct 24, 2019

chrisronline commented Oct 24, 2019

chrisronline commented Oct 16, 2019 •

edited

cachedout commented Oct 22, 2019 •

edited

mikecote commented Oct 22, 2019 •

edited

cachedout commented Oct 22, 2019 •

edited