Skip to content

Commit

Permalink
Update Alerts best practices round 2
Browse files Browse the repository at this point in the history
  • Loading branch information
kaileyhaynes committed Jul 14, 2021
1 parent 76acd90 commit e7a264c
Showing 1 changed file with 60 additions and 70 deletions.
Original file line number Diff line number Diff line change
Expand Up @@ -16,86 +16,42 @@ redirects:
- /docs/alerts-applied-intelligence/new-relic-alerts/get-started/alerts-best-practices
---

Improve your alert coverage by implementing the following recommendations.
Improve your Alerts coverage by implementing the following recommendations and get the most out of your alerts configuration.

Read on to learn the best practices for:
- [Policies](/docs/new-relic-solutions/best-practices-guides/alerts-applied-intelligence/alerts-best-practices/#policy-practices)
- [Notification channels](/docs/new-relic-solutions/best-practices-guides/alerts-applied-intelligence/alerts-best-practices/#channel-practices)
- [Incident preferences](/docs/new-relic-solutions/best-practices-guides/alerts-applied-intelligence/alerts-best-practices/#incident-practices)
- [Thresholds and violations](/docs/new-relic-solutions/best-practices-guides/alerts-applied-intelligence/alerts-best-practices/#threshold-practices)
- [Muting roles](/docs/new-relic-solutions/best-practices-guides/alerts-applied-intelligence/alerts-best-practices/#mute-practices)

<Callout variant="important">
Before reading this, we recommend you also read [Alerts concepts and workflow](/docs/alerts/new-relic-alerts/getting-started/alert-policy-workflow).
</Callout>
- Policies
- Notification channels
- Incident preferences
- Thresholds and violations
- Muting roles


## Recommended alerts [#recommend-alerts]

Use [recommended practices](https://discuss.newrelic.com/t/announcing-alert-recommendations-for-apm-applications/154542) if you are new to Alerts or if you want suggestions that optimize your alert coverage.
Use recommended practices if you are new to Alerts or if you want suggestions that optimize your alert coverage. See our [Explorers Hub post on recommended alerts practices](https://discuss.newrelic.com/t/announcing-alert-recommendations-for-apm-applications/154542).

## Policies [#policy-practices]

A policy is a container for alike conditions.
## Organize your policies [#policy-practices]

<Callout variant="tip">
Learn how to [create, edit, or find policies](/docs/alerts-applied-intelligence/new-relic-alerts/alert-policies/create-edit-or-find-alert-policy/) if you’re new to Alerts.
</Callout>
A policy is a container for similar [conditions](/docs/alerts-applied-intelligence/new-relic-alerts/alert-conditions/create-alert-conditions/).

If you’re new to Alerts, learn how to [create, edit, or find policies](/docs/alerts-applied-intelligence/new-relic-alerts/alert-policies/create-edit-or-find-alert-policy/).

Organize your policy by surrounding its conditions to a single entity and to the specific team that needs to be notified when a problem arises within that entity.
Organize your policy's scope to a single [entity](/docs/new-relic-one/use-new-relic-one/core-concepts/what-entity-new-relic/) when possible. Assign your policy to the essential team or teams that need to be notified when an [incident](/docs/alerts-applied-intelligence/new-relic-alerts/alert-policies/specify-when-alerts-create-incidents/) occurs. This way, you keep policies centralized and focused.

If a team is monitoring several groups of the same entity type, combine those entity clusters (like servers) together into one policy. This way, your team can be notified from one policy rather than navigating several policies at once.

Consider your team’s role when assigning them to policies:
You can use Alerts to monitor all of your entities. Consider your role and priorities when assigning yourself to a policy.

For example:

* **Software developers** may need [notifications](/docs/alerts/new-relic-alerts/managing-notification-channels/notification-channels-controlling-where-send-alerts) for both front-end and back-end performance, such as webpage response time and page load JavaScript errors.
* **Operations personnel** may need notifications for poor back-end performance, such as server memory and load averages.
* **The product owner** may need notifications for positive front-end performance, such as improved end user Apdex scores or sales being monitored in [dashboards](/docs/query-your-data/explore-query-data/dashboards/introduction-new-relic-one-dashboards).

Refrain from creating policies that include multiple teams to keep policies centralized.

## Notification channels [#channel-practices]

Tailor notifications to the most useful channel and policy so you can avoid alert fatigue and help the right personnel receive and respond to incidents they care about in a systematic way.

<Callout variant="tip">
Learn how to [set up notification channels](/docs/alerts-applied-intelligence/new-relic-alerts/alert-notifications/notification-channels-control-where-send-alerts/) if you’re new to Alerts.
</Callout>

Notify teams and individuals who:
- Can resolve a problem when an incident arises
- Need to stay updated when an incident arises

For staying updated, select a notification channel that is less intrusive like email.

For vital notifications and responding to the incident, select a notification channel that is immediate and reliable like PagerDuty or Slack. You are given a lot more control over who gets notified and when with PagerDuty and OpsGenie.

Do not rely on email for quick notifications in case of mail delays. An email list will also bounce if one is no longer valid.


## Incident preferences [#incident-practices]

Decide when you get incident notifications so you can respond to incidents when they happen.

<Callout variant="tip">
Learn more about your [incident preferences options](https://discuss.newrelic.com/t/relic-solution-alert-incident-preferences-are-the-key-to-consistent-alert-notifications/40867) if you’re new to Alerts.
</Callout>

The default incident preference setting combines all conditions within a policy into one incident. Change your default incident preference setting to better understand the exact scope and number of all violations.

Each organization will have different needs, and each policy within an organization will have different needs too. Ask your team two important questions when deciding your incident preferences:
- Do we want to be notified every time something goes wrong?
- Do we want to group all alike notifications together and be notified once?

When a policy and its conditions have a broader scope (like managing several entities), increase your incident preference sensitivity. You will need more notifications because two incidents will not necessarily relate to each other.

When a policy and its conditions have a focused scope (like managing one entity), opt for one notification. You will need less notifications when two incidents are related to each other or when the team is already notified and fixing an existing problem.

Decide how you get incident notifications by using our best [notification channel practices](/docs/new-relic-solutions/best-practices-guides/alerts-applied-intelligence/alerts-best-practices/#channel-practices).
## Set your condition thresholds and violations [#threshold-practices]


## Thresholds and violations [#threshold-practices]

Set meaningful [threshold](/docs/alerts/new-relic-alerts-beta/configuring-alert-policies/define-thresholds-trigger-alert) levels to optimize Alerts to your business. Here are some suggested guidelines:
Set meaningful [threshold](/docs/alerts/new-relic-alerts-beta/configuring-alert-policies/define-thresholds-trigger-alert) levels to optimize Alerts for your business. Here are some suggested guidelines:

<table>
<thead>
Expand Down Expand Up @@ -158,21 +114,55 @@ Set meaningful [threshold](/docs/alerts/new-relic-alerts-beta/configuring-alert-

In most of our products (except Infrastructure), the color-coded [health status indicator](/docs/using-new-relic/welcome-new-relic/get-started/glossary/#health-status) in the user interface changes as the alerting threshold escalates or returns to normal. This allows you to monitor a situation through our UI before a critical threshold passes, without needing to receive specific notifications about it.

There are two violation thresholds: critical (red) and warning (yellow). Define these thresholds with different criteria keeping in mind the suggestions above.
There are two violation thresholds: critical (red) and warning (yellow). Define these thresholds with different criteria, keeping in mind the suggestions above.

<Callout variant='important'>
Warning violations do **not** open incidents. A critical violation can open incidents, but you must define that decision through your [incident preferences](/docs/new-relic-solutions/best-practices-guides/alerts-applied-intelligence/alerts-best-practices/#incident-practices).
Warning violations do not open incidents. A critical violation can open incidents, but you must define that decision through your [incident preferences](/docs/new-relic-solutions/best-practices-guides/alerts-applied-intelligence/alerts-best-practices/#incident-practices).
</Callout>

## Muting roles [#mute-practices]

Mute alerts during routined events such as maintenance or planned downtime.
## Define your incident preferences [#incident-practices]

You can also silence another aspect of Alerts (like a policy, a specific entity, a condition) when needed. Incidents can still be opened, but you will not be notified.
Decide when you get incident notifications so you can respond to incidents when they happen.

If you’re new to Alerts, learn more about your [incident preferences options](https://discuss.newrelic.com/t/relic-solution-alert-incident-preferences-are-the-key-to-consistent-alert-notifications/40867).

The default incident preference setting combines all conditions within a policy into one incident. Change your default incident preference setting to increase or decrease the number of incidents and incident notifications you receive.

Each team within your organization will have different needs. Ask your team two important questions when deciding your incident preferences:
- Do we want to be notified every time something goes wrong?
- Do we want to group all similar notifications together and be notified once?

When a policy and its conditions have a broader scope (like managing the performance of several entities), increase the amount of incidents you receive. You will need more notifications because two incidents will not necessarily relate to each other.

When a policy and its conditions have a focused scope (like managing the performance of one entity), opt for the default incident preference. You will need less notifications when two incidents are related to each other or when the team is already notified and fixing an existing problem.

Decide how you get incident notifications by using our [notification channel best practices](/docs/new-relic-solutions/best-practices-guides/alerts-applied-intelligence/alerts-best-practices/#channel-practices).


## Select your notification channels [#channel-practices]

Tailor notifications to the most useful channel and policy so you can avoid alert fatigue and help the right personnel receive and respond to incidents they care about in a systematic way.

If you’re new to Alerts, learn how to [set up notification channels](/docs/alerts-applied-intelligence/new-relic-alerts/alert-notifications/notification-channels-control-where-send-alerts/).

Notify teams and individuals who needs to stay updated on or resolve a problem when an incident arises.

To stay updated, select a notification channel that is less intrusive, like email.

For vital notifications and incident responses, select a notification channel that is more intrusive, like PagerDuty or Slack.

Do not rely on email for quick notifications in case of delays.


## Understand muting roles [#mute-practices]

Mute alerts during routine events, such as maintenance or planned downtime.

You can also silence a policy, a specific entity, and a condition when needed. Incidents can still be opened, but you won't be notified.

If you're new to Alerts, learn how to [create and manage muting roles](/docs/alerts-applied-intelligence/new-relic-alerts/alert-notifications/muting-rules-suppress-notifications/).

<Callout variant='tip'>
Learn how to [create and manage muting roles](/docs/alerts-applied-intelligence/new-relic-alerts/alert-notifications/muting-rules-suppress-notifications/) if you’re new to Alerts.
</Callout>

## What's next?

Expand Down

0 comments on commit e7a264c

Please sign in to comment.