monitoring: encourage silencing, render entry for alerts w/o solutions #12731

bobheadxi · 2020-08-05T03:59:53Z

also adds a recommendation for how to silence each alert - see https://sourcegraph.slack.com/archives/CJX299FGE/p1596505757462700

…n't have solutions

slimsag · 2020-08-05T04:07:16Z

doc/admin/observability/alert_solutions.md

 **Possible solutions:**

 - **Get details on the exact queries that are slow** by configuring `"observability.logSlowSearches": 20,` in the site configuration and looking for `frontend` warning logs prefixed with `slow search request` for additional details.
 - **Check that most repositories are indexed** by visiting https://sourcegraph.example.com/site-admin/repositories?filter=needs-index (it should show few or no results.)
 - **Kubernetes:** Check CPU usage of zoekt-webserver in the indexed-search pod, consider increasing CPU limits in the `indexed-search.Deployment.yaml` if regularly hitting max CPU utilization.
 - **Docker Compose:** Check CPU usage on the Zoekt Web Server dashboard, consider increasing `cpus:` of the zoekt-webserver container in `docker-compose.yml` if regularly hitting max CPU utilization.

-# frontend: 90th_percentile_search_request_duration
+**Silence this alert:** If you are aware of this alert and want to silence notifications for it, add the following to your site configuration:


Instead of this:

frontend: 99th_percentile_search_request_duration

Descriptions:

frontend: 20s+ 99th percentile successful search request duration over 5m
Possible solutions:

...

Docker Compose: Check CPU usage on the Zoekt Web Server dashboard, consider increasing cpus: of the zoekt-webserver container in docker-compose.yml if regularly hitting max CPU utilization.

Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration:

{ "observability.silenceAlerts": [ "warning_frontend_99th_percentile_search_request_duration" ] }

How about this?

frontend: 99th_percentile_search_request_duration

Descriptions:

frontend: 20s+ 99th percentile successful search request duration over 5m
Possible solutions:

...

Docker Compose: Check CPU usage on the Zoekt Web Server dashboard, consider increasing cpus: of the zoekt-webserver container in docker-compose.yml if regularly hitting max CPU utilization.

Silence this alert: If you are aware of this alert and want to silence notifications for it, add the following to your site configuration:

"observability.silenceAlerts": [ "warning_frontend_99th_percentile_search_request_duration" ]

Ah, ie treat it as just another solution? that works 💯

for alerts that don't have solutions, do we want silencing to be the only solution, or should we prompt admins to open a ticket or contact us?

Silencing being the only solution is fine. I'm not worried about getting admins to contact us - they will.

Also note I remove the surrounding {} brackets which makes it shorter, and indented the code block to match the bulleted list

indented the code block to match the bulleted list

docsite doesn't support this it seems :(((

slimsag

LGTM once you address my feedback (no need for re-review).

codecov · 2020-08-05T04:14:31Z

Codecov Report

Merging #12731 into main will increase coverage by 0.00%.
The diff coverage is n/a.

@@           Coverage Diff           @@
##             main   #12731   +/-   ##
=======================================
  Coverage   50.82%   50.83%           
=======================================
  Files        1443     1443           
  Lines       81281    81281           
  Branches     6614     6560   -54     
=======================================
+ Hits        41315    41319    +4     
+ Misses      36403    36400    -3     
+ Partials     3563     3562    -1

Flag	Coverage Δ
#go	`52.20% <ø> (+<0.01%)`	⬆️
#integration	`24.20% <ø> (+0.01%)`	⬆️
#storybook	`14.11% <ø> (ø)`
#typescript	`47.07% <ø> (+<0.01%)`	⬆️
#unit	`47.51% <ø> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
.../internal/codeintel/resolvers/graphql/locations.go	`85.45% <0.00%> (ø)`
...odeintel/bundles/persistence/cache/reader_cache.go	`93.75% <0.00%> (+2.08%)`	⬆️
web/src/settings/SettingsPage.tsx	`76.47% <0.00%> (+11.76%)`	⬆️

…toring/more-silencing

pecigonzalo · 2020-08-05T12:37:40Z

I think it would be simpler to have a header that says "silencing alerts" and tells you how to do it in a generic fashion and we can reference/link that instead.

Silence an alert

If you are aware of an alert and want to silence notifications for it, add the following to your site configuration:

{
  "observability.silenceAlerts": [
    "ALERT_NAME"
  ]
}

You can find the ALERT_NAME on lorem ipsum

bobheadxi · 2020-08-05T12:52:13Z

@pecigonzalo we do have a page for that - I think the concern is you are unlikely to move away from the section once you've landed on it via an alert notification's solutions link. This gives admins a copy-paste solution that makes it clear we do have this capability, at very little cost (just a larger static page)

pecigonzalo · 2020-08-05T13:18:14Z

@bobheadxi I dont share that concern, if the have a link or a clear relationship to how to silence alerts, I think its actually more likely that someone will search for "silence alerts sourcegraph" than wait for an alert to pop up.
If we want to point them in the right direction, we could link to the "how to silence" page/section from the alert.
If its hard to find the sections and understand our documentation on how to monitor and manage alerts, we should fix that instead.

In general, I would actually not encourage silencing without expire, as its likely the silence will remain there after the issue is fixed.

bobheadxi · 2020-08-06T02:59:27Z

I think its actually more likely that someone will search for "silence alerts sourcegraph" than wait for an alert to pop up.

I'm not sure - I don't think there is a lot of awareness within our own team about this functionality, and I'd rather have this up front and center and focus on opening issues to improve alerts like @slimsag has done, rather than have us keep trying to remove alerts or have customers disable notifications altogether

In general, I would actually not encourage silencing without expire, as its likely the silence will remain there after the issue is fixed.

I think there's a possible solution for this - when customers upgrade, we can show a dismissible prompt reminding admins that they have silenced alerts and should reassess whether they should stay

I'll come up with some wording to better promote making sure silences are followed up on (d317534 ), but from discussions with stephen I think this is valuable to have at probably not a lot of cost

…ring/more-silencing

bobheadxi · 2020-08-06T08:20:31Z

In general, I would actually not encourage silencing without expire, as its likely the silence will remain there after the issue is fixed.

another idea here - we can simply change the name of the alert if there is a significant change in how it works. this might leave a lot of silences targeting alerts that no longer exist, but I feel like there is probably a programmatic way we can warn about this in the future

monitoring: encourage silencing, render entry even for alerts that do…

be96288

…n't have solutions

bobheadxi requested a review from pecigonzalo August 5, 2020 03:59

bobheadxi requested a review from slimsag as a code owner August 5, 2020 03:59

bobheadxi requested a review from a team August 5, 2020 03:59

fix spacing

c60c4de

slimsag reviewed Aug 5, 2020

View reviewed changes

slimsag approved these changes Aug 5, 2020

View reviewed changes

fix formatting

3c92cb0

bobheadxi added 2 commits August 5, 2020 12:20

add silencing alerts as a solution item

df7893c

Merge branch 'master' of github.com:sourcegraph/sourcegraph into moni…

405cb91

…toring/more-silencing

keegancsmith changed the base branch from master to main August 5, 2020 07:49

remove { and unindent

3eefe6e

bobheadxi added 3 commits August 6, 2020 11:00

Merge branch 'main' of github.com:sourcegraph/sourcegraph into monito…

4220f9c

…ring/more-silencing

re-evaluate silences

d317534

generate

7cd943b

bobheadxi merged commit cdd1b77 into main Aug 6, 2020

bobheadxi deleted the monitoring/more-silencing branch August 6, 2020 08:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

monitoring: encourage silencing, render entry for alerts w/o solutions #12731

monitoring: encourage silencing, render entry for alerts w/o solutions #12731

bobheadxi commented Aug 5, 2020

slimsag Aug 5, 2020 •

edited

bobheadxi Aug 5, 2020

slimsag Aug 5, 2020

slimsag Aug 5, 2020

bobheadxi Aug 5, 2020

slimsag left a comment

codecov bot commented Aug 5, 2020 •

edited

pecigonzalo commented Aug 5, 2020 •

edited

bobheadxi commented Aug 5, 2020 •

edited

pecigonzalo commented Aug 5, 2020

bobheadxi commented Aug 6, 2020 •

edited

bobheadxi commented Aug 6, 2020

monitoring: encourage silencing, render entry for alerts w/o solutions #12731

monitoring: encourage silencing, render entry for alerts w/o solutions #12731

Conversation

bobheadxi commented Aug 5, 2020

slimsag Aug 5, 2020 • edited

Choose a reason for hiding this comment

frontend: 99th_percentile_search_request_duration

frontend: 99th_percentile_search_request_duration

bobheadxi Aug 5, 2020

Choose a reason for hiding this comment

slimsag Aug 5, 2020

Choose a reason for hiding this comment

slimsag Aug 5, 2020

Choose a reason for hiding this comment

bobheadxi Aug 5, 2020

Choose a reason for hiding this comment

slimsag left a comment

Choose a reason for hiding this comment

codecov bot commented Aug 5, 2020 • edited

Codecov Report

pecigonzalo commented Aug 5, 2020 • edited

Silence an alert

bobheadxi commented Aug 5, 2020 • edited

pecigonzalo commented Aug 5, 2020

bobheadxi commented Aug 6, 2020 • edited

bobheadxi commented Aug 6, 2020

slimsag Aug 5, 2020 •

edited

codecov bot commented Aug 5, 2020 •

edited

pecigonzalo commented Aug 5, 2020 •

edited

bobheadxi commented Aug 5, 2020 •

edited

bobheadxi commented Aug 6, 2020 •

edited