monitoring: no easy way to trigger alerts for testing

[bobheadxi]

This is a pain point I've felt during dev, but I was able to work around this by changing thresholds manually

A bit trickier for customers - to test the alertmanager side of things (notifier config), they can do something like https://gist.github.com/cherti/61ec48deaaab7d288c9fcf17e700853a at the moment, with url='http://localhost:30090/alertmanager/api/v1/alerts' and kubectl port-forward svc/prometheus 30090, but this is not ideal, and the labels won't match what we actually have

We might want a way to say "create a test critical alert" easily. This could be a dummy 0/1 metric that has a toggle via API or UI or similar

[slimsag]

Yes, I think this is very much needed 👍

Whatever we do should work for testing both warning and critical alerts independently, I think

V1 of this could look like a GraphQL API request you make from https://sourcegraph.com/api/console that is linked in our docs. V2 could be an actual button in the UI near site config.

[bobheadxi]

Adding as a prerequisite to per-team alerting

[bobheadxi]

Added to per-team alerts project - would like to get this in before merging https://github.com/sourcegraph/deploy-sourcegraph-dot-com/pull/3073, since it'll make it easy to test

[bobheadxi]

Hm not actually important for per-team alerts, since the current implementation at #12532 doesn't support setting owners (and I think it might be a good idea to not add this - though open to thoughts)

Happy to merge https://github.com/sourcegraph/deploy-sourcegraph-dot-com/pull/3073 without it