Add a query parameter to test notification channels

[mariliamelo]

TL;DR

Consider adding a query param to the health endpoint that fires an alert and enables testing the notification channels.

Additional information

We can create custom metrics from logging and use it in monitoring
you can alert on "last value". The problem you have to be careful about is making sure that this metric always has a value. If it goes from over threshold to no data, the alert won't recover until the next time you have good data

[sethvargo]

Added the scaffolding for this in #1187 (can be merged as-is). You'll be able to hit /healthz?service=alerts. I'm gonna defer the actual alert and its notification setup to yall.

[sethvargo]

@mariliamelo @yegle is this still something we want to do? We've pushed it off for multiple milestones now.