Preview of alert utilizing templating and so on

[LuboVarga]

Make some ability to show/send alert for any of active alert rules to see, if custom template will not fail in runtime (when alert should be actually produced). It will also make nice ability to test, how will alert be formatted. I imagine some UI, where I can select rule, than see number of metrics/labels set, which could potentially trigger given rule and I would choose which metrics should trigger virtually given rule. As result, all configured paths (email/slack/customWebHook/...) would be virtually triggered and data for each of them would be shown. So for email I would expect to see html formatted by custom template, as it is configured.

[brancz]

I believe @fabxc has had some thoughts on this before. Maybe he can pitch in his thoughts to the discussion.

[nogweii]

Any word on this? I am currently tweaking alertmanager templates and trying to purposefully cause errors to see what the template results in is no fun. And I'm unfamiliar with Go's text/template syntax, so I feel like I'm messing with a black box. (Not to mention the already known poor documentation in this area, see prometheus/docs#359)

[TheMeier]

My go-to workflow is to start alertmanager localy (or in a dev enviroment), you can fake the receivers to something like netcat there and create arbitrary test-alerts with amtool. You can even use this approach to create CI/CD pipelines to validate changes in templates