-
Notifications
You must be signed in to change notification settings - Fork 2.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Arbitrary notifier integrations with Jsonnet #701
Comments
One thing to consider is that errors are very inconsistently signalled. There's a good mix of 4xx, 5xx and message content - sometimes within the same integration. |
It seems that accepting pull requests for new notifiers is no longer happening as the burden of maintaining them rests finally on the maintainers of alertmanager. In addition to the Telegram and Jira notifiers, AmazonSNS was not accepted (#709). |
https://github.com/hashicorp/go-plugin was recently brought to my attention, which seems like a well-used solution @gouthamve @brancz might be another issue for GSoC? |
How would that differ from the existing webhook? It looks like a slightly different way of doing the same thing. |
It adds a degree of type safety since a plugin has to implement the interface we describe (in this case, the The communication is done via a unix socket within the system, which is managed by the library. Operationally I think this could be simpler than coordinating between AM config and the webhook's location. Deploying a single docker container packaged with the correct plugins could also be a win vs. deploying AM and various webhooks (in a kubernetes-like environment). The bi-directional communication could (hopefully) make the connection between AM and a plugin feel like writing directly for the codebase. Tangentially, success with this might lead to implementing a similar plugin SD architecture in prometheus (no idea if that has been mentioned at all). |
That sounds like a more complicated and Go-specific way of doing things. There's nothing stopping users deploying all their webhooks into one binary today, the API/information a notifier gets is the same as what the webhook uses (i.e. there is no bi-directional communication), and Go plugins don't seem viable in our use case as we basically would have to distribute them all. From Prometheus SD the plan is to stick with file_sd, but there'll be an example Go program that takes the internal SD API and writes out the file_sd file. A similar thing could work here, though given the simpler use case I'm not sure it'd save much effort. |
I've written a design for how this could work with Jsonnet: https://docs.google.com/document/d/10PzPHOxrA9_3l0aYpmgt_j1NpGi1_o8f49IoEDB49Og/view Still a little draft but I'm intending to prototype this as a webhook receiver. |
Trying to do integration with IT helpdesk software Freshservice where it expects JSON in a fixed format. Since there is template support I thought I can modify the outgoing JSON content but found only later that is not for Webhooks. |
Similar to Prometheus SD integrations, Alertmanager struggles with an increasing demand for more notifier integrations. We can only maintain so many of them ourselves, contributed integrations are often not great in quality and/or have no long-term maintainers.
Many requested integrations are not designed for reliable alert delivery, yet there's an understandable demand by people with low-risk environments (personal websites etc.). As Alertmanager really is only about making single API calls, which are generally REST APIs using JSON, it's not too crazy think about templating such calls.
Jsonnet is a JSON data templating language by Google. In the past there was no Go implementation but now there's one open sourced.
From my understanding it has reasonably matured semantics (certainly better than us providing something on top of Go templating).
I'd like to consider it to allow people templating of JSON payloads with it. Combined with templating of request URLs, this should be sufficient to let people integrate with almost anything today.
I realize its throwing another templating language into the game and even nesting them if some JSON fields should be templated additionally.
But generally those would be write-once per integration and are easy to share among users without us having any maintenance burden. Ultimately, we could probably even implement a lot of the existing integrations with it.
I'd like to investigate this. Would definitely need a proposal doc though.
@stuartnelson3 @leominov @brian-brazil
The text was updated successfully, but these errors were encountered: