Custom webhook converters for Grafana, GitHub, ... (was: pretty grafana notifications?)

[DoTheEvo]

Love ntfy.
For grafana it works, but shows just plain ugly json on its own.
I use Grafana-to-ntfy to make it look good.

But I just noticed that straight up on ntfy homepage demo pictures, theres pretty grafana notification. Did you use also grafana-to-ntfy to make it so, or is it build in and needs some enabling?

Feels kinda clunky to be running another container to get it working.

[binwiederhier]

What you see there on the website is an "elaborate" fake 😄 It's not real.

I have Grafana alerts set up on my phone as well, and I get a giant JSON blob to ntfy and I hate it. However, if I start adding endpoints for one thing, everything under the sun now wants to add endpoints too: GitHub, Grafana, Prometheus, Uptime Kuma, all the things.

I suppose I could add a plugin-type thing that would allow incoming webhooks to be processed .... 🤔

[binwiederhier]

Thinking out loud:

Thoughts:

curl -d '{"alert":"something is down"}` ntfy.sh/alerts/$converter

e.g.

curl -d '{"alert":"something is down"}` ntfy.sh/alerts/grafana
curl -d '{"alert":"something is down"}` ntfy.sh/alerts/github
curl -d '{"alert":"something is down"}` ntfy.sh/alerts/myconverter
...

We could have built-in ones, and custom ones:

message-converter-dir: /etc/ntfy/converter.d

with scripts like this (e.g. /etc/ntfy/converter.d/myconverter):

#!/bin/bash
jq '{ message: .alert }'

FYI:

echo '{"alert":"hi there"}' | jq '{ message: .alert }'
{
  "message": "hi there"
}

The biggest problem with this is that this would run a subshell for every incoming message that uses an external converter, which can easily blow up the server.

[binwiederhier]

I guess we could use Go plugins (https://pkg.go.dev/plugin), though that would limit it to Go.

Or we could use an IPC approach using STDIN/STDOUT. PowerDNS uses that for its remote backend, and that scales quite well.

A converter script would look like this. Reading messages on STDIN (one line per message, or whatever format we decide), and then output the converted message on STDOUT (in one line):

$ cat myconverter
#!/bin/bash

while read input; do
	echo "$input" | jq -cr .alert
done

Here's an example usage (I manually typed the {"alert"...} messags:

$ ./myconverter
{"alert":"hi there"}
hi there
{"alert":"hi what's up"}
hi what's up

The problem with that is that is not necessarily in our control how messages come in to ntfy. So if Grafana send the JSON as pretty-print the whole one line thing goes out the window. 🤔

[wunter8]

What about using a library like this? https://github.com/buger/jsonparser

It will let you specify a path to a specific value in the JSON object. And it does everything in Go. So you could set up a YAML config like this

converters:
  - grafana:
    title: object.key
    message: object.nested.nested.key

Then you split the value of title and message at "." into a bytes array (slice?) and give it to jsonparser's getString command. It also supports a key like [0] for indexing into a list in the JSON object

[binwiederhier]

Just mentioning that I implemented a "templating approach" a while back here: #171 + #104

[skurfuerst]

If you wanted to go further here, you could also check out https://github.com/hashicorp/go-plugin - this also might be useful.

All the best,
Sebastian

[skurfuerst]

Btw I am unsure whether these kinds of conversions should happen in ntfy; or if this should be done e.g. in Caddy outside ntfy.