Telegram API size limitation 4096?

[darkfessing]

Hey there! We try to use telepush for alertmanager notifications from our clusters. But we encounter situation, when not all notifications are delivered. After some investigations:

ts=XXXXXXXXXXXXX caller=dispatch.go:352 level=error component=dispatcher msg="Notify for alerts failed" num_alerts=21 err="telepush/webhook[0]: notify retry canceled due to unrecoverable error after 1 attempts: unexpected status code 400: https://telepush.dev/api/inlets/alertmanager/XXXXXX: message too long (max is 4096 characters)"

Some software deployed with Helm have multiply labels, and it`s easy-to-go to step over this limit. How this issue can be sold? Maybe force Telepush to short notifications size for Telegram eats it?

[muety]

This is a limitation by Telegram's API itself see here, so not much we can do against that. We could technically implement some piece of logic that split a message into multiple chunks, but I'd rather refrain from implementing that, tbh.

[darkfessing]

Okay, but just in case: https://prometheus.io/docs/alerting/latest/configuration/#telegram_config

When using official docs, it's not pretty enough. But it works, message has been delivered. It's just became:
"level=warn integration=telegram msg="Truncated message"
...but most part of it is delivered (with "..." in the end).

When using telepush, we receive error, and message not delivered.

I've tried to overcame this with custom template, but it won't help:

  templates:
    limit_message_length.tmpl: |-
      {{- if gt (len .CommonAnnotations.message) 4000 }}
      {{ $shortMessage := slice .CommonAnnotations.message 0 4000 }}
      {{ $shortMessage }}
      ...
      {{- else }}
      {{ .CommonAnnotations.message }}
      {{- end }}

[muety]

This is the piece of code in alertmanager responsible for truncating the messages. We could implement the same in Wakapi. I'll create a new issue for this.