[Sentinel] Notification script fails to execute which fills up disk space.

[techmaniack]

Sample log entries:

1317:X 11 Nov 18:03:04.179 # -script-error /etc/notify.sh 99 0
1317:X 11 Nov 18:03:04.179 # -script-error /etc/notify.sh 99 0
1317:X 11 Nov 18:03:04.179 # -script-error /etc/notify.sh 99 0

Script.

#!/usr/bin/env bash
if [ $1 = '+promoted-slave' ]
then
	curl URI
fi

These log entries filled up entire disk which brought the host down.

In the readme it is mentioned that the notification script will be retried max of 10 times, why would it go in this dead loop?

redis version: 3.2.4

[antirez]

Thanks for reporting @techmaniack, this is an interesting thing, moreover there is a general problem with Redis in general terms: the logging system should be somewhat able to recognize duplicated entries (even not adjacent, but the same error reported multiple times in a short time) and limit it, syslog style, saying [55 times in the latest 2 minutes] ... and stuff like that. Here there is also the additional problem of the script being retried too many times, I'll check what is the exact theoretical semantics and the practical one as well.

[techmaniack]

You can close this issue if it's already fixed or not in the roadmap.

Or if you think that it's a low hanging fruit (can be worked upon by someone who have never seen the redis codebase), I could send a patch.