Cooldown for Log Alerts + increased max values for Cooldown

[floriangeigl]

Describe the feature you would like to see

I love we have Cooldown for Metric based alerts! It would be really cool to have the same functionality for Log Alerts . Additionally, it be amazing if we could increase the max for the current cooldown from 1h to let's say 24h? For most alerts, I only need to get notified once daily.

Describe how you would like to see this feature implemented

Similar as it is down for metrics, but just for Log Alerts -> a simple slider :)

Describe any alternatives you've considered

No response

[amir20]

Thanks for the suggestion! I want to push back a bit here.

Cooldown makes sense for metric/event alerts since those are inherently noisy and often duplicative. But log alerts feel different to me — if a log pattern is important enough to alert on, suppressing matches for up to 24h seems risky. You could miss real incidents that happen to fire shortly after the first one.

I get that it would be opt-in/configurable, so it wouldn't hurt anyone who doesn't enable it. But personally I wouldn't use this and I'm hesitant to add a knob that encourages silencing logs. Curious what your use case looks like — is it the same pattern firing repeatedly, or different log alerts you want batched? If it's the former, debouncing at the source (fix the noisy log) might be a better path.

Bumping the existing cooldown max from 1h → 24h for metrics seems more reasonable.

Going to move this to Discussions for now to gather more input before committing to an implementation.

[floriangeigl]

Hi @amir20 - thanks for considering this and asking for more input :)

My use-case is the following: I have a home-setup where I run https://github.com/jaroslawhartman/withings-sync in a docker container on my local server. It's basically transfering body-composition/weight from withings to garmin. This needs two tokens - one for each service. After some time, the tokens need to be manually refreshed (manual login into my server running some cmd line and doing some auth-token/MFA copy&paste from my browser/phone) - until I do this, every sync fails with an error message, which represents the basis for my alerting. Now the sync happens every hour - if I'm out of town/house when it starts failing, it means I get an alert to my phone (ntfy) every hour and I can't do anything about it (apart from unsubscripe/delete the app/disbale notifications of the app temp).
So long story short: It's a situation, where something starts failing until manually resolved. I know it will fail every time from the moment after first failing. I only need one alert for such a situation - I know 100% it won't resolve on it's own.

On your suggestions: fixing the noisy log is difficult, as I'm just a user of the tool - not the creator. (and I can see why some people might be perfectly fine with the re-occuring log if they don't alert on it)

[amir20]

Thanks for the update, @floriangeigl. The problem resonates with me a lot, and I have some thoughts on it.

This issue isn't straightforward and can be tricky to implement. I try to keep Dozzle unbiased, so there isn't much logic that users might not understand. In this situation, Dozzle has two options:

1. Cooldown by notification_id: Whenever that notification_id is triggered, it runs a cooldown logic. This is similar to metrics, but it's very risky in my opinion. If one container has a warning and then suddenly encounters a significant error, it won't send the error.
2. Perform logical checks by examining the log content: This is challenging because logs tend to be similar but not identical.

Neither of these solutions is ideal for Dozzle. The first involves actions the user is unaware of, while the second relies heavily on guessing and assumptions.

As I mentioned, this problem resonates with me. I believe the logic for what to hide or show should be pushed downstream to the alerting webhook. This is the main reason I created Dozzle Cloud. I am currently working on a "personal triage" solution that uses some machine learning and large language models to determine if the most recent logs are still relevant. The idea is as follows: when an event comes in, Dozzle Cloud will send it. When the next event arrives, Cloud reviews all previous alerts and decides whether the most recent alert is worth showing to the user. If not, it simply edits the message to say, "quick follow-up: I suspended two other alerts."

This feature is currently not public, but if you're interested, I can enable it for you if you're on Cloud.

Also, I'm not trying to upsell Cloud; the free option is quite generous. I'm still debating whether to keep the triage option as free or pro, but I'll address that later.

[floriangeigl]

Hi @amir20 - thanks for taking time to look into this more :) I fully understand the issue around log pattern detection you mention in 2 - this is not trivial. I'm not sure I fully understand the problem with 1. If I setup an alert, and define a cool-down for it, I think it's abvious, that if the same log alert rule matches a slightly different log (probably more server one), that this won't create a new alert again (the cool-down is per alert-rule, not per log-line/log-pattern). Personally, I as a user would expect that this works exactly like that and in my opinion preventing situations like you mentioned (server error supressed bc a less server one triggered the same alert that is now in cool-down) is responsiblity of the user. Also, the cool-down is optional - so users can decide. Maybe a small warning to raise awareness of such situations would be good, but apart from that I wouldn't see much issue here :) IMHO anything else would be some hidden magic (like cool-down per log-pattern within the same alert), which I fully agree, should not happen :)

[amir20]

I believe you missed my point. If someone creates an alert for all container with message contains "error", then when container A has an error, it will also trigger a cooldown for container B.

Either way, probably not top of my list to work.