Getting Inconsistent push notifications for new container updates

[begunfx]

Drydock version

1.5.0-rc.3

What happened?

I saw a new update for netbox but did not see any pushover notifications come through for it. Other updates have come through so I don't think it's a complete breakdown in the notifications.

What did you expect?

See above

Steps to reproduce

Not sure how to reproduce.

Relevant configuration

Logs

Installation method

Docker Compose

Docker version

24.0.1

OS / Architecture

Synology DSM 7

[s-b-e-n-s-o-n]

Thanks for the report — traced the root cause and it's now fixed on the RC5 branch.

Root cause

isThresholdReached() in trigger-threshold.ts was unconditionally rejecting updates with updateKind.kind === 'unknown' before checking whether the trigger threshold was set to all. Created-date-only updates (same tag, rebuilt image with a newer creation timestamp) produce updateKind: 'unknown' — so they showed as available in the UI but silently skipped all trigger notifications, even with the default threshold: all.

Fix

Moved the all threshold short-circuit ahead of the unknown rejection, so threshold: all now fires for every update regardless of kind. Semver-specific thresholds (major, minor, patch) still correctly reject unknown updates since they can't classify the severity.

Regression tests added at both the threshold-function level and the trigger-dispatch level.

This will ship in the next RC (rc.5). Please update and let us know if this resolves the issue for you.

[begunfx]

As of rc.5 I am getting push notifications for new containers post update. In other words I update a container, then I get a push notification for that same container. I wanna say I reported this somewhere else but don't remember where. Thanks!

[s-b-e-n-s-o-n]

Thanks for the follow-up. This should be resolved in rc.6.

The original rc.5 fix covered the threshold=all / updateKind=unknown case, and rc.6 also fixes the follow-up digest-buffer issue that could send an "update available" notification after the container had already been updated. It also uses event-specific templates for update-applied, update-failed, security-alert, agent-disconnect, and agent-reconnect notifications instead of reusing the generic update-available wording.

If you get a chance to try rc.6, let us know whether you still see any post-update push notifications.

[begunfx]

I think this is finally resolved. I got push notifications for all my updates yesterday. Feel free to close this and I'll continue to monitor should the behavior change. Thanks!

[begunfx]

Wait on this. I noticed I did have 1 additional container update available that did NOT have an associated push notification. It occured at 627am this morning.

1
[06:27:38.089]
debug
registry.hub.private
iib0011/omni-tools - Manifest found with [digest=sha256:b901704a39903f10dc0aab65308b779e495fe6f87333feb25abc39002100982b, version=2]

Please reference #282 for my cron schedules to see if that might have any impact.

[s-b-e-n-s-o-n]

Thanks for the follow-up — glad the bulk of notifications are working now.

For the iib0011/omni-tools miss at 6:27am, a couple of questions that'll help narrow it down:

1. Is this a new update, or has the omni-tools tag 2 update been showing in the UI since before you upgraded to RC6? If it carried over from an older RC, the most likely cause is the once deduplication — the first detection happened when the old threshold bug was still blocking notifications, and now subsequent scans see the same result so they don't re-fire. Clearing the store (or deleting/re-creating the container) would reset it.

2. Do you have dd.watch.digest=true set on that container? Docker Hub images have digest watching disabled by default (to avoid rate limiting), so if the tag hasn't changed (2 → 2), the detection path depends on either digest watch being explicitly enabled or the image creation date differing.

3. Which watcher/host is omni-tools on? Want to cross-reference the cron schedules from GMAIL: Batch and Digest Email Issues/Inconsistencies #282 with the 6:27am detection time.

If the answer to #1 is "it's been showing for a while," this isn't a new bug — it's the stale-store interaction with the original threshold fix. A store clear would resolve it. If it's a genuinely new detection that just happened today, we'll need to dig deeper.

[begunfx]

omni-tools was a completely new update.

I did have dd.watch.digest=true explicitly set on that container (I am removing it).

omni-tools is running on DataVault (same host as drydock).

[s-b-e-n-s-o-n]

Thanks for the quick answers — that rules out the stale-store scenario and the digest detection path.

Since this is a genuinely new update with dd.watch.digest=true set, the detection and threshold logic should be passing correctly. With the info we have, I can't pinpoint the silent filter without seeing what the trigger layer decided.

Could you temporarily set your log level to debug (DD_LOG_LEVEL=debug) and wait for the next scan cycle where an update is available but no push notification comes through? The key log lines to look for from your Pushover trigger:

• Threshold not reached => ignore — threshold filter (should not happen with the fix)
• Trigger conditions not met => ignore — the include/exclude filter on the container rejected it
• Run — trigger actually fired (if you see this but no notification, it's a Pushover API issue)
• No log at all for the container — means it was filtered earlier by shouldHandleSimpleContainerReport (the once/changed check) or by the notification rule allow-list

Also worth confirming: on the Notifications settings page in the UI, is the update-available rule enabled with Pushover in its trigger list? (Per #249, this was previously only configured for Pushover on some rules but not others — if update-available has an explicit trigger allow-list that doesn't include your Pushover trigger ID, it would silently skip.)

[begunfx]

I can confirm that the notifications are properly set for mobile AND gmail:

Also, good news, I already have debug log level set. Since there's a lot of testing going on I thought it made sense just to keep that level logging on for now.

I'll try to keep an eye out for the next scan and see if I can capture your callouts.

[s-b-e-n-s-o-n]

Notification rules look correct — both mobile and gmail are on the Update Available allow-list, so that's not the blocker.

Good that debug is already on. When the next miss happens, if you can grab the logs around the scan time for that container, that should tell us exactly where in the chain it stopped. Thanks for staying on it.