GMAIL: Batch and Digest Email Issues/Inconsistencies

[begunfx]

Drydock version

1.5.0-rc.6

What happened?

So I posted about this in issue #228 but I think this is better as it's own issue. I had gotten several updates around 5pm ish yesterday. I got all the push notifications, but no emails. I was expecting a batch email at some point.

This is what I saw in the logs:

Then. Around midnight last night I did get one email but it was for a pinned issue that I've had an update for a while:
- Container vaultwarden-DB running with tag 16 can be updated to tag 18
(maybe this is something stuck in the dd.json from previous RCs?)

I added my cron schedules and SMTP config below to see if that would have any impact.

Finally, this morning I did get a digest email at 6:06am (my digest is scheduled for 6am)

- Container drydock-socket-proxy running tag latest has a newer image available

- Container docker-socket-proxy running tag latest has a newer image available

- Container drydock-socket-proxy running tag latest has a newer image available

- Container docker-socket-proxy running tag latest has a newer image available

- Container drydock-socket-proxy running tag latest has a newer image available

- Container docker-socket-proxy running tag latest has a newer image available

- Container drydock-socket-proxy running tag latest has a newer image available

- Container docker-socket-proxy running tag latest has a newer image available

- Container vaultwarden-DB running with tag 16 can be updated to tag 18

What did you expect?

See above

Steps to reproduce

See above

Relevant configuration

Gmail SMTP Config
- DD_NOTIFICATION_SMTP_GMAIL_MODE=batch+digest
- DD_NOTIFICATION_SMTP_GMAIL_DIGESTCRON=0 6 * * *



# --- CRON SCHEDULE ---
- DD_WATCHER_DATAVAULT_CRON=15 */6 * * *
- DD_WATCHER_DATAVAULT_JITTER=900000

# --- CRON SCHEDULE ---
- DD_WATCHER_MEDIAVAULT_CRON=30 */6 * * *
- DD_WATCHER_MEDIAVAULT_JITTER=900000

# --- CRON SCHEDULE ---
- DD_WATCHER_SERVICEVAULT_CRON=0 */6 * * *
- DD_WATCHER_SERVICEVAULT_JITTER=900000

# --- CRON SCHEDULE ---
- DD_WATCHER_SERVICEVAULT2_CRON=0 */6 * * *
- DD_WATCHER_SERVICEVAULT2_JITTER=900000

# --- CRON SCHEDULE ---
- DD_WATCHER_TMVAULT_CRON=45 */6 * * *
- DD_WATCHER_TMVAULT_JITTER=900000

Logs

Installation method

Docker Compose

Docker version

24.0.1

OS / Architecture

Synology DSM 7

[s-b-e-n-s-o-n]

Thanks for the detailed report and logs screenshot — that was the key to tracking this down. I can see three separate things happening here, so I'll address each one:

1. Duplicate entries in the digest email — confirmed bug, fixed

This was the real code bug. Your screenshot shows 8 digest buffer entries across 4 watchers (servicevault, servicevault2, mediavault, tmvault), each seeing drydock-socket-proxy and docker-socket-proxy. The digest buffer correctly tracks these as distinct per-watcher entries — but the default notification template only used container.name without any host/watcher context. So all 4 entries for the same container rendered as identical lines in the email.

Fix (committed on RC7 branch): The default notification templates now append the watcher name for non-local watchers. Your digest email will render as:

- Container drydock-socket-proxy (servicevault) running tag latest has a newer image available
- Container docker-socket-proxy (servicevault) running tag latest has a newer image available
- Container drydock-socket-proxy (servicevault2) running tag latest has a newer image available
...

Local (single-host) setups are unaffected — the suffix is suppressed for local watchers.

2. Midnight email for vaultwarden-DB — expected behavior

In batch+digest mode, the batch handler fires immediately when a watcher cycle finds containers with changes. If one of your watchers ran around midnight and only vaultwarden-DB had a changed result (tag 16 → 18), a single-container batch email is the correct behavior. This is likely the first time that watcher detected the update after the RC upgrade, or the store state was refreshed.

3. Batch emails not arriving at ~5pm — needs more info

This is the one I can't fully confirm from the code alone. Both the batch and digest handlers use the same eligibility check (changed flag + once setting), and your screenshot proves changed=true for those containers (they were buffered for digest). If changed=true for digest, the batch handler should also fire.

The most likely explanation is a transient Gmail SMTP failure — the batch handler catches errors and either suppresses or logs them at warn/debug level, so the email silently failed to send.

Could you check your logs from around 5pm for any warn or error entries from the trigger.smtp.gmail component? That would confirm whether the batch handler fired and SMTP delivery failed, or whether the batch handler itself didn't trigger. If you can share those logs I can narrow it down further.

This will ship with RC7.

[begunfx]

"2. Midnight email for vaultwarden-DB — expected behavior
In batch+digest mode, the batch handler fires immediately when a watcher cycle finds containers with changes. If one of your watchers ran around midnight and only vaultwarden-DB had a changed result (tag 16 → 18), a single-container batch email is the correct behavior. This is likely the first time that watcher detected the update after the RC upgrade, or the store state was refreshed."

I'm not sure this was expected behavior (but I could be wrong). This container has been in this state for weeks now. Same new version was available. I haven't updated it due to it being a pinned container and the db version change impacting the service that uses it. So, then question begs: why would I have gotten a fresh update notification?

[s-b-e-n-s-o-n]

Good catch — you're right to question that. If the same update (tag 16 → 18) has been sitting there for weeks with no change, once=true (default) should have suppressed it. The changed flag only flips to true when resultChanged() detects a difference in the result data (tag, suggestedTag, digest, or created timestamp) compared to what's already in the store.

There are two things that could have caused a fresh notification for a weeks-old update:

1. RC upgrade reset the stored result — When upgrading from RC5 → RC6, if the store (dd.json) was regenerated or the container's result field was wiped during migration, the next watcher scan would see no prior result in the DB → resultChanged() returns true → changed=true → batch fires. Did you clear the store or did the upgrade create a fresh dd.json?

2. Upstream image rebuilt — If the PostgreSQL 18 image was rebuilt upstream with the same tag but a different creation timestamp, resultChanged() would return true because result.created changed. This happens when image maintainers push a rebuild (e.g., a security patch) to the same tag. This would be a legitimate new detection.

If it's scenario 1, that's a one-time thing after the upgrade and won't recur. If it's scenario 2, the notification was actually correct — the update result genuinely changed even though the version numbers look the same.

Do you still have access to your logs from around midnight? Specifically anything from the watcher that owns vaultwarden-DB — it would show whether the scan detected a new created timestamp vs a first-time detection.

[s-b-e-n-s-o-n]

Follow-up: your question about the vaultwarden-DB midnight email led me to find a second bug.

4. Double-send in batch+digest mode — fixed

In batch+digest mode, a watcher cycle fires two independent events:

1. Per-container event → digest handler buffers the container
2. Batch event → batch handler sends immediately

These two paths had no coordination. After the batch handler successfully sent an email, the same containers stayed in the digest buffer and were sent again at the next digest cron flush.

This is exactly what happened with your vaultwarden-DB: the batch handler sent it at midnight, and then the digest flushed it again at 6am.

Fix (also committed on RC7): After a successful batch send, all sent containers are now evicted from the digest buffer. If the batch send fails (e.g., SMTP timeout), the entries are retained so the digest flush acts as fallback delivery.

So RC7 has two fixes for this issue:

1. 727bbb83 — watcher context in notification templates (no more identical-looking lines)
2. 982b4d74 — batch→digest eviction (no more double-send)

The remaining open questions (5pm batch non-delivery and the weeks-old vaultwarden re-notification) still depend on your logs if you have them.

[s-b-e-n-s-o-n]

One more fix for the batch delivery gap (the 5pm missing emails):

5. Batch mode now retries failed sends and audits delivery failures

Previously, when a batch notification send failed (e.g., Gmail SMTP timeout), the error was caught, logged once at warn level (or suppressed entirely if the same error was seen within 15 seconds), and the containers were dropped. No retry, no audit record, no Prometheus counter increment. In pure batch mode (no digest fallback), those notifications were permanently lost.

What changed:

• Retry buffer — failed containers are retained in a batchRetryBuffer and revalidated against current store state on the next watcher cycle. If the container still has an update available, it's retried automatically.
• Audit trail — failed batch sends now write notification-delivery-failed entries to the audit store, visible in the Audit view regardless of log level. So even if you don't catch the warn log, you'll see it in the UI.
• Prometheus counter — batch handler now increments the trigger counter on both success and failure (it was missing entirely before — simple mode and digest mode had it, batch didn't).
• Error suppression fix — the error dedup signature was keyed with undefined instead of the actual container watcher, so identical SMTP errors from different watchers (servicevault, mediavault, etc.) were all suppressed after the first one. Now each watcher's failure is logged independently.

In batch+digest mode, the digest buffer already acts as fallback (from fix #4), so the retry buffer is only active in pure batch mode.

So for RC7, this issue has 5 fixes total:

1. 727bbb83 — watcher context in notification templates
2. 982b4d74 — batch→digest eviction (no double-send)
3. 0594e971 — await event ordering guarantee
4. c31f78eb — batch retry buffer + audit + counter + suppression fix
5. ae5309b3 — notification-delivery-failed visible in Audit UI

[begunfx]

Unfortunately, I don't have the logs from when the Vaultwarden-db container update fired.

[begunfx]

Since the Vaultwarden-db container showed as an update again last night, this time I was able to capture the logs:

1
[06:25:35.759]
debug
registry.hub.private
vaultwarden/server - Manifest found with [digest=sha256:73f9a159204917843875eb12c5ccd9acbcbf8f15ff7f2ce43cf912eef9f97eff, version=2]
2
[06:25:34.948]
debug
registry.hub.private
vaultwarden/server - Calling registry to get docker-content-digest header
3
[06:25:34.948]
debug
registry.hub.private
vaultwarden/server - Manifest found with [digest=sha256:73f9a159204917843875eb12c5ccd9acbcbf8f15ff7f2ce43cf912eef9f97eff, mediaType=application/vnd.oci.image.manifest.v1+json]
4
[06:25:34.947]
debug
registry.hub.private
vaultwarden/server - Filter manifest for [arch=amd64, os=linux, variant=undefined]
5
[06:25:34.947]
debug
registry.hub.private
vaultwarden/server - Manifests media type detected [application/vnd.oci.image.index.v1+json]
6
[06:25:34.947]
debug
registry.hub.private
vaultwarden/server - Manifests found with schemaVersion = 2
7
[06:25:34.947]
debug
registry.hub.private
vaultwarden/server - Found manifests [{"schemaVersion":2,"mediaType":"application/vnd.oci.image.index.v1+json","manifests":[{"mediaType":"application/vnd.oci.image.manifest.v1+json","digest":"sha256:d8b0e22421c9faffb2c89f77f882aac33d7cbed83b14e5f33495741f399e144c","size":1249,"platform":{"architecture":"arm","os":"linux","variant":"v7"}},{"mediaType":"application/vnd.oci.image.manifest.v1+json","digest":"sha256:ce813355f42da1585f9c15f7216be1dd7472d63c4b9c3e1bef4b9496e03b2ff0","size":567,"annotations":{"vnd.docker.reference.digest":"sha256:d8b0e22421c9faffb2c89f77f882aac33d7cbed83b14e5f33495741f399e144c","vnd.docker.reference.type":"attestation-manifest"},"platform":{"architecture":"unknown","os":"unknown"}},{"mediaType":"application/vnd.oci.image.manifest.v1+json","digest":"sha256:73f9a159204917843875eb12c5ccd9acbcbf8f15ff7f2ce43cf912eef9f97eff","size":1249,"platform":{"architecture":"amd64","os":"linux"}},{"mediaType":"application/vnd.oci.image.manifest.v1+json","digest":"sha256:db0e68388d1889b6a40fe739eedf9283b58568525404270e0c60cd557f94d76d","size":567,"annotations":{"vnd.docker.reference.digest":"sha256:73f9a159204917843875eb12c5ccd9acbcbf8f15ff7f2ce43cf912eef9f97eff","vnd.docker.reference.type":"attestation-manifest"},"platform":{"architecture":"unknown","os":"unknown"}},{"mediaType":"application/vnd.oci.image.manifest.v1+json","digest":"sha256:b71e9c8b98634e9d341d3a966191e6f263d5a90bbc89c1f5c379f5919849c7d7","size":1249,"platform":{"architecture":"arm","os":"linux","variant":"v6"}},{"mediaType":"application/vnd.oci.image.manifest.v1+json","digest":"sha256:1f71840f6b44659e4d32f88a4f0ef9a3e5de648ee6ed83ab0532293cf097508e","size":567,"annotations":{"vnd.docker.reference.digest":"sha256:b71e9c8b98634e9d341d3a966191e6f263d5a90bbc89c1f5c379f5919849c7d7","vnd.docker.reference.type":"attestation-manifest"},"platform":{"architecture":"unknown","os":"unknown"}},{"mediaType":"application/vnd.oci.image.manifest.v1+json","digest":"sha256:c2d1ee0812b5b45ebbaefa885c5e4d9478418f09e921a3b75742edd5e0a3f348","size":1249,"platform":{"architecture":"arm64","os":"linux"}},{"mediaType":"application/vnd.oci.image.manifest.v1+json","digest":"sha256:f09b0783561778ab62b37b7da908a5c10776bdbf3ef09eb6c6773cf7a6c4506a","size":567,"annotations":{"vnd.docker.reference.digest":"sha256:c2d1ee0812b5b45ebbaefa885c5e4d9478418f09e921a3b75742edd5e0a3f348","vnd.docker.reference.type":"attestation-manifest"},"platform":{"architecture":"unknown","os":"unknown"}}]}]
8
[06:25:34.681]
debug
registry.hub.private
hub.private - Get vaultwarden/server:sha256:43498a94b22f9563f2a94b53760ab3e710eefc0d0cac2efda4b12b9eb8690664 manifest
9
[06:25:34.680]
debug
registry.hub.private
vaultwarden/server - Manifest found with [digest=sha256:73f9a159204917843875eb12c5ccd9acbcbf8f15ff7f2ce43cf912eef9f97eff, version=2]
10
[06:25:33.830]
debug
registry.hub.private
vaultwarden/server - Calling registry to get docker-content-digest header
11
[06:25:33.830]
debug
registry.hub.private
vaultwarden/server - Manifest found with [digest=sha256:73f9a159204917843875eb12c5ccd9acbcbf8f15ff7f2ce43cf912eef9f97eff, mediaType=application/vnd.oci.image.manifest.v1+json]
12
[06:25:33.830]
debug
registry.hub.private
vaultwarden/server - Filter manifest for [arch=amd64, os=linux, variant=undefined]
13
[06:25:33.830]
debug
registry.hub.private
vaultwarden/server - Manifests media type detected [application/vnd.oci.image.index.v1+json]
14
[06:25:33.830]
debug
registry.hub.private
vaultwarden/server - Manifests found with schemaVersion = 2
15
[06:25:33.830]
debug
registry.hub.private
vaultwarden/server - Found manifests [{"schemaVersion":2,"mediaType":"application/vnd.oci.image.index.v1+json","manifests":[{"mediaType":"application/vnd.oci.image.manifest.v1+json","digest":"sha256:d8b0e22421c9faffb2c89f77f882aac33d7cbed83b14e5f33495741f399e144c","size":1249,"platform":{"architecture":"arm","os":"linux","variant":"v7"}},{"mediaType":"application/vnd.oci.image.manifest.v1+json","digest":"sha256:ce813355f42da1585f9c15f7216be1dd7472d63c4b9c3e1bef4b9496e03b2ff0","size":567,"annotations":{"vnd.docker.reference.digest":"sha256:d8b0e22421c9faffb2c89f77f882aac33d7cbed83b14e5f33495741f399e144c","vnd.docker.reference.type":"attestation-manifest"},"platform":{"architecture":"unknown","os":"unknown"}},{"mediaType":"application/vnd.oci.image.manifest.v1+json","digest":"sha256:73f9a159204917843875eb12c5ccd9acbcbf8f15ff7f2ce43cf912eef9f97eff","size":1249,"platform":{"architecture":"amd64","os":"linux"}},{"mediaType":"application/vnd.oci.image.manifest.v1+json","digest":"sha256:db0e68388d1889b6a40fe739eedf9283b58568525404270e0c60cd557f94d76d","size":567,"annotations":{"vnd.docker.reference.digest":"sha256:73f9a159204917843875eb12c5ccd9acbcbf8f15ff7f2ce43cf912eef9f97eff","vnd.docker.reference.type":"attestation-manifest"},"platform":{"architecture":"unknown","os":"unknown"}},{"mediaType":"application/vnd.oci.image.manifest.v1+json","digest":"sha256:b71e9c8b98634e9d341d3a966191e6f263d5a90bbc89c1f5c379f5919849c7d7","size":1249,"platform":{"architecture":"arm","os":"linux","variant":"v6"}},{"mediaType":"application/vnd.oci.image.manifest.v1+json","digest":"sha256:1f71840f6b44659e4d32f88a4f0ef9a3e5de648ee6ed83ab0532293cf097508e","size":567,"annotations":{"vnd.docker.reference.digest":"sha256:b71e9c8b98634e9d341d3a966191e6f263d5a90bbc89c1f5c379f5919849c7d7","vnd.docker.reference.type":"attestation-manifest"},"platform":{"architecture":"unknown","os":"unknown"}},{"mediaType":"application/vnd.oci.image.manifest.v1+json","digest":"sha256:c2d1ee0812b5b45ebbaefa885c5e4d9478418f09e921a3b75742edd5e0a3f348","size":1249,"platform":{"architecture":"arm64","os":"linux"}},{"mediaType":"application/vnd.oci.image.manifest.v1+json","digest":"sha256:f09b0783561778ab62b37b7da908a5c10776bdbf3ef09eb6c6773cf7a6c4506a","size":567,"annotations":{"vnd.docker.reference.digest":"sha256:c2d1ee0812b5b45ebbaefa885c5e4d9478418f09e921a3b75742edd5e0a3f348","vnd.docker.reference.type":"attestation-manifest"},"platform":{"architecture":"unknown","os":"unknown"}}]}]
16
[06:25:32.193]
debug
registry.hub.private
hub.private - Get vaultwarden/server:latest manifest
17
[06:25:31.560]
debug
registry.hub.private
Get vaultwarden/server tags
18
[00:28:07.224]
debug
registry.hub.private
vaultwarden/server - Manifest found with [digest=sha256:73f9a159204917843875eb12c5ccd9acbcbf8f15ff7f2ce43cf912eef9f97eff, version=2]
19
[00:28:06.490]
debug
registry.hub.private
vaultwarden/server - Calling registry to get docker-content-digest header
20
[00:28:06.490]
debug
registry.hub.private
vaultwarden/server - Manifest found with [digest=sha256:73f9a159204917843875eb12c5ccd9acbcbf8f15ff7f2ce43cf912eef9f97eff, mediaType=application/vnd.oci.image.manifest.v1+json]
21
[00:28:06.490]
debug
registry.hub.private
vaultwarden/server - Filter manifest for [arch=amd64, os=linux, variant=undefined]
22
[00:28:06.490]
debug
registry.hub.private
vaultwarden/server - Manifests media type detected [application/vnd.oci.image.index.v1+json]
23
[00:28:06.490]
debug
registry.hub.private
vaultwarden/server - Manifests found with schemaVersion = 2
24
[00:28:06.490]
debug
registry.hub.private
vaultwarden/server - Found manifests [{"schemaVersion":2,"mediaType":"application/vnd.oci.image.index.v1+json","manifests":[{"mediaType":"application/vnd.oci.image.manifest.v1+json","digest":"sha256:d8b0e22421c9faffb2c89f77f882aac33d7cbed83b14e5f33495741f399e144c","size":1249,"platform":{"architecture":"arm","os":"linux","variant":"v7"}},{"mediaType":"application/vnd.oci.image.manifest.v1+json","digest":"sha256:ce813355f42da1585f9c15f7216be1dd7472d63c4b9c3e1bef4b9496e03b2ff0","size":567,"annotations":{"vnd.docker.reference.digest":"sha256:d8b0e22421c9faffb2c89f77f882aac33d7cbed83b14e5f33495741f399e144c","vnd.docker.reference.type":"attestation-manifest"},"platform":{"architecture":"unknown","os":"unknown"}},{"mediaType":"application/vnd.oci.image.manifest.v1+json","digest":"sha256:73f9a159204917843875eb12c5ccd9acbcbf8f15ff7f2ce43cf912eef9f97eff","size":1249,"platform":{"architecture":"amd64","os":"linux"}},{"mediaType":"application/vnd.oci.image.manifest.v1+json","digest":"sha256:db0e68388d1889b6a40fe739eedf9283b58568525404270e0c60cd557f94d76d","size":567,"annotations":{"vnd.docker.reference.digest":"sha256:73f9a159204917843875eb12c5ccd9acbcbf8f15ff7f2ce43cf912eef9f97eff","vnd.docker.reference.type":"attestation-manifest"},"platform":{"architecture":"unknown","os":"unknown"}},{"mediaType":"application/vnd.oci.image.manifest.v1+json","digest":"sha256:b71e9c8b98634e9d341d3a966191e6f263d5a90bbc89c1f5c379f5919849c7d7","size":1249,"platform":{"architecture":"arm","os":"linux","variant":"v6"}},{"mediaType":"application/vnd.oci.image.manifest.v1+json","digest":"sha256:1f71840f6b44659e4d32f88a4f0ef9a3e5de648ee6ed83ab0532293cf097508e","size":567,"annotations":{"vnd.docker.reference.digest":"sha256:b71e9c8b98634e9d341d3a966191e6f263d5a90bbc89c1f5c379f5919849c7d7","vnd.docker.reference.type":"attestation-manifest"},"platform":{"architecture":"unknown","os":"unknown"}},{"mediaType":"application/vnd.oci.image.manifest.v1+json","digest":"sha256:c2d1ee0812b5b45ebbaefa885c5e4d9478418f09e921a3b75742edd5e0a3f348","size":1249,"platform":{"architecture":"arm64","os":"linux"}},{"mediaType":"application/vnd.oci.image.manifest.v1+json","digest":"sha256:f09b0783561778ab62b37b7da908a5c10776bdbf3ef09eb6c6773cf7a6c4506a","size":567,"annotations":{"vnd.docker.reference.digest":"sha256:c2d1ee0812b5b45ebbaefa885c5e4d9478418f09e921a3b75742edd5e0a3f348","vnd.docker.reference.type":"attestation-manifest"},"platform":{"architecture":"unknown","os":"unknown"}}]}]
25
[00:28:06.161]
debug
registry.hub.private
hub.private - Get vaultwarden/server:sha256:43498a94b22f9563f2a94b53760ab3e710eefc0d0cac2efda4b12b9eb8690664 manifest
26
[00:28:06.160]
debug
registry.hub.private
vaultwarden/server - Manifest found with [digest=sha256:73f9a159204917843875eb12c5ccd9acbcbf8f15ff7f2ce43cf912eef9f97eff, version=2]
27
[00:28:05.171]
debug
registry.hub.private
vaultwarden/server - Calling registry to get docker-content-digest header
28
[00:28:05.171]
debug
registry.hub.private
vaultwarden/server - Manifest found with [digest=sha256:73f9a159204917843875eb12c5ccd9acbcbf8f15ff7f2ce43cf912eef9f97eff, mediaType=application/vnd.oci.image.manifest.v1+json]
29
[00:28:05.171]
debug
registry.hub.private
vaultwarden/server - Filter manifest for [arch=amd64, os=linux, variant=undefined]
30
[00:28:05.171]
debug
registry.hub.private
vaultwarden/server - Manifests media type detected [application/vnd.oci.image.index.v1+json]
31
[00:28:05.171]
debug
registry.hub.private
vaultwarden/server - Manifests found with schemaVersion = 2