Severity levels for internal job failure notifications

[itsbrody]

Description

All internal job failures are routed through send_internal_notification() and delivered as a generic "General Notification", regardless of actual severity or user-facing impact. For example, CheckHelperImageJob failing (a background version check with no effect on running deployments, see #10935) triggers the exact same notification treatment as a deployment failure or an unreachable server.

This makes it hard to distinguish "safe to ignore, will retry automatically" from "requires attention now" without opening the dashboard or logs to check job context every time. It also means users receiving these on high-signal channels (Telegram, Pushover, PagerDuty-style setups) can't filter or de-prioritize routine job noise.

Suggested Improvement

Introduce a severity/level distinction for internal notifications, e.g.:

• info / warning / critical (or similar) attached to each send_internal_notification() call
• Jobs with automatic retry and no direct operational impact (e.g. CheckHelperImageJob, other periodic version/metadata checks) tagged as low severity
• Optionally: per-channel or per-severity notification filtering in settings, so users can choose to only receive warning+ on noisy channels while still logging info internally

Related

• [Bug]: CheckHelperImageJob fails with HTTP 403 + HTML page, while direct curl to the same URL succeeds #10935 — concrete example of a low-impact job failure currently notifying at full severity
• [Enhancement]: Add Server or Project Identifier to Notifications (Especially Telegram) #8788 / Discussion [Enhancement]: Add Server or Project Identifier to Notifications (Especially Telegram) #8795 — related notification clarity gap (missing server/project identifier), same underlying theme of notifications lacking context to triage quickly
• Discussion Notifcation Improvements #3163 — general notification improvements thread, no severity-level proposal yet

Happy to help test if useful.