feat: Avoid cron jobs for daily and weekly emails

[awais-ansari]

As of now, in order to trigger daily and weekly notification emails (digests), instance operators need to configure 2 separate cron jobs. See the operator docs for more details: Enable Notifications — Latest documentation.

Problems
Silent failure: Digest emails rely on a scheduler outside the app (cron). If it’s missing, misconfigured, or changes over time, the platform can still show email preferences even though no digests will be sent.

Retry is clumsy: With cron, retrying a failure means rerunning a big batch job, and we need extra tracking to avoid duplicate emails. With per-user tasks, retries are easier and more targeted.

Note: Retry for email digests does NOT exist in current implementation.

Inaccurate time info: Because the timing is controlled outside the app, the preference center can’t reliably show when a user will actually receive a digest. Any fixed time shown in the UI can easily be wrong (as is the case at present).

Bad fit for global users: Cron runs on server time. That doesn’t match users across timezones, and it makes it hard to later support “send in my timezone” or “send at my preferred time” without hacks.

Proposed solution
Replace cron-triggered digests with celery tasks. When a new notification is generated for a user, we will create or update that user’s next digest task based on their cadence (daily/weekly). For performance reasons, we need to keep, at most, one pending digest task per user per cadence

Pros
No manual configuration: Operators won’t have to configure cron jobs manually.

Reliability: Digest scheduling is owned by the application so cron jobs won’t fail silently.

Optimized delivery: In future, we can improve these tasks to respect user’s timezone or even their specified times/days.

Info correctness: Preference center can show the actual next scheduled send time (and later linked with user’s timezone).

Retries: In future, we can add retries for failed emails comparatively more easily.

Cons
A lot of celery workers will be engaged if a large course creates a course wide notification like course update and if a lot of users have set email cadence to daily/weekly.

Rollout consideration
We need to make sure that instances that are currently using cron jobs don’t start sending duplicate emails once they upgrade and this feature enabled.

[feanil]

@awais-ansari how will the celery tasks be triggered? There has been previous discussion about celery beat but as far as I know we haven't added that to the default build of the platform. Is there going to be some other triggering mechanism?

[AhtishamShahid]

Current State:
Currently, a cron job checks for notifications within a specific time window and sends emails to individual users.

Proposed Change:
We will transition to creating a delayed Celery task for each user at the moment a notification is generated. This task will be scheduled to trigger at the user's configured delivery time.

Workflow:

1. Notification Trigger: A new notification (e.g., for a new comment) is generated.
2. Task Management: A new task is pushed to Celery. If a pending task for that user already exists, a duplicate will not be created.
3. Execution: The task executes according to the user's settings (e.g., on weekends) and sends the email.

Pros & Cons:

• Pro: Removes the need for infrastructure managers to manually configure and maintain cron jobs for this application.
• Con: Significantly increases the volume of tasks stored in the message broker.

Note on Celery Beat:
While we could use Celery Beat, there currently seems to be no consensus on whether we should add it as dependency.