High: Make Codeforces contest sync single-execution safe across multiple backend instances

[coderabbitai]

Summary

Make the Codeforces contest synchronization scheduler safe when the backend runs in more than one process or deployment instance.

This follow-up tracks the High-severity finding from the production review of #269: the scheduler created in server/jobs/contestSync.js runs inside every application process. The current implementation is correct for a single server process, but it has no leader-election or distributed-lock mechanism.

Backlinks

• Source PR: feat: upcoming contests tracker for Codeforces with in-app reminders  #269
• Source review: feat: upcoming contests tracker for Codeforces with in-app reminders  #269 (comment)
• Requested by: @kunalverma2512

Current behavior

On successful database connection, server/server.js calls startContestSyncJob(). That function:

1. performs an initial Codeforces sync; and
2. registers a node-cron task for the top of every hour.

Because this happens in-process, each independently running server instance registers and executes its own schedule. Examples include PM2 cluster mode, container replicas, autoscaling hosts, rolling deployments with overlap, or multiple dynos.

The persistence writes are largely idempotent because contest records are upserted, so this is not expected to corrupt data immediately. However, idempotency does not make duplicated work safe or desirable at production scale.

Why this is needed

A horizontally scalable web application must not assume that a scheduled task is executed by only one process. Without coordination, a deployment with N replicas produces N initial syncs and N hourly syncs at the same time.

Production impact without this change

• External API pressure: every replica calls the public Codeforces API at the same cadence, multiplying outbound traffic and increasing the chance of rate limiting or temporary blocks.
• Database pressure: every replica performs the same bulk upserts and stale-reminder cleanup, producing avoidable write load and lock contention.
• Operational noise: logs report multiple “synced” executions for one intended schedule, complicating incident investigation and monitoring.
• Scaling regression: increasing availability by adding instances unintentionally increases background-work volume instead of keeping it constant.
• Future correctness risk: today’s sync is mostly idempotent; a later non-idempotent side effect (email, push notification, analytics event, cleanup) added to the job could become duplicated immediately.

This is a standard distributed-systems concern: an in-memory scheduler coordinates only one process, not a fleet.

Scope and affected areas

• server/jobs/contestSync.js
• server/server.js
• Potentially a small persistence abstraction/model for job locks or job-run leases
• Potentially configuration/deployment documentation for the selected scheduling strategy
• Automated tests for lock/lease acquisition and release behavior

Required changes

Choose and document one production-appropriate single-execution strategy. Suitable approaches include:

1. Dedicated scheduler/worker deployment

   • Run the sync only from one explicitly configured worker process or managed platform scheduler.
   • Keep web instances free of cron ownership.

2. Database-backed lease / distributed lock

   • Atomically acquire a short-lived lease before starting a sync.
   • Only the lock holder performs the work.
   • Use an expiry/TTL or lease timestamp so a crashed process cannot block future syncs indefinitely.
   • Release or expire the lease safely after the run.

3. Queue-based repeatable job

   • Use a queue system with a documented single scheduler/worker ownership model.

The implementation must avoid a check-then-act race. For example, “read lock document, then insert it” is not sufficient under concurrent startup; lease acquisition must be atomic at the database/queue layer. Preserve the current startup sync and hourly refresh intent, but ensure only one actor executes each logical run.

Also define observable behavior:

• log whether an instance acquired or skipped the lease;
• log failures with enough context to distinguish lock acquisition from Codeforces API/sync failure;
• avoid holding a lock forever if a process crashes;
• ensure a slow sync cannot overlap with another run unintentionally.

Acceptance criteria

• Starting two or more backend instances results in one logical contest sync per scheduled interval.
• The initial sync is similarly protected from concurrent process startup.
• Lock/leader acquisition is atomic and safe against concurrent attempts.
• A crash, timeout, or abandoned lock cannot permanently prevent later syncs.
• A long-running sync cannot cause unintended concurrent work from another instance.
• The approach is documented in code/configuration so deployment owners know which process owns scheduled work.
• Logs/metrics make it possible to tell whether a process ran or intentionally skipped a scheduled sync.
• Tests cover concurrent acquisition, lease expiry/recovery, successful sync, failed sync, and skipped execution.
• The existing single-instance development workflow remains simple and functional.

Suggested validation scenarios

1. Launch two application processes against the same database and verify that only one makes the Codeforces request for the same interval.
2. Simulate a process terminating after acquiring the lock and verify another instance can run after the lease expires.
3. Simulate a Codeforces/API failure and confirm the next scheduled run is not permanently suppressed.
4. Simulate a sync taking longer than the normal interval and verify the chosen policy prevents unsafe overlap.
5. Verify normal one-instance startup still runs exactly one initial synchronization.

Contributor learning guide

Before implementation, review these topics:

• Distributed locks and leader election: why process-local state is insufficient in a replicated deployment.
• Leases versus mutexes: leases expire and recover from crashes; understand clock/TTL implications.
• Atomic compare-and-set operations: MongoDB conditional updates, unique constraints, and why “find then create” races.
• At-least-once execution and idempotency: scheduled jobs may run more than once; design writes and side effects accordingly.
• Job observability: structured logs, run IDs, duration/error metrics, and diagnosing skipped versus failed work.
• Deployment topology: PM2 cluster mode, containers, autoscaling, rolling releases, and managed cron/worker services.
• Failure-mode design: API outage, database outage, process crash, slow work, and overlapping schedules.

Definition of done

The contest sync is safe to deploy across multiple backend replicas without multiplying scheduled work, recovers automatically from failed owners, and is covered by focused automated tests and clear operational documentation.