Skip to content

feat: harden contest sync lease with safe release, overlap guard, tes… - #1

Closed
ida-jemi wants to merge 2 commits into
fix/contest-sync-distributed-lockfrom
feat/contest-sync-lease-hardening
Closed

feat: harden contest sync lease with safe release, overlap guard, tes…#1
ida-jemi wants to merge 2 commits into
fix/contest-sync-distributed-lockfrom
feat/contest-sync-lease-hardening

Conversation

@ida-jemi

Copy link
Copy Markdown
Owner

📌 Pull Request Summary

🔗 Related Issue

Closes kunalverma2512#277


📝 Description

Changes Made

  • Added ownerId to the SyncLock model so a lease is tagged with exactly which instance/run acquired it, allowing safe, scoped release that can never delete another instance's active lease.
  • Updated server/jobs/contestSync.js to release the lease immediately after each run completes (success or failure) instead of relying solely on TTL expiry, so the next legitimate run doesn't wait unnecessarily.
  • Added an in-process isRunning guard so a single instance cannot start an overlapping sync if a previous run is still in flight when the next scheduled tick fires.
  • Improved logging to include a run ID and owner ID per attempt, clearly distinguishing acquired / skipped / succeeded / failed outcomes for easier debugging in production.
  • Added automated tests (server/jobs/contestSync.test.js, using Node's built-in node:test runner) covering concurrent lock acquisition, DB-unreachable degradation, lease handoff after expiry, and owner-scoped release.
  • Documented the chosen distributed-lease strategy and its deployment implications in server/jobs/README.md.
  • Wired up npm test (node --test) since the repo had no test runner configured yet.

Motivation

This is a follow-up to kunalverma2512#276 covering the full High-severity finding from the review of kunalverma2512#269: an in-process cron scheduler has no leader-election or distributed-lock mechanism, so every backend replica independently runs its own schedule. kunalverma2512#276 introduced a basic TTL-based lock, but this issue requires the harder production-safety guarantees called out in its acceptance criteria, safe lock release scoped to the correct owner, protection against overlapping runs on the same instance, recovery from a crashed lock holder, and test/documentation coverage, none of which kunalverma2512#276 addressed on its own. This PR closes those gaps so the contest sync job is safe to run across any number of replicas without multiplying external API calls, DB writes, or log noise.


🚀 Type of Change

Select all that apply:

  • Bug Fix
  • New Feature
  • Enhancement
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • DevOps / Tooling
  • Other

🧪 Testing

Verification

  • Tested Locally
  • Existing Tests Passed
  • New Tests Added
  • No Testing Required

Test Details

  • Automated: Added 6 unit tests via node --test, covering: successful lock acquisition, concurrent acquirers racing for the same lease (only one wins via duplicate-key handling), graceful false return (no throw) when the DB is unreachable, reacquisition after lease handoff, owner-scoped releaseLock deletion, and releaseLock not throwing on DB failure. All 6 pass.
  • Manual - true concurrency: Temporarily added an artificial delay inside the sync to widen the race window, then started two node server.js instances against the same MongoDB within that window. Instance 1 logged Acquired lease ... starting syncsynced 24 contest(s); Instance 2 logged only Skipped ... another instance holds the lease. with no acquire/sync lines, confirming exclusivity under real concurrency. Delay was removed before committing.
  • Manual - prompt release: Ran two instances sequentially; the second instance acquired a new lease immediately after the first released (not after waiting out the 10-minute TTL), confirming release-on-completion works correctly.
  • Manual - lock document: Verified in MongoDB Atlas that the synclocks collection holds exactly one document for jobName: "contestSync", and that it is removed immediately after a successful run rather than persisting for the full TTL window.

📸 Screenshots / Demo (If Applicable)

Terminal 1 & 2:

image

npm test:

image

✅ Checklist

  • I have read and followed the contribution guidelines.
  • I have self-reviewed my changes.
  • My changes are limited to the scope of this issue.
  • Documentation has been updated where necessary.
  • No unnecessary files or unrelated changes have been included.
  • The related issue has been linked correctly.
  • All applicable testing and validation steps have been completed.

📚 Additional Notes

This PR is based on top of the fix/contest-sync-distributed-lock branch from kunalverma2512#276 rather than main, since it extends that work directly, please merge kunalverma2512#276 first (or review this PR's diff against that branch) to avoid seeing unrelated changes. The chosen approach is a MongoDB-backed lease (Option 2 from the issue) rather than a dedicated worker process or queue, since it requires no new infrastructure; the tradeoffs and a migration path to a dedicated worker are documented in server/jobs/README.md for future reference.

kunalverma2512 and others added 2 commits July 21, 2026 23:55
…distributed-lock

fix: add MongoDB-based distributed lock to contest sync cron job
…ts & docs

- SyncLock now tracks ownerId so releases are scoped to the acquiring
  instance and can never delete another instance's active lease.
- Lease is released immediately after each run instead of relying
  solely on TTL expiry, reducing unnecessary wait time between ticks.
- Added an in-process isRunning guard so a slow sync cannot overlap
  with the next scheduled tick on the same instance.
- Logs now include a run ID and owner ID, and distinguish skipped vs
  failed vs successful runs.
- Added automated tests (node:test) covering concurrent acquisition,
  DB failure, lease handoff, and scoped release.
- Documented the chosen strategy in server/jobs/README.md.

Closes kunalverma2512#277
@ida-jemi
ida-jemi force-pushed the feat/contest-sync-lease-hardening branch from 40f1915 to 1573ec2 Compare July 23, 2026 09:15
@ida-jemi

ida-jemi commented Jul 23, 2026

Copy link
Copy Markdown
Owner Author

Closing, this was a staging PR before kunalverma2512#280 merged upstream. kunalverma2512#277's work is now being opened directly against kunalverma2512/CodeLens:main here: kunalverma2512#284

@ida-jemi ida-jemi closed this Jul 23, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants