fix: recover corrupt distributed records

[yohamta0]

Summary

• make file-backed atomic writes crash-durable by syncing temporary file contents and parent-directory metadata
• add crash-safe exclusive record creation and typed corrupt-record errors
• recover distributed lease and active-run stores from invalid JSON without leaving quarantine files in collection scans
• remove stale corrupt records after the grace period while keeping recent lease corruption fail-closed
• immediately remove and recreate corrupt records when an authoritative upsert is available

Root cause

The file backend renamed temporary files without syncing the file contents or the containing directory. A host power loss could therefore persist the renamed directory entry while losing the file data, leaving zero-byte or otherwise invalid JSON records.

Distributed lease listing treated any unreadable lease as a global failure. A single corrupt lease could make queue-capacity checks fail closed and leave scheduled runs queued indefinitely. Active-run listing skipped unreadable entries but retained them permanently, causing repeated scans and warnings.

Impact

New writes are durable across process or host crashes when the underlying filesystem honors fsync. Existing corrupt distributed records are removed safely after the configured grace period, and fresh coordinator updates replace corrupt records immediately. Cleanup deletes the damaged file instead of accumulating quarantine artifacts, so collection scans do not degrade over time.

Validation

• make fmt
• make lint
• go test ./internal/cmn/fileutil ./internal/persis/file ./internal/persis/store ./internal/service/scheduler ./internal/service/coordinator -count=1
• focused race tests for file persistence and distributed-store recovery
• Windows cross-compilation for the affected file persistence packages

Closes #2375

---

Summary by cubic

Make file-backed writes crash-durable and recover distributed records from invalid JSON. Coalesce lease heartbeats and safely clean up corrupt files so scheduling and capacity checks continue.

• Bug Fixes
  • internal/cmn/fileutil: Sync temp file and parent directory for atomic writes; add WriteFileAtomicExclusive; add durable delete; use write-through on Windows.
  • persis: Add ErrCorrupt and propagate it from the file backend and single-record store.
  • internal/persis/file: Create uses WriteFileAtomicExclusive; Get returns ErrCorrupt for invalid JSON; add RemoveCorrupt with durable deletes.
  • internal/persis/store: On upsert, replace corrupt lease/active-run records; on list, remove stale corrupt files after a grace period; lease listing no longer blocks capacity checks; active-run listing skips and cleans corrupt entries; grace period configurable via WithCorruptRecordGracePeriod.
  • internal/service/coordinator: Coalesce RunHeartbeat lease updates when already fresh; isolate corrupt leases by cancelling only the affected run.

^{Written for commit 605ba9f. Summary will update on new commits.}

Summary by CodeRabbit

• New Features

  • Added configurable handling for corrupted persisted records.
  • Added support for identifying and removing stale corrupted records.
  • Added typed corruption errors for clearer failure reporting.

• Bug Fixes

  • Improved atomic file operations to ensure data and directory changes are durably persisted.
  • Prevented failed exclusive writes from overwriting existing data.
  • Distributed leases and active runs can now recover from corrupted records without blocking scheduling or capacity checks.

[coderabbitai]

📝 Walkthrough

Walkthrough

Durable atomic file operations, typed corruption errors, corrupt-record removal, and configurable recovery are added to file persistence. Distributed lease and active-run stores now repair corrupt records during upserts and remove stale corrupt records during listing.

Changes

Corrupt-record recovery

Layer / File(s)	Summary
Durable file operations internal/cmn/fileutil/*	Atomic writes now sync temporary files and directories; exclusive creation and durable removal are added with Unix and Windows implementations.
Persistence corruption contract internal/persis/errors.go, internal/persis/file/*, internal/persis/store/singlerecord.go	Invalid JSON uses shared persis.ErrCorrupt; file collections support durable stale-aware removal and retain exclusive-create conflict behavior.
Distributed recovery framework internal/persis/store/distributed_corruption.go, internal/persis/store/util.go	Distributed stores accept a corrupt-record grace period and use read-error callbacks to remove or skip corrupt records.
Distributed store recovery flows internal/persis/store/distributed_{lease,active}.go, internal/persis/store/distributed_test.go, internal/service/scheduler/queue_processor_test.go	Lease and active-run upserts repair corrupt records, listings remove stale corrupt records, and tests cover persistence, logging, and scheduler capacity behavior.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Scheduler
  participant LeaseStore
  participant FileCollection
  participant FileUtil
  Scheduler->>LeaseStore: ListAll leases
  LeaseStore->>FileCollection: Read record
  FileCollection-->>LeaseStore: persis.ErrCorrupt
  LeaseStore->>FileCollection: RemoveCorrupt(staleBefore)
  FileCollection->>FileUtil: RemoveFileDurable
  FileUtil-->>FileCollection: Durable removal result
  FileCollection-->>LeaseStore: Removed stale record
  LeaseStore-->>Scheduler: Active lease count

Loading

Possibly related PRs

• dagucloud/dagu#2206 — Refactors the listing utility used by the corrupt-record read-error handling.
• dagucloud/dagu#2219 — Overlaps with the file collection’s atomic exclusive-create path.
• dagucloud/dagu#2229 — Relates to shared corruption error semantics in persistence stores.

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 24.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Linked Issues check	✅ Passed	The changes address #2375 by making writes durable and letting lease/active-run stores recover or remove corrupt records.
Out of Scope Changes check	✅ Passed	The changes are focused on durable file writes, corrupt-record recovery, and tests; no clear unrelated scope was introduced.
Title check	✅ Passed	The title is concise and accurately captures the main change: recovering corrupt distributed records.
Description check	✅ Passed	The description includes summary, root cause, impact, validation, and the related issue, but omits the template's Changes and Checklist sections.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch fix/corrupt-distributed-record-recovery

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}