Skip to content

fix(worker): prevent scheduled workspace purge from failing on overlapping runs - #931

Merged
viethung0o0 merged 1 commit into
mainfrom
fix/workspace-purge-overlap
Aug 10, 2026
Merged

fix(worker): prevent scheduled workspace purge from failing on overlapping runs#931
viethung0o0 merged 1 commit into
mainfrom
fix/workspace-purge-overlap

Conversation

@viethung0o0

Copy link
Copy Markdown
Contributor

Summary

The hourly workspace-purge cron was surfacing repeated errors in production when it overlapped with another run (multiple worker instances or a run still in progress). The losing run retried for the full lock window and then threw, turning normal contention into a hard failure.

This change makes the cron treat "another run already holds the lock" as a benign skip instead of an error, and speeds up the per-workspace teardown so a single run finishes well within its window.

Changes

  • Skip quietly when another purge already holds the distributed lock, instead of throwing.
  • Add a short lock-acquire retry window so a contended run gives up quickly rather than blocking.
  • Guard against a second local run starting while one is still in progress.
  • Tear down due workspaces with bounded concurrency so a large batch completes faster.

Test plan

  • redis unit tests (distributed-store.exists)
  • business workspace service tests
  • worker purge-workspaces handler tests
  • pnpm lint

Note: the repo-wide type-check hook currently fails on unrelated, pre-existing errors in ui / database / unbuilt imports package; the packages changed here type-check clean.

@github-actions github-actions Bot added the bug Something isn't working as expected label Aug 10, 2026
@viethung0o0
viethung0o0 changed the base branch from develop to main August 10, 2026 03:36
@viethung0o0
viethung0o0 marked this pull request as ready for review August 10, 2026 03:37
@viethung0o0
viethung0o0 merged commit ee93200 into main Aug 10, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working as expected

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants