v0.2.0
Upgrading runs a migration: python manage.py migrate django_ox
Fixed
- A worker whose task had been taken back by the reaper could still write its
own outcome over the row, so a task that had already finished could be moved
back to READY and run a second time after its result had been reported. Every
claim now stamps the row with a lease number, and every finish write carries
that number in its WHERE clause, so a write from a worker that no longer
holds the task matches nothing and is dropped instead of applied. No
completion is signalled for a dropped write. - The reaper no longer records a failure it did not observe. When a lock aged
out with no attempts left it wrote FAILED and invented aTaskAbandoned
exception to explain it, on no evidence beyond a clock. It now records the
task as LOST, which says the worker stopped reporting and the outcome was
never seen, and nothing more. - Lock timestamps are written and compared using the database server's clock
rather than each worker's own, so two hosts with drifting clocks no longer
produce false reclaims. This applies whenUSE_TZis on. WithUSE_TZoff
the worker's clock is used instead, because the database's clock does not
always match what these columns hold: SQLite's is UTC while the columns carry
naive local time, and reading one against the other would makeox_prune --older-thantreat rows that finished seconds ago as hours old. - On databases without
SELECT ... FOR UPDATE SKIP LOCKED, which includes
SQLite, a claim read its row back in a second statement and could come away
holding a lease granted to a different worker, if the reaper reclaimed the
row in the gap between the two. The read is now pinned to the lease the claim
was granted, so a worker that lost the row inside that gap comes back with
nothing rather than with someone else's lease.
Added
- Lease renewal. A worker refreshes the lock on the tasks it is running,
one statement per interval however many are in flight, and keeps doing so
through a graceful drain. A long task on a healthy worker is no longer
reclaimed while it is still running.LOCK_TIMEOUTnow bounds how long a
worker may go unresponsive, not how long a task may take. The renewal
interval isLOCK_TIMEOUT / 3, overridable asrenew_intervalwhen
embeddingWorkerdirectly. OxTask.Status.LOST, a fifth value in django-ox's own status column. It
reads asFAILEDthroughdjango.tasks, which has four statuses and gets no
fifth from us, andis_finishedis true for it, so callers waiting on a
result still terminate. The row keeps the distinction:queue_stats()
reports alostcolumn andox_prune --include-failedcovers it. If the
worker holding a LOST task comes back and records a real outcome, that
outcome replaces LOST; only that one execution can.task_lease_lostandlease_renew_failed, two WARNING log events. Both are
documented on the Monitoring page.
Changed
- A migration ships with this release. Run
python manage.py migrate django_oxwhen you upgrade. It adds thelease_epochcolumn and the new
status choice. task_reclaimednow reportsstatusasREADYorLOST, where it
previously reportedREADYorFAILED.QueueStatshas a fifth field,lost. It is keyword-defaulted, so existing
code that constructs one keeps working.