Skip to content

3.4.0

Latest

Choose a tag to compare

@coleifer coleifer released this 04 Sep 02:05

Backwards-incompatible. The django stats app writes to its own sqlite database by default (huey-stats.db in settings.BASE_DIR), instead of DATABASES['default']. To keep recording in your main Django database, specify the database explicitly:

HUEY_STATS = {'database': 'postgresql://user:password@localhost/my_db'}

HUEY_STATS['database'] takes a db-url or a peewee Database, or use
HUEY_STATS['filename'] to set the sqlite path. See the Django docs section
for building a peewee database from your existing DATABASES entry.

The admin Events log is now rendered from the stats database through
peewee, rather than being a ModelAdmin via ORM. It is now linked from the
dashboard's Recent events heading.


Consumer:

  • Add --shutdown-timeout option to bound graceful shutdown time.
  • Add --graceful-signal option to allow making SIGTERM the graceful shutdown signal (e.g. -g TERM).

User-facing APIs:

  • Add ttl to lock_task() and put_if_empty() so locks expire on their own. Supported by memory and redis storages, others raise NotImplementedError. RedisHuey requires server hash-field TTL support (redis 7.4+ or valkey 9+), RedisExpireHuey works on any version.
  • Add timeout parameter to aget_result().

Small stuff:

  • Ensure one raising signal receiver does not prevent the remaining receivers from running.
  • Fix crontab ranges with a step (0-30/5) and wrap-around ranges (22-2, day_of_week='1-7'). Raise ValueError when a field matches no values.
  • Pass failed chord members to the callback as huey.Error and skipped (revoked, expired, cancelled) members as huey.SKIPPED instead of a raw exception or None.
  • Delete chord partial results explicitly on completion, so they no longer linger until expiry on RedisExpireStorage.
  • Write the error of a failed pipeline stage to every skipped downstream stage, so their Result handles raise instead of blocking.
  • Accept an explicit id in TaskWrapper.s(), matching schedule().

Storage:

  • Skip messages that fail to deserialize in Huey.pending() and Huey.scheduled(), logging the error, as read_schedule() already does.
  • Add clean_name option to RedisHuey (default to True). Specify False to stop stripping non-alphanumeric characters from the storage namespace. The default will change in a future release.
  • Reduce Redis round-trips for peek_data, pop_data and delete_data, and make RedisExpireStorage.incr() set the TTL atomically w/ the increment.
  • Skip the exclusive lock in SqliteStorage.dequeue() when the queue is empty, so idle consumers no longer block producers and the scheduler.
  • Batch the SqliteStorage.read_schedule() delete to avoid sqlite's parameter limit when many tasks come due at once. Same fix in SqlHuey.
  • Order SqliteStorage.read_schedule() and SqlHuey.read_schedule() by (timestamp, id), matching PostgresStorage.
  • Add queue to the sqlite task index (task_queue_priority_id) so queues sharing a file no longer scan each other's rows. The old task_priority_id index is dropped when the schema is initialized. I'll get rid of the temporary in-band drop in a few releases.
  • Add peek_many() to the storage API. Revocation checks now fetch the task and task-class keys in one round trip.
  • Return str keys from result_items() on every storage backend. Redis and File storage previously returned bytes.

Stats:

  • Better handling for timeout, locked, rate-limited and retrying in stats recorder.
  • Log dropped events in the stats recorder instead of failing silently.
  • Fix stats recorder max_events pruning to count rows per queue. With multiple queues sharing a stats db, each queue retained only a fraction of the configured cap.
  • Add inflight_hours option to stats recorder, replacing the hardcoded 6 hour in-flight cutoff.
  • Require the change_hueydashboard permission for the django admin dashboard's revoke, restore and flush controls. Previously any staff user who could view the dashboard could also flush the queue.
  • Add huey[stats] extra which installs ol' peewee.

View commits

im-888905ac53