Skip to content

hammerwork 2.1.0

Choose a tag to compare

@CodingAnarchy CodingAnarchy released this 10 Oct 06:07
· 26 commits to master since this release
780417e

Upgrading from 2.0

  • Struct literals. WorkerConfig gained result_cleanup_enabled and
    result_cleanup_interval, and PayloadEncryptionConfig gained key_rotation. Code
    that builds these structs with a full struct literal must add the new fields or end
    with ..Default::default(). Configuration files, from_file, Default and the
    builders are unaffected. (#81
    will make such structs #[non_exhaustive] in 3.0.)
  • Result cleanup is on by default. Worker pools now delete expired job results every
    5 minutes. Expired results were already unreadable through get_job_result, but
    Job::result_data from get_job no longer shows them once cleaned. Set
    result_cleanup_enabled = false or call without_result_cleanup() to opt out.
  • Batch claims are opt-in. with_batch_processing_enabled(true) alone still claims one
    job at a time; use with_batch_size(n) to claim several per poll.

Added

  • Batch claims. DatabaseQueue::dequeue_batch(queue, n) and
    dequeue_batch_leased(queue, weights, lease, n) claim up to n runnable jobs in one
    round trip, with the same guarantees as a single claim: each job's lease is written by
    the claiming statement, attempts is incremented, paused queues, unfinished
    dependencies and disabled recurring jobs are honoured, jobs come back in
    priority DESC, scheduled_at ASC order, and concurrent claimers never get the same job.
    PostgreSQL claims with one UPDATE ... WHERE id IN (SELECT ... LIMIT n FOR UPDATE SKIP LOCKED) RETURNING ...; MySQL with SELECT ... FOR UPDATE SKIP LOCKED LIMIT n and one
    UPDATE in the claim transaction (READ COMMITTED, retried on deadlock); TestQueue
    matches them. Strict priority fills a batch in strict order; weighted priorities pick a
    level by weight per batch, fill from it and top up from other levels. At most
    queue::MAX_DEQUEUE_BATCH_SIZE (1000) jobs per call.
  • DatabaseQueue::release_job_run hands a claimed job that never ran back to
    Pending, taking back the claim's attempt, so it can run elsewhere at once.
  • Batch-mode workers. Worker::with_batch_size(n) makes a worker claim up to n jobs
    per poll (with_batch_processing_enabled(true) alone keeps worker::DEFAULT_BATCH_SIZE
    = 1, so existing workers still claim one job at a time), and
    Worker::with_batch_concurrency(m) runs up to m of them at once (default 1). Each job
    is processed exactly as in single-job mode (timeouts, retries, hooks, events, spawning,
    result storage, encryption, statistics). Jobs waiting for their turn have their leases
    renewed by heartbeats, so the stale job reaper never reclaims them while the worker is
    alive. A throttle's max_concurrent permit is held per claimed job, and a batch claims
    no more jobs than it has rate limit tokens (unused tokens are refunded). On shutdown,
    claimed jobs that have not started are released at once. Workers without batch
    processing claim one job at a time as before. New getters: is_batch_processing_enabled,
    batch_size, batch_concurrency. In a local benchmark, batch claims of 50 were 5-6x
    faster than single claims on PostgreSQL and MySQL.
  • Automatic cleanup of expired job results. WorkerPool now clears expired job
    results (DatabaseQueue::cleanup_expired_results) every 5 minutes in a background task,
    as the README promised. Configure it with WorkerPool::with_result_cleanup(interval) /
    without_result_cleanup(), or with the new [worker] settings
    result_cleanup_enabled (default true) and result_cleanup_interval (default "5m"),
    which WorkerPool::from_config and from_hammerwork_config apply. It is on by default
    because expired results are already unreadable through get_job_result; clearing them
    only frees space (and removes them from Job::result_data). Configuration files without
    the new keys keep loading.
  • Key rotation from configuration. WorkerPool::with_key_rotation(key_manager, interval) rotates KeyManager keys as they fall due and stops on shutdown. The new
    [encryption.key_rotation] section (enabled, default false; master_key_source,
    check_interval_secs, max_key_versions, and the matching
    HAMMERWORK_ENCRYPTION_KEY_ROTATION_* variables) turns it on for pools built with
    from_hammerwork_config, with a KeyManager on the pool's database.
  • Key audit log API. KeyManager::audit_log(&KeyAuditFilter) reads
    hammerwork_key_audit_log, newest first, filtered by key, operation, outcome and time
    range, with limit/offset paging, on PostgreSQL and MySQL. Also new:
    KeyOperation::ALL, parse_key_operation and FromStr/Copy/PartialEq for
    KeyOperation.
  • cargo hammerwork encryption audit shows the key audit log as a table or JSON, with
    --key-id, --operation, --since/--until (RFC 3339 or 30m/24h/7d),
    --success/--failed, --limit and --offset.

Fixed

  • Concurrent key rotation. Rotating the same key from several processes at once could
    fail with a duplicate-key error, and an automatic rotation could rotate a key that another
    process had just rotated. A rotation now writes its version only if the key is still at
    the version it read (and, for automatic rotation, still due), in one transaction; the
    unique (key_id, key_version) index decides between simultaneous writers. Automatic
    rotation skips keys rotated elsewhere, and rotate_key reports "rotated concurrently"
    without writing anything. A key never has two active versions.
  • KeyManager::start_rotation_service runs each pass in its own task, so dropping or
    aborting the service no longer cancels a rotation part-way through its transaction.