hammerwork 2.0.0

A hardening release (#7). It fixes
jobs that could run twice, lost spawned jobs, concurrent migrations crashing, data loss in
CLI backups, plaintext writes to encrypted queues, SQL injection in the CLI and several
dashboard authentication flaws. It also finishes features that were stubs or ignored their
configuration. It is a major version because of the breaking changes listed below.

Upgrading from 1.x

1. Run the migrations before starting 2.0 workers: cargo hammerwork migration run --database-url <url>, or auto_migrate. 2.0 reads new columns on every job query.
   Migrations are idempotent and serialized by a database lock, so several replicas can
   run them at once. On MySQL 8, 2.0 also completes schemas that 1.15's migrations left
   half-applied.
2. Encryption (encryption feature):
   • Upgrade every process that decrypts before any process encrypts with 2.0; older
     versions can't verify the new payload hash.
   • vault:// and azure:// keys must be exactly 32 bytes; padding is no longer applied.
   • AWS and GCP KMS deployments store a data key on their first start.
3. Dashboard (hammerwork-web):
   • It refuses to start with authentication enabled and no password. Configure a
     password hash, or pass --no-auth to run open explicitly.
   • --cors needs at least one --allowed-origin.
   • API clients must send Content-Type: application/json and Content-Length with
     request bodies.
   • The auth feature (bcrypt) is on by default.
4. CLI (cargo-hammerwork):
   • --database-url is -u and the queue flag is -n/--queue everywhere; the old
     spellings still work.
   • backup create writes format 2.0; 1.x backups still restore.
   • workflow create needs --jobs-file, and the archive *-policy commands are gone.
   • Connections time out after 10 s (connect_timeout_secs).
5. API:
   • Settings and methods that never had any effect are removed (see Removed). Old
     configuration files that still contain them keep loading.
   • Signatures changed:
     • DatabaseQueue::heartbeat_job takes the dequeued &Job.
     • JobSpawnExt::with_spawn_config / with_spawning return Result.
     • RetryStrategy::Custom holds an Arc.
     • hammerwork_web::server::app_routes takes the allowed origins.
   • Several config and stats structs gained fields. Build them with
     ..Default::default().
6. Behaviour:
   • WorkerPool::new() no longer autoscales.
   • A spawn failure now fails (and retries) the parent.
   • A disabled recurring job's pending run is held.
   • A recurring job that starts late after an outage runs once and then follows its
     schedule; it no longer runs a second, immediate catch-up.
   • Zero polling_interval / job_timeout values are rejected.
   • WorkerPool::from_hammerwork_config archives and purges on a schedule when
     [archive] enabled = true.

Added

• Live dashboard updates (#71). Connected WebSocket clients now receive JobUpdate messages when jobs are created, started, completed, failed or timed out, and QueueUpdate messages when a queue's counts change, filtered by each client's subscriptions. Nothing published them before, so the dashboard's live updates never fired. The standalone dashboard runs apart from the workers and cannot see their in-process events, so it polls the database while at least one client is connected: every websocket.live_update_interval (default 2 s, zero disables), at most websocket.live_update_max_jobs changed jobs per poll (default 100). Both settings are optional in configuration files. A dashboard embedded in the workers' process can also forward their EventManager events as they happen with WebDashboard::with_event_manager. New hammerwork_web::live module (LiveUpdates, forward_job_events) and JobHistory::database_now / recent_job_changes.
• Automatic archival ([archive] enabled = true used to do nothing at runtime, and purge_archived_after / delete_after were never applied):
  • WorkerPool::with_archival(policy, config, interval) / without_archival() (off by default). Every interval the pool archives the jobs of every queue that the policy makes eligible, batch by batch (reason Automatic, archived by "worker_pool"), then purges archived jobs older than purge_archived_after, and logs the result. The first pass runs at startup. Passes run in their own task, so workers are never blocked; on shutdown no new pass starts and a running pass stops before its next batch. Concurrent pools and processes are safe (rows are claimed with FOR UPDATE SKIP LOCKED; verified on both backends).
  • WorkerPool::from_hammerwork_config enables it when archive.enabled, using archive_after, delete_after, compression_level and the new archive.check_interval (default "1h"; ArchiveConfig::validate rejects zero, and HammerworkConfig::from_file calls it). Files without check_interval still load.
  • JobArchiver::run_scheduled_pass(queue, archived_by, should_stop) (returns ScheduledArchivalPass), JobArchiver::set_default_policy / with_default_policy / default_policy (the policy for queues without their own and for operations over all queues; was always ArchivalPolicy::default()), and JobArchiver::with_config.
  • ArchivalConfig::verify_compression (on by default) is now applied: each compressed payload is decompressed and compared with the original before it is stored, and a mismatch fails the batch with HammerworkError::Archive (the transaction is rolled back).
• Dashboard table maintenance: POST /api/system/maintenance vacuum, reindex and optimize work on PostgreSQL and MySQL (they returned 501). PostgreSQL runs VACUUM (ANALYZE), REINDEX TABLE and ANALYZE respectively; MySQL runs OPTIMIZE TABLE for all three (InnoDB rebuilds the table and its indexes and refreshes statistics). They run on every Hammerwork table that exists, or only on target when it names one; other names are rejected with 400 and never reach SQL. The response lists each statement (and MySQL's result messages), and dry_run lists them without running anything. New JobHistory::table_maintenance, TableMaintenance and HAMMERWORK_TABLES in hammerwork_web::api::history.
• Job::with_scheduled_at(DateTime<Utc>) builder to schedule a job at a specific time (the guides already used it).
• Real data in place of placeholders (part of #41):
  • CLI worker status [--queue] [--jobs]: running jobs per queue with their lease/heartbeat state (active, expired, none), using the database clock.
  • CLI webhook add/list/update/toggle/remove now persist to webhooks.json beside config.toml (HAMMERWORK_WEBHOOKS_FILE overrides; mode 0600) instead of silently discarding the configuration, and webhook test sends a real request (configured method, headers, auth and HMAC signature) and fails when the endpoint is unreachable or returns an error status. Not-found and unconfirmed removals are now errors instead of exit 0.
  • Web GET /api/stats/trends, /api/stats/detailed and /api/queues/{name}: hourly completed/failed counts, average processing time and error patterns (real first_seen, last_seen and affected_queues) are computed from hammerwork_jobs for PostgreSQL and MySQL; time_range is honoured and validated. Hours with no activity are zero-filled; avg_processing_time_ms is null when nothing completed.
  • Web GET /api/queues/{name}/jobs returns the queue's jobs (it returned a stub message), last_job_at / oldest_pending_job are filled in on queue details, and POST /api/queues/{name}/actions clear_completed works (it always failed). The dashboard's "clear queue" button now calls it (it called a nonexistent endpoint).
  • Web maintenance cleanup dry run reports the real number of dead jobs it would delete.
• Delivery that does what it reports (#41, library part). See Fixed for what used to happen.
  • Webhook payload templates: WebhookConfig::payload_template (new builder with_payload_template) is a JSON document whose strings can contain {{event.*}} placeholders (event, event.job_id, event.queue_name, event.error.message, event.payload.<path>, event.metadata.<key>, ...). A string that is exactly one placeholder keeps the value's JSON type; placeholders inside text are interpolated. Rendering works on the parsed JSON, so event data cannot inject JSON. New webhooks::PayloadTemplate (parse, render) and WebhookConfig::validate. See docs/webhooks.md.
  • SMTP email alerts (alerting feature, via lettre with rustls): AlertTarget::Email has a new smtp: Option<SmtpConfig> field (host, port, from, username, password as SecretSource::Environment/Static, SmtpTls::StartTls (default, required) / Implicit / None, timeout). New AlertingConfig::email_via_smtp, AlertingConfig::validate, AlertManager::try_new. SmtpConfig and SmtpTls are re-exported at the crate root. See docs/monitoring.md.
  • StreamRetryPolicy::delay_for_retry(n), StreamStats::last_error, and docs/streaming.md.
• 🔐 Job payloads are encrypted at rest (#11), PostgreSQL and MySQL. Job::with_encryption / with_pii_fields / with_retention_policy used to be stored on the Job and ignored, so "encrypted" payloads were written and processed in plaintext (see Security).
  • New JobQueue::with_encryption(engine) (and encryption_engine()). enqueue, enqueue_batch, enqueue_workflow and enqueue_cron_job encrypt every job that has an encryption config before anything is written and fill in the migration 011 columns (is_encrypted, encryption_key_id, encryption_algorithm, encrypted_payload, encryption_nonce, encryption_tag, encryption_metadata, payload_hash, pii_fields, retention_policy, retention_delete_at, encrypted_at). No new migration is needed.
  • The payload column never holds encrypted data in plaintext: it is {"encrypted": true} when the whole payload is encrypted, or the payload with each PII field's value replaced by "[ENCRYPTED]" (encryption::job_payload). PII field names are paths ("customer.ssn").
  • Workers decrypt the payload just before calling the handler (new JobQueue::decrypt_job); hooks, events and the recorded outcome keep the redacted job. dequeue, get_job and the other reads return jobs as stored, so the web dashboard and CLI show the redacted payload and never need a key.
  • Fails closed: enqueueing a job with an encryption config on a queue without an engine, or with a different algorithm or key id than the engine's, fails with the new HammerworkError::Encryption and writes nothing. A worker that cannot decrypt a job (no engine, unknown key, tampered data, or a build without the encryption feature) fails the run without calling the handler.
  • Archiving and restoring copy the encrypted columns between hammerwork_jobs and hammerwork_jobs_archive without decrypting (also in builds without the encryption feature); get_job of an archived job reports is_encrypted and pii_fields.
  • Retention is enforced by the new DatabaseQueue::purge_expired_encrypted_jobs() (returns EncryptedJobPurge): it deletes finished (Completed, Failed, Dead, TimedOut) encrypted jobs whose retention_delete_at has passed, and expired encrypted jobs in the archive. Pending and running jobs are never deleted. Default implementation returns an error, so custom DatabaseQueue implementations keep compiling.
  • EncryptionEngine::with_decryption_key(key_id, &source) adds a decrypt-only key, so jobs encrypted under a previous key id stay readable after switching keys. New EncryptionEngine::config(), key_id() and algorithm().
  • CLI: cargo hammerwork maintenance purge-encrypted [--dry-run | --confirm]; cargo hammerwork job show prints the key id, algorithm and retention of encrypted jobs.
• 🔐 Encryption follow-ups (#38)
  • Ciphertexts are bound to their job. Job payloads are encrypted with AEAD associated data covering the job id, queue name, key id, algorithm, PII fields and encrypted fields, so swapping encrypted_payload / nonce / tag between rows, moving a row to another queue, or changing its key id or PII fields makes decryption fail (the worker fails the run without calling the handler). encryption_metadata records the new format_version (EncryptionMetadata::FORMAT_JOB_BOUND); payloads without it (written before this change) still decrypt without associated data and are not re-encrypted. Archive and restore keep every bound value. New EncryptionEngine::encrypt_payload_with_associated_data / decrypt_payload_with_associated_data and encryption::job_payload::job_associated_data. No migration.
  • [encryption] configuration section (HammerworkConfig::encryption, PayloadEncryptionConfig): algorithm, key source, key id, compression, default retention, purge interval, encrypted_queues and decrypt-only decryption_keys. JobQueue::from_config (new JobQueue::apply_encryption_config) builds the engine with new_with_pool, so a configured queue encrypts without code. Keys are only referenced (KeySourceRef: env://VAR, aws://, gcp://, vault://, azure://), never written in the file, printed by Debug or saved by save_to_file. Fails closed on unknown fields, invalid key sources or algorithms, unloadable keys, and an enabled section without the encryption feature. HammerworkConfig::from_env reads HAMMERWORK_ENCRYPTION_* and rejects invalid values. The section is optional, so existing files still load.
  • JobQueue::with_encrypted_queues(["payments"]) ("*": all) encrypts every job on those queues, also jobs without with_encryption.
  • Scheduled retention purge: WorkerPool::with_encrypted_job_purge(interval) / without_encrypted_job_purge() (off by default) and WorkerPool::from_hammerwork_config(worker, &config) (uses encryption.purge_interval_secs). Each purge runs in its own task, so shutdown never cancels one mid-transaction; the stale job reaper now runs the same way.
  • TestQueue encryption: TestQueue::with_encryption, with_encrypted_queues, encryption_engine and decrypt_job apply the same sealing, redaction, fail-closed and decryption semantics as the database backends; purge_expired_encrypted_jobs is implemented against the MockClock.
  • Archived encrypted jobs must be restored to be decrypted (documented in docs/encryption.md); decrypt_job on an archived job now says so instead of reporting a missing ciphertext.
• ⚙️ HammerworkConfig is now consumed by the library (#3)
  • JobQueue::<Postgres|MySql>::from_config builds the connection pool from database (URL, pool size, connection timeout), runs migrations when auto_migrate is set, and registers per-queue throttles from rate_limiting
  • Worker::with_config(&WorkerConfig) applies poll interval, job timeout, priority weights and retry strategy
  • Worker::with_hammerwork_config(&HammerworkConfig) also applies the matching rate-limit throttle and alerting config
  • WorkerPool::from_config creates pool_size workers and configures autoscaling from WorkerConfig
  • WebhookManager::from_config and StreamManager::from_config build managers from the webhooks and streaming sections
  • ArchiveConfig::archival_policy / archival_config, RateLimitingConfig::throttle_for, WorkerConfig::autoscale_config, DatabaseConfig::connection_timeout
• 🩺 Job leases and stale job recovery (#19 C2): jobs left Running by a crashed or killed worker are no longer stuck forever
  • Migration 015_add_job_leases adds last_heartbeat_at and lease_expires_at to hammerwork_jobs (PostgreSQL and MySQL). Run cargo hammerwork migration run before upgrading workers.
  • Workers heartbeat running jobs every third of their lease (Worker::with_lease_duration, default 5 minutes)
  • DatabaseQueue::heartbeat_job and DatabaseQueue::requeue_stale_jobs(older_than) (returns StaleJobRecovery). Expired jobs go back to Pending, or to Dead when out of attempts. Safe to run concurrently (FOR UPDATE SKIP LOCKED plus a status = 'Running' guard). Both methods have default implementations, so custom DatabaseQueue implementations keep compiling.
  • WorkerPool runs the reaper every 60 seconds (with_stale_job_reaper, without_stale_job_reaper)
  • CLI: cargo hammerwork job requeue-stale [--older-than-secs N]
• Worker::with_shutdown_grace_period (default 30 seconds) and RetryStrategy::validate
• 🔒 Job lifecycle state machine (#19 C3/H2/H3/H5/H6/M1–M3)
  • DatabaseQueue::finish_job_run(&run, JobOutcome) -> Option<RecordedOutcome>: workers record each run through it. It applies only while the job is still Running the run that was dequeued (same attempts and started_at) and returns None otherwise, so a zombie worker whose job was reclaimed by the stale job reaper, changed by an operator or re-run elsewhere cannot overwrite it. Has a default implementation (non-transactional, composed from the manual methods), so custom DatabaseQueue implementations keep compiling.
  • New types JobOutcome, RecordedOutcome and JobTransition (re-exported at the crate root) and module queue::lifecycle; new error HammerworkError::InvalidJobTransition.
  • Migration 018_add_job_retry_strategy adds hammerwork_jobs.retry_strategy (JSON) and an index on (batch_id, status). Required: every job query reads the new column, so run cargo hammerwork migration run before upgrading.

Changed

• TestQueue reschedules recurring jobs in place (#71), like PostgreSQL and MySQL: after a run (finish_job_run) and on reschedule_cron_job the same job (same id) goes back to Pending for its next run, instead of a new job being created for it. reschedule_cron_job now checks the job's status like the database backends (not from Completed or Archived), get_due_cron_jobs lists pending recurring jobs whose time has come (it listed Completed ones), and completing a recurring job by hand with complete_job no longer computes a next run. Tests that counted the extra job need updating.
• hammerwork_web::config::WebSocketConfig has two new public fields, live_update_interval and live_update_max_jobs; code that builds it with a struct literal needs ..Default::default().
• Upgrade notes for #64 (see Security):
  • vault:// and azure:// keys must be exactly 32 bytes (base64 in the Vault key field). A source that relied on padding or passphrase hashing now fails to load; store a real 32-byte key. Jobs encrypted under the old derived key cannot be decrypted with the new one, so drain or re-enqueue them before switching.
  • cargo hammerwork backup create writes format 2.0: every column, with priority as its numeric level. backup restore still reads 1.0 files.
  • cargo hammerwork job enqueue, batch enqueue, cron create and workflow create fail with an error when they would store a plaintext job on a queue that holds encrypted jobs, or when the configured encryption key cannot be loaded; hammerwork-web does not start in the latter case.
  • ExternalKmsConfig, WebhookConfig, WebhookAuth, AlertTarget, StreamBackend, DatabaseConfig, cargo_hammerwork::config::Config, hammerwork_web::DashboardConfig and AuthConfig implement Debug by hand (secrets redacted) instead of deriving it. cargo_hammerwork::config::Config has a new encryption_config field.
• WorkerPool::new() no longer autoscales (#64 H9). Autoscaling is enabled only by with_autoscaling(...) or WorkerPool::from_config with worker.autoscaling_enabled. AutoscaleConfig::default() and the presets are still enabled, so with_autoscaling(AutoscaleConfig::default()) keeps working.
• Alerts are checked by the worker's monitoring task, not its polling loop (#64 H10): queue depth, worker starvation, error rate and processing time alerts are evaluated every Worker::with_monitoring_interval (new, default 30 seconds) instead of on every poll.
• The rate limiter only spends a token when a job is claimed (#64 M6). New RateLimiter::refund and TokenBucket::refund.
• Zero durations (#64 M5): HammerworkConfig::from_file / from_env and WorkerPool::from_hammerwork_config reject worker.polling_interval = 0 and worker.job_timeout = 0 (new WorkerConfig::validate). Worker::with_poll_interval waits at least MIN_POLL_INTERVAL (10ms) between idle polls, and with_default_timeout(Duration::ZERO) means no default timeout.
• DatabaseQueue::heartbeat_job takes the run (&Job as dequeued) instead of a job id, and RecordedOutcome has a new spawned field.
• CLI cron enable / cron disable use the library's enable_recurring_job / disable_recurring_job (see Fixed).
• CLI flag consistency (#61); old spellings keep working as aliases:
  • queue commands take -u/--database-url like every other group (-d remains as a hidden alias).
  • The queue filter is -n/--queue everywhere (job purge, spawn list/stats/pending/monitor, archive run/list/stats); -Q and archive --queue-name still work.
  • job retry and job cancel accept the job id positionally (job retry <JOB_ID>); --job-id still works, and passing both is an error.
  • archive run --no-compress disables compression (--compress false is still accepted); batch enqueue --progress-every is an alias of --batch-size.
  • Help text fixes: job retry also retries timed-out jobs, batch enqueue --queue is required, --batch-size only controls progress output, batch retry --max-attempts-reached is a flag.
• Migrations run exactly as written. Both runners used to re-serialize each migration through sqlparser (PostgreSQL also patched the output to restore EXECUTE FUNCTION f()), and fell back to naive splitting with an ERROR log when the parser couldn't handle a statement such as a DO $$ block. They now share a small splitter that only finds top-level ; terminators, skipping string literals, quoted identifiers, comments and PostgreSQL dollar-quoted bodies, and executes the original text. The sqlparser dependency is removed. The resulting schema is identical on both backends.
• Web POST /api/queues/{name}/actions clear_dead now only deletes the named queue's dead jobs older than 7 days; it previously purged dead jobs from every queue while reporting it had cleared one. /api/system/metrics custom_metrics_count and performance_metrics.{database_response_time_ms, active_workers, worker_utilization} are now nullable, and hourly_trends[].avg_processing_time_ms is null (not a global average) for hours without completions. Dashboard GET /api/stats/detailed reports the real uptime.
• Breaking (alerting): AlertTarget::Email has a new smtp field, so code that constructs or exhaustively matches the variant must add it (smtp: None / ..). Configuration files without it still parse, but from_file then rejects the target. AlertingConfig::email(recipient) is deprecated because the target it creates cannot send; use email_via_smtp.
• Breaking: StreamStats has a new last_error field (defaults to None when deserializing); struct literals must set it.
• Behaviour: StreamRetryPolicy::max_attempts is the total number of delivery attempts per event, including the first one (it was never applied before).
• MSRV is now 1.91.1 (first raised to 1.88 for the Azure SDK 1.x, then to 1.91.1 because every AWS SDK release containing the fix for GHSA-8ffr-xgwf-xj56 in aws-smithy-json requires it). Was previously declared as 1.86, but dependencies already needed a newer compiler.
  The workspace uses resolver 3, so the committed Cargo.lock is resolved for 1.91.1, and CI builds every crate and target on 1.91.1.
• Cargo.lock is committed for reproducible builds.
• Breaking: RetryStrategy::Custom now holds an Arc<dyn Fn(u32) -> Duration + Send + Sync> (retry::CustomRetryFn) instead of a Box, so it can be cloned. Code using RetryStrategy::custom(..) is unaffected; code constructing the variant directly must switch Box::new to Arc::new. Two Custom strategies now compare equal when they share the same function.
• WorkerPool::shutdown now waits until every worker has stopped (bounded by each worker's shutdown grace period) instead of returning as soon as the signal is sent
• CI: replaced the disabled Integration Tests workflow with .github/workflows/ci.yml: rustfmt, clippy (--all-targets --all-features -D warnings), unit tests, PostgreSQL 16 and MySQL 8 integration jobs, and a cargo audit job. Runs on pushes and pull requests to master and on demand (#7).
• Behaviour (#19): manual status transitions are guarded. complete_job, fail_job, retry_job, mark_job_dead, mark_job_timed_out, reschedule_cron_job and retry_dead_job only apply from the statuses in JobTransition::allowed_from (e.g. Completed is final, Dead is only left through retry_dead_job, timeouts only apply to Running jobs) and return HammerworkError::InvalidJobTransition otherwise, or JobNotFound for a missing job. They used to update any row by id and return Ok even when the job did not exist. retry_dead_job now also re-runs TimedOut jobs.
• Behaviour (#19): complete_job, fail_job, mark_job_dead and mark_job_timed_out apply the job's side effects in the same transaction: dependents are resolved or failed, the workflow's FailurePolicy and the batch's PartialFailureMode are applied, and workflow and batch progress is updated.
• Behaviour (#19 H3): the job's max_attempts is the retry limit. Worker::with_max_retries(n) now only caps it (min(max_attempts, n)) and workers have no cap by default; it used to replace max_attempts (default 3). Workers no longer leave jobs in a terminal Failed state: the last failed attempt makes the job Dead.
• Behaviour (#19 M1): a timed-out run is retried while the job has attempts left (the on_job_timeout hook and TimedOut event fire for every timeout, followed by the retry hook/event); TimedOut is only terminal on the last attempt.
• Behaviour (#19 M2): enqueueing a job whose retry strategy is RetryStrategy::Custom fails with InvalidJobPayload, since a closure cannot be stored; it used to be dropped silently. Set custom strategies with Worker::with_default_retry_strategy.
• hammerwork-web: the retry job action re-runs Dead jobs through retry_dead_job (which resets attempts) and other jobs through retry_job; non-retryable statuses (e.g. Completed) return an error.
• AWS KMS clients (aws-kms feature) now load config with BehaviorVersion::latest() instead of the deprecated v2025_01_17, matching the Kinesis client. This picks up the SDK's newer defaults, including HTTP(S) proxy settings from the environment.

Removed

• CLI and dashboard placeholders that printed or returned made-up data (part of #41):
  • cargo hammerwork worker start, worker list and worker stop: there is no worker registry, so they could only print simulated output. Use worker status (new, below) and job requeue-stale.
  • cargo hammerwork monitor logs: it printed simulated log lines; no log storage exists.
  • cargo hammerwork webhook stats: delivery statistics only exist in the memory of the process running the WebhookManager.
  • The whole cargo hammerwork streaming command: it never read or wrote configuration, and its test, stats and health subcommands printed success without contacting anything. Configure streams in code with StreamManager.
  • The web dashboard's GET /api/spawn/info placeholder route, which advertised spawn endpoints that never existed (also removed from the web README and docs/job-spawning.md). Spawn trees remain available through cargo hammerwork spawn.
  • Made-up dashboard values: mock recent_operations in GET /api/archive/stats, estimated database_response_time_ms, and active_workers / worker_utilization in GET /api/stats/detailed (running jobs are not workers). These are now measured or null.
• Unused ArchiveConfig fields archive_directory, max_file_size_bytes and include_payloads. Existing TOML files containing them still load.
• Breaking: settings and APIs that never did anything (they were briefly deprecated during the #64 hardening and are removed for 2.0.0). Configuration files that still contain the removed keys keep loading; the keys are ignored.
  • PriorityWeights::with_fairness_factor / fairness_factor (and the fairness_factor key). Weighted selection already gives every priority with runnable jobs a share proportional to its weight, so nothing starves; raise a weight instead.
  • AutoscaleConfig::idle_timeout / with_idle_timeout. Scale-down follows evaluation_window and cooldown_period.
  • AlertingConfig::custom_thresholds / with_custom_threshold (and the [alerting.custom_thresholds] table). No alert read them.
  • StatsConfig::cleanup_interval_secs. Old events are pruned on every insert.
  • DatabaseConfig::create_tables (and the create_tables key). Tables are created by migrations (auto_migrate).
  • MetricsConfig::registry_name (and the registry_name key). The Prometheus registry never used it.
  • encryption: EncryptionEngine::set_key_manager. The engine never used the key manager.
  • hammerwork-web: DashboardConfig::request_timeout (and the request_timeout key; the dashboard applies no timeout, put one on a reverse proxy) and the unused api::archive::PolicyRequest / PolicyResponse types.
  • cargo-hammerwork: the unused public commands::spawn::SpawnOperation type.
• Breaking: ArchivalConfig::max_payload_size and ArchivalConfig::with_max_payload_size. Nothing ever read the limit, and skipping or truncating oversized payloads would leave jobs that can never be archived or restored intact. Serialized configs containing the field still deserialize.

Security

• Encryption and data-safety hardening (part of #64: C3, H1, H2, H4, M14, M15, M17):
  • C3 — cargo hammerwork backup destroyed encrypted payloads. backup create exported 13 columns, so an encrypted job was backed up as its redacted placeholder and restored with is_encrypted = false and no ciphertext (the payload was unrecoverable); cron schedules, recurrence, timeouts, retry strategies, dependencies, workflows, batches, results and tracing ids were dropped too. JSON backups (format 2.0) now hold every column of hammerwork_jobs, read from information_schema so columns added by future migrations are included (a column type the backup cannot copy is an error), with one encoding for both backends (bytes as base64, timestamps as RFC 3339 with microseconds, arrays as JSON arrays). backup restore writes every column back, checks the whole file first and inserts in one transaction; existing ids are skipped, never overwritten or duplicated (MySQL no longer uses INSERT IGNORE, which turned conversion errors into warnings); a non-null value for a column the database lacks is an error. Format 1.0 backups still restore. CSV stays an export and cannot be restored. New cargo_hammerwork::utils::job_rows.
  • H1 — the CLI and the web dashboard wrote plaintext to encrypted queues. They built their queue without the application's [encryption] settings. Commands that write jobs (job enqueue, batch enqueue, cron create, workflow create) and POST /api/jobs now load the application's settings (new PayloadEncryptionConfig::load: the [encryption] section of a hammerwork.toml plus HAMMERWORK_ENCRYPTION_*; CLI config set encryption_config <path> / HAMMERWORK_ENCRYPTION_CONFIG, dashboard HAMMERWORK_ENCRYPTION_CONFIG) and encrypt like the application. If the key cannot be loaded the CLI refuses to write and the dashboard refuses to start. Without settings, both refuse to store a plaintext job on a queue that already holds encrypted jobs (new JobQueue::with_plaintext_guard, applied to enqueue, enqueue_batch, enqueue_workflow and enqueue_cron_job). New job enqueue --encrypt / --pii-field. cargo-hammerwork and hammerwork-web now enable hammerwork's encryption feature; cargo-hammerwork gains aws-kms, gcp-kms, vault-kms and azure-kv features for KMS key sources.
  • H2 — get_dead_jobs / get_dead_jobs_by_queue returned encrypted jobs as plaintext. They selected 15 columns and hard-coded is_encrypted = false, so decrypt_job returned the redacted payload without error. They now return the full job (ciphertext, key id, PII fields, cron, batch, workflow, tracing and result columns) on both backends; a test covers every read path that builds a Job.
  • H4 — after a KMS data-key rotation, other processes sent new jobs to Dead. An engine only knew the key versions it loaded at start. EncryptionEngine::new_with_pool engines now keep the pool and, when a payload does not decrypt with the keys they hold, load the stored versions they lack (and other key ids stored in the table) and retry; they then encrypt with the newest active version. New EncryptionEngine::reload_kms_keys(). A failed reload leaves the run failed and retried as before.
  • M14 — secrets in Debug output, logs and config show. Debug no longer prints the database URL password (DatabaseConfig, CLI Config, web DashboardConfig), webhook signing secrets, header values, auth tokens and passwords and URL paths (WebhookConfig, WebhookAuth, AlertTarget), streaming credentials and config values (StreamBackend), KMS credentials (ExternalKmsConfig, the Azure client secret) or the dashboard password hash (AuthConfig). Adding a webhook logs only the scheme and host of its URL. cargo hammerwork config show masks the database password. New hammerwork::config::redact_url and redact_url_path.
  • M15 — weak Vault and Azure keys were padded. A Vault key that was not base64 of exactly 32 bytes was hashed or padded into a key (a 1-byte value gave a 32-byte key with 8 bits of entropy), and short Azure key material was padded. Both are now rejected with EncryptionError::InvalidConfiguration, for engine data keys and KeyManager master keys.
  • M17 — key material is zeroized. Engine keys (now shared, not copied per operation), KMS data keys, KeyManager master, root and cached keys, and keys read from the environment, files, Vault and Azure are held in zeroize::Zeroizing and wiped when dropped (new dependency zeroize, part of the encryption feature).
• SQL injection in cargo-hammerwork (part of #7): cron list/cron next, monitor dashboard/metrics, backup create, queue stats/health, spawn list/stats/pending spliced the --queue name into the SQL text inside quotes, so a queue name such as x' OR '1'='1 changed the statement. Every value in these statements (queue names, hour/day windows, LIMIT, cutoff timestamps, status names) is now a bound parameter with the placeholder style of the backend ($n PostgreSQL, ? MySQL), built by the new cargo_hammerwork::utils::sql::SqlParams; job list's --last-hours/--limit, job purge --older-than-days, maintenance vacuum cutoffs and the bulk selectors behind batch retry/cancel and maintenance cleanup/check (JobSelector, whose free-form extra SQL fragments were replaced by typed, bound filters) no longer format numbers or timestamps into the SQL either. hammerwork-web was audited and already binds everything. Each fixed command has database-backed tests with a queue name full of quotes and SQL metacharacters on both backends, and tests/sql_query_tests.rs now exercises the real query builders instead of copies of their SQL.
• Job payload encryption was never applied (#11): with the encryption feature, jobs created with with_encryption / with_pii_fields were stored and processed in plaintext and the encryption columns stayed empty. They are now encrypted at rest (see Added). Jobs enqueued by earlier versions remain in plaintext.
• The payload integrity hash stored next to an encrypted payload is now an HMAC-SHA256 keyed by the data key (hmac-sha256:<hex>) instead of a plain SHA-256 of the plaintext, which could be reversed by brute force for low-entropy values such as SSNs or card numbers. Payloads with the old hash still decrypt.
• Decrypting a payload with a malformed nonce, tag or key returns an error instead of panicking.
• aws:// and gcp:// keys are stored only in KMS-encrypted form (envelope encryption) and are stable across restarts; see Fixed (#26). Plaintext data keys are never written to the database.
• External key loading fails closed (#16). With the encryption feature, the AWS KMS, GCP KMS, HashiCorp Vault and Azure Key Vault loaders in KeyManager (master key) and EncryptionEngine (data key) used to log a failure and continue with a key derived from the public source string (key ID, region, Vault path or address, vault URL). The same happened when the source's cargo feature was not enabled. They now return an error, and KeyManager::new / EncryptionEngine::new fail:
  • KMS or vault unreachable, credentials missing or rejected, secret or key field missing: EncryptionError::KeyManagement
  • malformed source (Vault path without a mount, GCP resource not projects/<p>/locations/<l>/..., azure:// without a vault host) or the source's feature (aws-kms, gcp-kms, vault-kms, azure-kv) not enabled: EncryptionError::InvalidConfiguration
  • Vault no longer defaults to https://vault.example.com; set addr= or VAULT_ADDR. A Vault secret without a string key field is an error instead of a key hashed from the path.
  • There is no opt-in fallback. For development without a KMS use KeySource::Static (base64 key) or KeySource::Generated (random, in memory).
• KeyManager never writes plaintext key material or the configured master key to the database: keys are encrypted with AES-256-GCM under the master key, and key_source stores only a label. Previously KeySource::Static(key) would have been stored with the key in key_source (#9).
• cargo audit is clean and the CI security audit job is now blocking (#7). Dependency upgrades that clear the open advisories:
  • azure-kv: moved from azure_security_keyvault / azure_identity / azure_core 0.20 to the 1.x Azure SDK (azure_security_keyvault_keys, azure_identity, azure_core). Fixes RUSTSEC-2026-0275 (legacy azure_core logged the authorization header) and drops http-types (RUSTSEC-2026-0174), rand 0.7 (RUSTSEC-2026-0097), instant and paste (unmaintained).
  • metrics / hammerwork-web: warp 0.3 → 0.4 (hyper 1.x, h2 0.4).
  • aws-kms / kinesis: aws-sdk-kms and aws-sdk-kinesis no longer enable the legacy rustls feature (hyper 0.14 + rustls 0.21); they use the SDK's default-https-client.
  • gcp-kms / google-pubsub: google-cloud-kms 0.6, google-cloud-auth 0.4, google-cloud-pubsub 0.25 and google-cloud-googleapis 0.13 replaced by the same author's renamed, maintained gcloud-kms, gcloud-auth, gcloud-pubsub and gcloud-googleapis 1.x (tonic 0.14). Drops ring 0.16 (RUSTSEC-2025-0009), rustls-webpki 0.101 (RUSTSEC-2026-0104 / -0098 / -0099) and rustls-pemfile (unmaintained).
  • tracing: opentelemetry / opentelemetry_sdk / opentelemetry-otlp 0.22/0.15 → 0.33 and tracing-opentelemetry 0.23 → 0.34 (drops tonic 0.11 / hyper 0.14).
  • Together these remove h2 0.3 (RUSTSEC-2026-0258) and hyper 0.14 from the dependency tree.
  • cargo-hammerwork: indicatif 0.17 → 0.18 (drops unmaintained number_prefix).
• RUSTSEC-2023-0071 (rsa Marvin attack, no fixed release) is ignored in .cargo/audit.toml: it comes only from sqlx-mysql, which uses RSA public-key encryption of the password during caching_sha2_password auth; the vulnerable private-key decryption path is never used.
• Web dashboard hardening (part of #64: H5, H6, H7, H8, M11, M12, M13, M16):
  • H5: the dashboard compared the password to password_hash as plain text unless built with the auth feature, so the documented bcrypt hash was the password. auth (bcrypt) is now a default feature, the hash is only ever verified with bcrypt, and a build without auth refuses to start with authentication enabled (DashboardConfig::validate).
  • H6: the lockout never ended for an account that kept receiving bad attempts, five bad requests locked the administrator out for everyone, and the failure table grew with every made-up username. Failures are now counted per client address (the connection's peer address, not a header) and username, a lockout ends after lockout_duration and the count starts over, made-up usernames share one record per address, and at most 10,000 clients are tracked.
  • H7: bcrypt ran on the async worker threads for every request. It now runs on the blocking pool (at most one verification per CPU), and a successful login is remembered for 60 seconds (or session_timeout, if shorter) under a keyed hash, so polling does not pay for bcrypt each time.
  • H8: any web page could make the browser send state-changing requests (enqueue, bulk delete, purge, ...) to the dashboard. Requests with a body now need Content-Type: application/json (415), and POST/PUT/DELETE requests and WebSocket handshakes that a browser sends from another origin are refused (403), judged by Sec-Fetch-Site or Origin against Host. Clients that send neither header (curl, scripts) are unaffected. New allowed_origins setting / --allowed-origin flag for other trusted origins, and the new hammerwork_web::security module.
  • M11: JSON bodies are limited to 1 MiB (413; a Content-Length header is required, 411), a bulk job action to 1,000 IDs, and /api/archive/jobs filters, counts and pages in the database (new JobHistory::archived_jobs) instead of reading the whole archive. The purge dry run counts in the database too.
  • M12: the WebSocket settings were ignored. max_message_size now limits incoming messages and frames, each connection queues at most message_buffer_size outgoing messages (messages for a client that does not read are dropped), Subscribe accepts only the known event types, and the handshake checks the origin.
  • M13: limit=5000&page=2 skipped 4,000 rows because the offset used the unclamped limit, and (page - 1) * limit could overflow. The offset now uses the clamped limit (1 to 1,000) with saturating arithmetic.
  • M16: usernames are compared in constant time (keyed hash, subtle), and bcrypt runs whether or not the username matched. CORS (enable_cors / --cors) is only granted to the origins in allowed_origins instead of any origin; enabling it without any is a startup error.
  • Behaviour changes: --cors / enable_cors = true now needs at least one --allowed-origin / allowed_origins entry. API clients must send Content-Type: application/json and Content-Length with request bodies. server::app_routes takes the allowed origins as a new last argument, and api::archive::archive_routes needs a JobHistory queue.

Changed (breaking, feature-gated)

• encryption: EncryptionEngine is no longer generic over the database (#11). Its encryption methods used to require u64: Encode<DB>, so they only worked with MySQL; the engine now works with PostgreSQL, MySQL or no database. new_with_pool and rotate_kms_key are generic over the pool's database instead. Code naming the type with a parameter (EncryptionEngine<sqlx::MySql>, EncryptionEngine::<sqlx::MySql>::new) must drop it; EncryptionEngine::new(config) is unchanged.
• encryption: encrypt_payload, encrypt_payload_with_retention, decrypt_payload, cleanup_expired_data and rotate_kms_key take &self instead of &mut self (the engine is shared by queues through an Arc). Existing calls compile; let mut engine bindings now trigger the unused_mut lint.
• encryption: payloads encrypted by this version carry the keyed payload hash (see Security), which earlier versions cannot verify, so they cannot decrypt them. Upgrade every process that decrypts before any process starts encrypting.
• encryption: KeyManager's methods now require DB: KeyManagerBackend (implemented for sqlx::Postgres and sqlx::MySql) instead of the previous generic sqlx bounds. Code using KeyManager<Postgres> or KeyManager<MySql> is unaffected.
• encryption: external key sources fail instead of falling back to a derived key (see Security). Deployments that ran on the fallback key without noticing will fail to start; data encrypted under a fallback key can only be read by recreating that key.
• azure-kv: the Azure SDK 1.x has no DefaultAzureCredential. Hammerwork now picks a credential from the environment: ClientSecretCredential when AZURE_TENANT_ID / AZURE_CLIENT_ID / AZURE_CLIENT_SECRET are set, WorkloadIdentityCredential when AZURE_FEDERATED_TOKEN_FILE is set, otherwise managed identity followed by the Azure CLI / Azure Developer CLI. The optional dependency (and implicit feature) azure_security_keyvault is renamed azure_security_keyvault_keys; enable azure-kv rather than the dependency name.
• tracing: shutdown_tracing() now flushes and shuts down the provider installed by init_tracing() (OpenTelemetry 0.33 removed the global shutdown hook). Code that uses the opentelemetry crates directly alongside Hammerwork must move to 0.33.

Fixed

• The dashboard's documented configuration file did not load. hammerwork-web/README.md writes durations as strings (ping_interval = "30s", lockout_duration = "15m"), but DashboardConfig only read serde's { secs, nanos } tables, so the example failed to parse. Dashboard durations now use the new public hammerwork::config::serde_duration format (the one HammerworkConfig already used), which reads "500ms"/"30s"/"5m"/"1h"/"2d", bare seconds, and the old table form, and writes the readable strings. A test loads the README example.
• Sub-second and huge job timeouts were stored wrongly on PostgreSQL and MySQL. timeout_seconds held as_secs() as i32, so with_timeout(Duration::from_millis(200)) was stored as 0 and the job timed out immediately, and a timeout above i32::MAX seconds wrapped to a negative value that every later read of the row rejected. Fractional timeouts are now rounded up to whole seconds, huge ones saturate at i32::MAX seconds, and a zero job timeout (stored or in memory) means no job-level timeout, so the worker's default applies.
• One undecodable row no longer breaks a whole listing (#71). A leftover row with, say, a negative timeout_seconds made the dashboard's job list, job search, queue jobs and archive list answer 500, and the library's listing methods fail. get_ready_jobs, get_dead_jobs, get_dead_jobs_by_queue, get_recurring_jobs and list_archived_jobs on PostgreSQL and MySQL, and the dashboard's archive listing, now skip such rows and log a warning with the job id. Reading the job itself (get_job, GET /api/jobs/{id}) still returns the error.
• Archiving and restoring retry deadlocks (#71). archive_jobs and restore_archived_job move rows between hammerwork_jobs and hammerwork_jobs_archive under row locks, and a concurrent archive, restore or job transition could abort them as a deadlock victim (MySQL error 1213; PostgreSQL 40P01). Both now run again, like job transitions do, on both backends.
• Dashboard WebSocket messages: the dashboard script expected stats_update, job_update and queue_update messages the server never sends; it now handles the server's JobUpdate, QueueUpdate and SystemAlert messages, coalescing reloads to at most one per second, and no longer answers a ping text message with a pong the server rejects. Its "is the section visible" check required the whole section on screen, so tall job and queue tables were never refreshed.
• Job lifecycle hardening (#64 C1, C2, H3, H9, H10, M1, M2, M5, M6, M8, M10):
  • C1: the stale job reaper no longer reclaims jobs that are still running. A claim wrote no lease, so until the worker's first heartbeat (a third of its lease in) any reaper with a shorter staleness window, such as another pool's or cargo hammerwork job requeue-stale --older-than-secs N, moved the running job back to Pending and it ran twice. The claim now writes lease_expires_at (and last_heartbeat_at) in the same statement on PostgreSQL, MySQL and TestQueue, and the reaper reclaims a job only once its lease expired; older_than only applies to jobs claimed by older versions without a lease. New DatabaseQueue::dequeue_leased(queue, weights, lease) (workers pass their lease; dequeue takes queue::DEFAULT_LEASE_DURATION, 5 minutes). A zero lease never expires (the reaper leaves the job alone; it used to be reclaimed after the pool's fallback window). heartbeat_job only extends the run it belongs to (same attempts and started_at), so a worker whose job was reclaimed and re-run elsewhere gets false and no longer keeps the new run's lease alive. No migration: the columns come from 015_add_job_leases.
  • C2: spawned children are no longer lost. The worker completed the parent and only then enqueued its children one at a time, so a database error, a spawn handler error, SpawnLimitExceeded, a crash or a shutdown left a Completed parent with some or none of its children. The children are now built before the completion (SpawnManager::prepare_spawn) and enqueued in the same transaction that completes the parent (new DatabaseQueue::complete_job_run_with_children); any spawn failure fails the parent's run, so it is retried. on_spawn_complete runs after the commit. The completion hook no longer fires twice for spawning jobs.
  • H3: concurrent migrations no longer fail. MigrationManager::run_migrations holds a migration lock for the whole run (pg_advisory_lock on PostgreSQL, GET_LOCK on MySQL, on a dedicated connection that runs every statement), re-reads the applied migrations after acquiring it, and releases it on every path. Several replicas with auto_migrate, or the CLI and an application, can migrate at once. Recording a migration ignores an existing row. New MigrationRunner::acquire_migration_lock / release_migration_lock (default: no lock). The PostgreSQL runner checks for the tracking table in current_schema() instead of public.
  • H9: autoscaling only starts or retires workers of the queue it scales (the template's, or the first worker's). It used to retire the highest-index workers whatever their queue, so other queues stopped being processed.
  • H10: a slow or failing alert target no longer stalls job processing. Alert HTTP requests time out after alerting::ALERT_HTTP_TIMEOUT (10 seconds), and an alert that reached no target is retried after FAILED_ALERT_RETRY_DELAY (60 seconds, or the cooldown if shorter) instead of on every check.
  • M1: disable_recurring_job stops the next run, and enable_recurring_job revives a finished job. Disabling used to leave the pending occurrence runnable; it is now held (a job with a cron schedule is only dequeued while it is recurring) and next_run_at is cleared. Enabling reschedules the job for the next occurrence after now (Pending, attempts reset) unless it is running, so a job whose last run finished while disabled runs again. Both report JobNotFound for missing jobs and an error for jobs without a cron schedule, on every backend.
  • M2: the Worker::with_max_retries example claimed it raises the attempt limit; it can only lower it. Worker::new no longer documents a 3-attempt cap.
  • M5: the 500ms default polling_interval was saved to TOML as "0s" and loaded back as zero; durations with sub-second precision are now written in milliseconds ("500ms"), and ms is accepted when parsing.
  • M6: the token bucket dropped the sub-millisecond part of the elapsed time on every refill, so frequent checks refilled slower than the configured rate.
  • M8: a worker with a metrics collector no longer runs COUNT(*) for the queue depth on every poll; the monitoring task already updates it.
  • M10: a worker's monitoring task is aborted however run ends (an error, a panic, or the future being dropped), so supervisor restarts no longer leak monitoring loops.
  • TestQueue::mark_job_dead and mark_job_timed_out apply the workflow failure policy and a fail-fast batch's failure like the database backends, TestQueue::finish_job_run reports the jobs failed with a run in RecordedOutcome::cancelled, and a stale job the TestQueue reaper marks Dead applies the policy too.
• Webhook and stream delivery, and config values that were ignored (part of #64: H11, H12, H13, M3, M4, M7, M9):
  • H11: WebhookManager::update_webhook started a second listener next to the old one, so after N updates every event was POSTed N+1 times. It now swaps the configuration in place (no gap, statistics kept); removing a webhook or stream, or adding one again under the same id, stops the old listener at once.
  • H12: a webhook or stream whose endpoint was down piled up one task per event without limit. At most WebhookManagerConfig::max_pending_deliveries deliveries per webhook (new, default 1000) and BufferConfig::max_events events per stream are pending; beyond that the listener waits, and events that overflow the event manager's buffer are dropped oldest-first, logged, and counted in the new WebhookStats::dropped_events / StreamStats::dropped_events. Retry backoff no longer holds a max_concurrent_deliveries / max_concurrent_processors slot.
  • H13: the webhook timeout covered only the response headers, so an endpoint that trickled its body held a delivery slot forever. timeout_secs now covers the whole attempt including the body, and at most max_response_body_size bytes of the body are read.
  • M9: webhook and stream listeners subscribed in their own task, so events published right after add_webhook / add_stream returned were missed. The subscription is now made before those calls return.
  • M7: InMemoryStatsCollector shifted its whole buffer on every event once full and copied up to 100,000 events per statistics query. Recording is now amortised O(1) (a time-ordered ring buffer) and queries aggregate the events in their window in place.
  • M4: settings that were accepted and never read now work: WebhookManagerConfig::default_timeout_secs (used when a webhook's timeout_secs is 0 or missing; also webhooks.global_settings.default_timeout_secs), StreamManagerConfig::global_flush_interval_secs (caps every stream's max_buffer_time_secs), MetricsConfig::update_interval (the worker monitoring interval when a metrics collector is set; it was fixed at 30 s), StatsConfig::max_event_age_secs (applied on every insert) and StatsConfig::collect_timing, and [logging] through the new LoggingConfig::try_init(). JobBatch::with_batch_size is documented as what it is: the chunk size of into_chunks (enqueue_batch always writes the whole batch in one transaction).
  • Behaviour changes: webhook statistics survive update_webhook; an attempt whose status arrived in time but whose body did not counts by its status; a webhook with timeout_secs = 0 uses the manager's default instead of timing out at once; a stream flushes when its oldest buffered event has waited max_buffer_time_secs (it used to flush a lone event after an idle period immediately); with a metrics collector the worker monitoring task runs every update_interval (15 s by default) instead of every 30 s; InMemoryStatsCollector drops events older than max_event_age_secs (1 hour by default) without waiting for cleanup_old_events. Breaking for struct literals: WebhookManagerConfig (max_pending_deliveries), WebhookGlobalSettings (default_timeout_secs, max_pending_deliveries), WebhookStats and StreamStats (dropped_events) have new fields; use ..Default::default() where available. tracing-subscriber now enables its json feature.
• The CLI no longer hangs when the database is unreachable: connecting gives up after 10 seconds with a clear error. Configure with connect_timeout_secs in config.toml, cargo hammerwork config set connect_timeout_secs N, or HAMMERWORK_CONNECT_TIMEOUT.
• Queue failure semantics are the same on every backend (#59):
  • PostgreSQL/MySQL fail_job_dependencies now applies the workflow's FailurePolicy, as documented. It used to fail the failed job's dependents under every policy: under Manual it failed jobs that should keep waiting, and under FailFast it left the workflow's other pending jobs and its status alone. It now locks the job and its workflow like a terminal transition, applies FailFast (the workflow's Pending/Retrying jobs and all dependents fail, workflow failed), ContinueOnFailure (only the transitive dependents fail) or Manual (nothing changes), and updates the workflow's counters and status.
  • TestQueue::fail_job is a terminal Failed, as the trait documents and the database backends implement. It used to increment attempts and move the job to Retrying, or to Dead once attempts ran out. It now also applies the workflow policy and a FailFast batch's failure, like the database backends. Code that relied on the old behaviour should do what a worker does: retry_job while attempts < max_attempts, then mark_job_dead.
  • TestQueue::dequeue counts an attempt (attempts + 1) like the database backends, so a Worker's retry/dead decision works the same on it.
  • TestQueue::fail_job_dependencies applies the policies like the database backends (it failed nothing under ContinueOnFailure and ignored jobs outside workflows), get_workflow_status derives the status from the policy (a failure under ContinueOnFailure or Manual no longer makes the workflow failed while jobs can still run), and enqueue_workflow honours each job's own depends_on.
  • The FailurePolicy and fail_job_dependencies docs describe the behaviour of each policy, including Manual (dependents wait until the failed job is re-run or the workflow is cancelled).
• CLI and web bugs found by testing every command and endpoint against both databases (#44):
  • CLI backup restore bound text into UUID/INT columns on PostgreSQL (it never worked there), dropped timestamps and the error message, and counted skipped duplicates as restored; --skip-existing compared a UUID with text. CSV backups now quote fields instead of replacing commas.
  • CLI cron was mostly stubs: create wrote text ids/priorities that PostgreSQL rejected and accepted five-field schedules the scheduler cannot run; list/next printed "coming soon"; update did nothing. They are now real (library scheduling, real next-run times, update recomputes the next run). cron create --description (never stored) was removed.
  • CLI queue list/pause/resume/paused use the library's queue-pause API (UTC clock on MySQL) and queue list includes paused queues without jobs.
  • CLI job show failed on PostgreSQL (UUID/priority decoded as text) and on MySQL (priority); a missing job is now an error. batch enqueue wrote text into UUID/INT columns on PostgreSQL and panicked on --batch-size 0; batch export printed "coming soon" and is now implemented. maintenance reindex named indexes that do not exist on PostgreSQL. spawn tree/lineage failed on MySQL (invalid JSON path), listings panicked on multi-byte queue names, and spawn monitor always errored. workflow list/show/cancel were placeholders and create stored nothing: they now use the library (create takes --jobs-file); dependents are derived from depends_on (the column is never written). archive --compress false could not disable compression, the set-policy/get-policy/remove-policy subcommands only printed that policy storage does not exist and were removed. webhook options: --priorities took only capitalised names, header values and basic-auth passwords containing =/: were rejected, a lone --api-key was silently ignored.
  • CLI raw SQL used MySQL NOW() (session time zone) where the library uses UTC_TIMESTAMP(6); windows now use the database UTC clock. HAMMERWORK_CONFIG (documented) is honoured, a malformed config.toml is reported instead of ignored, and URLs are redacted in migration logs. The unused utils::db_helpers module was removed.
  • Web: archive routes were mounted at /api/api/archive/...; unauthenticated or unknown /api requests were answered with the HTML page (200) instead of 401/404; non-auth rejections (400/404/405) were reported as 500; the lockout status was never produced; an empty configured password accepted an empty login; one WebSocket client held the global lock for its whole connection (blocking other clients, pings and broadcasts) and received nothing unless it subscribed; a client Ping was answered to everyone; POST /api/jobs ignored cron_schedule and silently turned unknown priorities into normal; job search listed recurring jobs twice; status=failed never matched dead jobs; paging reported a different page size than served; retry/delete of unknown jobs were 500s (now 404/409); archive purge dry runs ignored older_than, archive list ignored all filters except the queue. hammerwork-web command-line defaults overrode the configuration file, DATABASE_URL was documented but unused, and authentication defaulted on with no password. Behaviour change: hammerwork-web now refuses to start when authentication is enabled (the default) and no password is configured; configure one (--auth --password/--password-file, or auth.password_hash in the config file) or pass the new --no-auth flag to run without authentication explicitly, which logs a warning.
  • Dashboard script called endpoints that do not exist (/api/stats/throughput, /api/jobs/{id}/retry, DELETE /api/jobs/{id}), read data.jobs/data.queues instead of data.items, used the wrong overview field names, and passed fetch options to a helper that expected a method string (archive actions). New GET /api/stats/trends?hours=N.
• Job results visible after completion: the worker recorded a job as Completed and only then stored its result, so a client that saw the job complete could read no result (and a failed result write left the job Completed with only a warning). The result is now stored before the completion is recorded, and a failed write is logged as an error. Result TTLs too large for chrono::Duration saturate instead of silently becoming 24 hours.
• Worker pool autoscaling never added or removed a worker (#44). It only updated AutoscaleMetrics::active_workers. The pool now starts workers from its template when the autoscaler scales up (up to max_workers) and gracefully retires workers when it scales down (down to min_workers); retired workers are not restarted.
• ThrottleConfig::max_concurrent was ignored (#44). Workers given the throttle (and their clones, so a whole WorkerPool) now run at most max_concurrent jobs at once (0 is treated as 1). Worker::with_throttle_config ignores a throttle with enabled = false; it used to apply its rate limit anyway.
• Queue depth alerts from a worker needed a Prometheus collector (#44): the worker's monitoring task only checked queue_depth_threshold when with_metrics_collector was also set.
• encryption.purge_interval_secs = 0 was accepted when encryption was disabled (#44), and WorkerPool::from_hammerwork_config then ran the retention purge in a tight loop. It is now rejected whether or not encryption is enabled.
• InMemoryStatsCollector counted Started events as processed jobs (#44), so total_processed and throughput_per_minute were doubled and error_rate could never exceed 50% (a queue where every job failed reported 0.5, so alert_on_high_error_rate(0.5) never fired). total_processed now counts finished runs, and runs that failed but are retried (Retried) count as errors.
• PrometheusMetricsCollector::new panicked with custom gauges or histograms unless it ran inside a multi-threaded Tokio runtime (it used block_in_place) (#44). MetricsConfig::custom_labels ("labels added to all metrics") and collect_histograms = false were ignored; both now take effect, and invalid label names are rejected with HammerworkError::Metrics.
• shutdown_tracing() blocked the async runtime (#44): it shut the tracer provider down on the calling runtime thread, which the OTLP exporter needs to flush, so on a current-thread runtime it stalled for the provider's 5 second timeout. It now waits on a blocking thread.
• Retry strategies without a maximum delay could not be loaded from TOML (#44). max_delay_ms was a required field (null meant no maximum), but TOML has no null, so HammerworkConfig::save_to_file wrote a worker.retry_strategy (the default Linear/Exponential/Fibonacci without a cap) that from_file then rejected. A missing max_delay_ms now means no maximum.
• KeySource::Generated replaced its stored key on every start (#44). An EncryptionEngine with Generated("file:///path") (or a plain path, or env://VAR) generated a new key and overwrote the stored one each time it was created, so nothing encrypted before a restart could be decrypted. The stored key is now reused; a new key is generated only when none is stored (created with create_new and mode 0600, so concurrent starts do not overwrite each other), and a stored value that is not a valid key is an error instead of being replaced. stdout:// still generates a new key each time.
• A vault:// key source with an invalid addr panicked inside the Vault client (#44); it is now an EncryptionError::InvalidConfiguration. The vault-kms feature now enables the url dependency (already in the dependency tree).
• EncryptionEngine::identify_pii_fields skipped arrays inside objects (#44): {"contacts": [{"phone": ...}]} reported nothing; it now reports contacts[0].phone.
• KeySource::Static keys were printed by Debug and Display (#44), so logging a KeySource, EncryptionConfig or KeyManagerConfig exposed the key. Both now print Static(<redacted>).
• Creating a Google Cloud KMS or Pub/Sub client panicked (#44) with "Could not automatically determine the process-level CryptoProvider": the dependency tree enables both of rustls' crypto backends, so rustls cannot choose one. Hammerwork now installs the aws-lc-rs provider as the process default (unless the application installed one) before creating these clients. gcp-kms and google-pubsub now enable a direct rustls dependency (already in the tree).
• enqueue_cron_job on PostgreSQL and MySQL accepted jobs without a cron schedule (#44) and stored them as one-off jobs (not recurring), unlike TestQueue, which rejects them. Both backends now reject a job without a valid schedule, mark the job recurring, and compute next_run_at (and schedule the job for it) when it is not set.
• A stream disabled with StreamManager::disable_stream never delivered again after enable_stream (#44): disabling stopped the stream's listener for good. A disabled stream now drops events but keeps listening, so enabling it resumes delivery.
• WebhookManager::disable_webhook delivered one more event and then stopped the webhook for good (#44): the listener used the configuration it read before waiting for the event, then exited, so enable_webhook never resumed delivery. The state is now checked when each event arrives, and a disabled webhook keeps listening.
• TestQueue differed from the database backends (#44): get_job_counts_by_status keyed counts by lowercase names ("pending") instead of the stored status ("Pending") and listed empty statuses, and a fail-fast fail_job_dependencies left the failed jobs' dependency_status as Waiting. Code tested against TestQueue that reads the counts by key needs the capitalised names.
• Event streams never sent anything (#41). StreamManager prepared each batch and then recorded it as delivered without calling the processor, so Kafka, Kinesis and Pub/Sub streams dropped every event while their stats reported success. Batches now go through the stream's StreamProcessor::send_batch. Events that fail (or get no delivery result) are retried per the stream's StreamRetryPolicy: max_attempts attempts in total, exponential backoff capped at max_delay_secs, optional jitter. Statistics come from the per-event results: successful_deliveries, failed_deliveries, success_rate, avg_delivery_time_ms, last_success_at / last_failure_at and the new last_error. Events that cannot be serialized count as failed. StreamManager::shutdown stops retries: no retry starts after shutdown begins, a batch waiting out its backoff stops waiting, and the events it gives up on are counted as failed.
• Kinesis and Pub/Sub ignored the stream's serialization format and always sent JSON. They now send the bytes produced by serialization (MessagePack, Avro, Protobuf), like Kafka.
• Pub/Sub health_check always returned true. It now fetches the topic (bounded by health.check.timeout.ms, default 5000): Ok(true) if the topic exists, Ok(false) if it does not, if Pub/Sub is unavailable or if the check times out, and an error if Pub/Sub rejects the request (for example PERMISSION_DENIED).
• Webhook payload_template was ignored and the raw event was sent. It is now rendered (see Added). Invalid templates (bad JSON, unknown placeholder, unterminated {{) are rejected by add_webhook, update_webhook (which leaves the old webhook in place), WebhookManager::from_config and HammerworkConfig::from_file.
• Email alert targets did nothing: send_email_alert logged the alert and returned Ok. Email targets now send through SMTP (see Added). A target without SMTP settings is a configuration error in AlertingConfig::validate, AlertManager::try_new and HammerworkConfig::from_file; AlertManager::new logs it and every alert to that target fails.
• Alert delivery failures were swallowed. AlertManager logged a failed target and returned Ok. check_thresholds, check_queue_depth, check_worker_starvation and send_custom_alert now return HammerworkError::Alerting naming every target that failed (all targets are still tried). The cooldown only starts once some target received the alert, so an alert that reached nobody is retried on the next check.
• JobSpawnExt::with_spawn_config / with_spawning no longer drop the configuration silently or panic (part of #7): they stored the config in the payload only when it was a JSON object, silently skipped any other payload, and unwrapped serialization. Breaking: both now return Result<Job> and fail with HammerworkError::InvalidJobPayload when the payload is not a JSON object (the job schema has no spawn column, so the configuration travels in the payload under the new public spawn::SPAWN_CONFIG_KEY). New spawn::spawn_config_from_payload reports an invalid _spawn_config instead of ignoring it, and the worker logs it as a warning instead of a debug line.
• Encrypted jobs now spawn children: the worker looked for _spawn_config in the stored payload, which for an encrypted job is a placeholder or redacted, so spawning silently never happened. It now reads the configuration from the decrypted payload and hands the spawn handler the decrypted parent. cargo hammerwork spawn list/stats/pending query the stored payload and therefore do not list encrypted jobs (documented in docs/job-spawning.md).
• CLI commands that were never exercised against a database now work: queue stats --detailed, monitor dashboard and backup create read priority as text from an integer column (and backup create read the PostgreSQL UUID id as text) and failed; status icons matched lowercase names although statuses are stored capitalized; spawn list/stats joined hammerwork_jobs to itself with an unqualified queue_name/created_at (ambiguous column on PostgreSQL); spawn list/pending on MySQL could not decode the JSON_EXTRACT result.
• Queue backend follow-ups (#31, #17), PostgreSQL and MySQL. No migration.
  • Late dependents no longer wait forever: a job enqueued (alone, in a batch or in a workflow) after its dependencies finished was left waiting, because dependents are only released when a dependency finishes. Enqueue now settles it: satisfied when every dependency completed (archived ones included); inserted as Failed (dependency_status = failed, error naming the dependency, together with the jobs of the same insert that depend on it) when one failed, died or timed out, unless that dependency's workflow uses FailurePolicy::Manual. The dependencies are locked FOR SHARE during the insert, so one completing at the same moment cannot strand the new job.
  • Queue pause is enforced by the dequeue queries (dequeue, dequeue_with_priority_weights), not only by the worker's pre-check. TestQueue does the same.
  • Weighted dequeue no longer starves low priorities (#19 M4): it only looked at the 20 highest-priority runnable jobs, so a lower priority never ran while 20 higher-priority jobs were queued. It now probes each priority level for a runnable job, picks a level by weight among those that have one, and claims that level's oldest job (FOR UPDATE SKIP LOCKED), trying the other levels if all its jobs are locked.
  • Database clock (#19 M6): due-time checks (dequeue, get_ready_jobs, get_due_cron_jobs, result expiry), run timestamps (started_at, completed_at, failed_at, timed_out_at), heartbeats and leases, the stale-job reaper and cron rescheduling use the database's clock instead of each application server's. A worker's retry backoff keeps its length on the database clock. On MySQL all times are written and compared with UTC_TIMESTAMP(6) instead of the session-local NOW() (pause_queue wrote local times when the session time_zone was not UTC).
  • MySQL year-2038 limit (#19 M7): MySQL's TIMESTAMP(6) columns end at 2038-01-19 03:14:07 UTC. Enqueueing a job scheduled after that, and retry_job / reschedule_cron_job / store_job_result with such a time, now fail with HammerworkError::InvalidJobPayload naming the limit instead of a database error; library-computed times (worker retry backoff, leases) are capped, and a recurring job whose next occurrence is beyond the limit is not rescheduled. The columns are not converted to DATETIME(6): MySQL rebuilds the table for that, blocking writes to hammerwork_jobs, which an automatically applied migration must not do. See docs/migrations.md.
  • cancel_workflow cancels running jobs too: their status becomes Failed ("Workflow cancelled") along with pending and retrying jobs. The handler cannot be interrupted, but its outcome is discarded by the lifecycle guard (finish_job_run returns None, heartbeats return false).
  • Cron next run is computed from the run's slot, not from when the run ended, so a run that overran its next slot no longer skips it. Occurrences that pass while a run is still going are coalesced into one catch-up run, due immediately; a run that starts late (workers were down, queue paused) is itself the catch-up, so it runs once and the job goes straight back on schedule. Missed occurrences never pile up. See docs/cron-scheduling.md.
  • PostgreSQL get_queue_depth and the dequeue queries compare status with a literal, so the partial polling index applies to generic (prepared) plans; with a bind parameter the depth count scanned every pending job of every queue.
  • Dependency resolution on completion uses a constant number of queries for all dependents instead of one per dependent. MySQL's JSON_CONTAINS lookup of dependents still cannot use an index (documented).
  • get_priority_stats swallowed every database error and always reported empty average processing times: the processing time is NUMERIC (PostgreSQL) / DECIMAL (MySQL) and never decoded as an integer. It is now cast in SQL and errors propagate. get_processing_times failed with a decode error for the same reason. PostgreSQL get_jobs_completed_in_range matched the lowercase status 'completed' and never found anything.
  • Rows are read with try_get(..)? instead of panicking Row::get (queue backends and migration runners). Corrupt column values are errors instead of silent defaults: depends_on / dependents / pii_fields that are not JSON lists, unknown dependency_status values, undecodable retry_strategy / result_config (including in the archive), unparseable archived batch_ids and batch error ids. Negative or out-of-range integers (timeout_seconds, result_ttl_seconds, counts) are errors instead of wrapping to huge values. Archiving a job no longer stores an empty string for a retry strategy that fails to serialize.
• Abandoned database transactions on worker shutdown: the worker loop raced acquire_job() against shutdown, and the shutdown grace period cancelled outcome recording, so a dequeue or finish_job_run transaction could be dropped mid-flight. That could return a pooled connection with the transaction still open and its row locks held: on MySQL, Lock wait timeout exceeded and error 1568 on every later claim on that connection. Shutdown now interrupts only the worker's waits; dequeues always complete, and outcomes are recorded in a spawned task. As a safety net, the MySQL claim path rolls back a connection it finds still inside an abandoned transaction.
• 🚨 Webhooks and event streams dropped events: each webhook and stream listener re-subscribed to the event channel on every loop iteration, so events published while it handled the previous one were skipped. In a test burst of 100 events, a stream processor received 1. Listeners also exited on any receive error, so a burst larger than the event buffer (RecvError::Lagged) permanently stopped delivery for that webhook or stream. Listeners now keep a single subscription, log and continue when they fall behind, and stop only when the channel closes or the webhook or stream is removed.
• CLI correctness (#31)
  • cargo hammerwork spawn stats, spawn pending, archive ... and job purge no longer panic in debug builds: -q was both --queue and the global --quiet. The short flag for --queue on those subcommands is now -Q (long flags unchanged; -q is the global --quiet). A unit test runs clap's debug_assert over the whole CLI.
  • job retry, job cancel, batch retry, batch cancel and the maintenance dead-job cleanup and integrity fixes no longer rewrite hammerwork_jobs with raw SQL (which wrote a lowercase 'pending' the library never reads). They find the affected jobs, then apply each change through retry_job / retry_dead_job / mark_job_dead / delete_job, so they get the lifecycle transition guards and side effects. Jobs that changed state in the meantime are skipped and reported instead of overwritten.
  • Lowercase status literals in queue list, queue clear --pending-only, queue health, monitor and job purge matched nothing (statuses are stored capitalized); they now use Pending, Running, Completed, Failed, Dead. job purge binds the queue name instead of interpolating it. maintenance check compared the integer priority column with text; it now checks 0..=4.
• Detached tasks are tracked (#17 M6): WebhookManager and StreamManager keep their listener and delivery/batch tasks instead of dropping the JoinHandles, so a panic is logged (and counted in the new task_panics()) rather than lost. New WebhookManager::shutdown(grace) and StreamManager::shutdown(grace) stop the listeners, wait up to grace for in-flight deliveries (and their retries), abort stragglers and return how many were aborted.
• Robustness: web dashboard, CLI, stats, webhooks, streaming and metrics (#17, medium items)
  • hammerwork-web handlers no longer turn database errors into empty results: job/queue/stats/archive endpoints log the error and answer 500 with the usual {"success": false, "error": ...} body (bad IDs are 400, unknown queues 404). The archive dry-run purge no longer reports 0 on error, and a negative job age (clock skew) no longer wraps to ~1.8e19 seconds. The dashboard shows the server's error message. WebDashboard::new and DashboardConfig::from_file reject a zero websocket.ping_interval (new DashboardConfig::validate) instead of panicking later in the ping task.
  • cargo-hammerwork reads rows with try_get(..)? (schema drift is an error, not a panic) and no longer swallows decode/query errors in spawn, workflow, monitor and job commands; they exit non-zero. backup restore rejects unparseable timestamps and out-of-range integers instead of substituting "now"/truncating. job enqueue --delay/--max-attempts and monitor dashboard --refresh 0 are validated. The spawn statistics and monitor metrics averages are cast to double in SQL (they are NUMERIC/DECIMAL, which never decoded as f64).
  • WebhookManager::try_new returns an error instead of panicking when the HTTP client cannot be built (new still panics, now documented); from_config uses it. max_concurrent_deliveries / max_concurrent_processors of 0 (every delivery waits forever) or above Semaphore::MAX_PERMITS (panic) are clamped with a warning. The webhook delivery duration, stream delivery durations and retry backoff no longer use lossy casts, and Kinesis access-key masking no longer panics inside a multibyte character. A non-numeric Kafka health.check.timeout.ms is a configuration error rather than silently the default.
  • InMemoryStatsCollector recovers from poisoned locks and no longer panics on huge windows (Duration::MAX); AlertManager::check_worker_starvation no longer panics on a huge threshold.
  • PrometheusMetricsCollector::start_exposition_server binds the address before returning, so a port that is in use or not permitted is returned as HammerworkError::Metrics instead of the server task dying silently.
• Migrations are safe to re-run after a partial failure (#7): MySQL DDL auto-commits, so a migration that failed part-way stayed half-applied and every retry failed with Duplicate column name / Duplicate key name. All MySQL migrations now guard each ADD COLUMN, CREATE INDEX, DROP INDEX and ADD/DROP CHECK against information_schema (one statement per column instead of multi-column ALTERs). The resulting schema is unchanged. PostgreSQL already ran each migration in a transaction; its remaining non-idempotent statements (ADD CONSTRAINT, migration 013 indexes, the one-way JSONB-to-array conversion in 012) were guarded too. New migration_idempotency_tests replay every migration over a finished schema on both backends and compare the schema.
• Streams for disabled backends now fail at configuration time (#23): without the kafka, kinesis or google-pubsub cargo feature, KafkaProcessor::new / KinesisProcessor::new / PubSubProcessor::new return HammerworkError::Streaming naming the feature, so StreamManager::add_stream and from_config reject the stream immediately. Previously the Kafka placeholder silently "delivered" events (simulated delay and random failures, nothing sent) and the others only failed per event at send time. The simulation settings batch.delay.ms and test.error.rate are gone. Behaviour change: configurations that relied on the placeholder must enable the matching feature.
• The crate failed to compile without the webhooks feature (including --no-default-features --features postgres or mysql), because HammerworkConfig used events::EventConfig from a module gated on webhooks. The events module and its re-exports no longer require webhooks; CI now checks the library across feature combinations.
• 🚨 AWS and GCP KMS keys no longer change on every restart (#26). aws:// and gcp:// key sources called AWS KMS GenerateDataKey / GCP KMS GenerateRandomBytes on every load, so each process got a new random key and anything encrypted or wrapped by an earlier process could not be decrypted after a restart. They now use envelope encryption:
  • On first use the KMS generates a data key and only its KMS-encrypted form is stored, in the new hammerwork_kms_data_keys table, with the KMS key id, version and creation time. Later loads call KMS Decrypt on the stored blob, so every process and restart gets the same key. Processes starting at once converge on one stored key (unique (key_name, kms_provider, kms_key_id, key_version), insert-if-absent, read back).
  • The ciphertext is bound to the key's name (AWS encryption context, GCP additional authenticated data).
  • KeyManager::new does this for aws:// / gcp:// master keys. New EncryptionEngine::new_with_pool(config, &pool) does it for engine data keys (PostgreSQL or MySQL); EncryptionEngine::new now returns EncryptionError::InvalidConfiguration for these sources, since it has nowhere to store the key.
  • Rotation: new KeyManager::rotate_kms_master_key and EncryptionEngine::rotate_kms_key add a new active version and retire the previous one, which stays decryptable (keys and key-encryption keys wrapped by an older master key version, and payloads encrypted with an older engine key, still decrypt).
  • Migration 017 (017_add_kms_data_keys, PostgreSQL and MySQL) creates hammerwork_kms_data_keys. Run cargo hammerwork migration run before upgrading.
  • Upgrading: existing AWS/GCP deployments generate and store a key on their first start after the upgrade. Data encrypted by the earlier per-process keys was already unrecoverable and stays so.
  • GCP sources now also need permission to Encrypt/Decrypt with the configured CryptoKey (previously only GenerateRandomBytes was called); AWS sources need kms:Decrypt in addition to kms:GenerateDataKey.
• 🚨 Workflow dependencies never resolved (#19 C3): nothing called resolve_job_dependencies or fail_job_dependencies, so dependents stayed waiting forever and failure policies and workflow counters were never applied. Completing a job now marks dependents whose dependencies all completed as satisfied (dependents are locked first, so two parents completing at once cannot both miss each other). A terminal failure applies the workflow's policy: FailFast fails the workflow's pending jobs, ContinueOnFailure fails the jobs that (transitively) depend on the failed one, Manual leaves them waiting for retry_dead_job. Workflow completed_jobs, failed_jobs, status, completed_at and failed_at are kept up to date. Stale jobs the reaper marks Dead get the same treatment. The PostgreSQL dependency lookup now uses the GIN index on depends_on.
• Terminal jobs could be overwritten or resurrected (#19 H2): status updates had no current-status guard, so a TimedOut job could be completed and a Dead one put back to Pending, and a late worker could overwrite a job the reaper had reclaimed (see finish_job_run and the guarded manual transitions above).
• Retry limit ignored the job's max_attempts (#19 H3): a job with max_attempts = 5 stopped after 3 runs in a Failed state that nothing retried, and max_attempts = 1 still ran 3 times.
• Recurring jobs stopped recurring after one failure (#19 H5): a cron job whose run died or timed out stayed Dead/TimedOut forever. It is now rescheduled for its next run; the failed run's error and failure time stay on the job until then.
• Batch failure modes were not enforced (#19 H6): PartialFailureMode::FailFast now fails the batch's not-yet-started jobs on the first terminal failure, and the hammerwork_batches row (status, counters, completed_at, error_summary) is written when the batch finishes. Removed the worker comment that claimed the queue layer already did this.
• Timeouts were always final (#19 M1): see Changed.
• Job::with_retry_strategy was lost on enqueue (#19 M2): it is stored in the new retry_strategy column and read back by every query, so the worker's per-job strategy is used.
• enqueue_batch dropped job fields (#19 M3): PostgreSQL batch inserts dropped the result config, dependency, workflow and trace fields; MySQL dropped the dependency, workflow and trace fields, so batch jobs with dependencies ran immediately. enqueue, enqueue_batch, enqueue_workflow and archive restore now share one insert that stores every field.
• MySQL dequeue reported a started_at slightly later than the one stored.
• 🚨 Duplicate job execution with weighted priority dequeue (PostgreSQL): dequeue_with_priority_weights ran its FOR UPDATE SKIP LOCKED candidate query outside a transaction, so the row locks were released before the claiming UPDATE, and that UPDATE didn't check the job's status. Concurrent workers could claim and run the same job more than once. In a 16-worker test, 76 of 200 jobs ran more than once. Workers configured with non-strict PriorityWeights (including those built through Worker::with_config) used this path. The lock and claim now happen in one transaction, and the claim requires status = 'Pending'.
• Weighted dequeue on PostgreSQL returned jobs with default values for result storage, dependencies, workflow and trace fields, so job results were not stored. The job is now built from the full row.
• MySQL dequeues and the PostgreSQL weighted dequeue ignored dependency_status and could run a job before its dependencies completed.
• MySQL dequeues now use FOR UPDATE SKIP LOCKED (MySQL 8.0+), so concurrent workers skip rows being claimed instead of blocking on them.
• MySQL job claims run at READ COMMITTED and retry when InnoDB aborts them as deadlock victims (error 1213), which concurrent workers otherwise hit intermittently.
• Worker robustness (#17, #19)
  • Errors from completing, failing or retrying a job (and from dequeuing) are logged and followed by an exponential backoff that honours ThrottleConfig::backoff_on_error, instead of being discarded with the worker spinning hot (H1)
  • A panicking job handler no longer kills the worker: the panic fails the job with Job handler panicked: <message> and goes through the normal retry / dead path (H2)
  • WorkerPool supervises its workers: a worker that dies is logged and restarted instead of the pool returning on the first failure and detaching the rest (H3)
  • WorkerPool::shutdown is graceful: in-flight jobs finish (up to the grace period) and record their outcome before workers stop, and shutdown waits for them. Dropping the pool also shuts it down. Previously shutdown cancelled the running handler and left the job Running (C2)
  • Huge retry delays no longer panic in chrono::Duration::from_std(..).unwrap(); retry delays are clamped to 365 days (H4)
  • RetryStrategy::calculate_delay no longer panics on overflow (exponential from attempt ~65, Fibonacci ~94, huge linear values) or on a NaN / negative multiplier; it saturates and clamps to max_delay. Jitter with an invalid factor no longer panics (H5)
  • Cloning a RetryStrategy::Custom (e.g. through Worker::clone for autoscaling) no longer panics (H6)
  • Webhook delivery no longer panics when truncating a response body in the middle of a multi-byte UTF-8 character (H7)
  • The autoscaler no longer panics on a zero evaluation_window
• KeyManager persistence works on PostgreSQL and MySQL (#9). Key storage had never worked: the generic storage methods were stubs returning "Database-specific implementation required", so generate_key, get_key, rotate_key and generate_master_key (with auditing) always failed.
  • Storage is implemented through a sealed KeyManagerBackend trait for sqlx::Postgres and sqlx::MySql; the KeyManager methods are now generic over it (is_key_due_for_rotation, get_keys_due_for_rotation, query_database_statistics and the rotation-schedule methods were previously separate per-backend inherent methods with the same signatures).
  • rotate_key inserts the new version and retires the old one in one transaction; old versions stay readable with the new KeyManager::get_key_version and are pruned to max_key_versions (0 keeps all). Generating an existing key ID is an error instead of an overwrite.
  • perform_automatic_rotation rotates keys whose next_rotation_at has passed (it was a no-op) and reports keys it failed to rotate. start_rotation_service rejects a non-positive interval instead of panicking.
  • refresh_stats (also run by KeyManager::new) reads key counts and ages from the database; it was a no-op. The statistics queries no longer fail to decode AVG results.
  • generate_master_key stores the new key-encryption key (encrypted with the configured master key) and retires the previous one; keys it encrypted can still be decrypted, and new KeyManager instances load it. A KeyManager whose configured master key cannot decrypt the stored key-encryption key fails to start.
  • key_source is stored as a label that satisfies the schema CHECK constraint (Generated, ...) instead of Generated(rotation). parse_key_source accepts these bare labels (and Derived) as well as the parenthesised form.
  • PostgreSQL rotation_interval is bound and read as an INTERVAL (it was bound as text and failed).
  • Key usage counters only count the active version.
• Migration 016 (016_versioned_encryption_keys, PostgreSQL and MySQL): hammerwork_encryption_keys is unique on (key_id, key_version) instead of key_id, so rotated versions are kept, and the audit log accepts the Update operation.
• Doctests: fixed the encryption engine, key manager, lib.rs configuration and hammerwork-web queue/archive examples to match the current API. They all compile (and ignored examples pass under --include-ignored).
• Archiving (#14), PostgreSQL and MySQL:
  • archive_jobs now moves jobs: the archive insert and the delete from hammerwork_jobs run in one transaction, and candidates are selected with FOR UPDATE SKIP LOCKED so concurrent archivers do not collide. Previously the row stayed in hammerwork_jobs with its old status.
  • get_job falls back to the archive table and returns archived jobs with JobStatus::Archived (previously it returned the stale pre-archive row)
  • restore_archived_job no longer fails with a duplicate key; it moves the row back atomically and returns HammerworkError::JobNotFound for an id that is not archived. Rows left in hammerwork_jobs by the old archiver are cleaned up on restore.
  • list_archived_jobs reports the stored ArchivalReason instead of always Automatic. Added ArchivalReason::as_str / parse_from_db (accepts the plain and legacy JSON-quoted forms).
  • MySQL get_archival_stats no longer fails to decode SUM() results (DECIMAL)
  • JobArchiver::archive_jobs_with_progress / archive_jobs_with_events archive batch after batch until no eligible jobs remain (bounded by MAX_ARCHIVAL_BATCHES_PER_OPERATION) instead of a single batch_size batch. The progress callback is called after each batch; archive_jobs_with_events publishes BulkArchiveProgress between batches.
• Batches (#14):
  • PostgreSQL get_batch_status failed to decode the VARCHAR status column. BatchStatus and JobStatus now accept every column type String decodes from.
  • get_batch_status (PostgreSQL and MySQL) reports live progress tallied from the batch's jobs, including archived ones. The counters stored in hammerwork_batches were never updated after enqueue, so pending/completed/failed counts and the status were frozen.
• hammerwork-web (#14):
  • WebDashboard::start() no longer panics when CORS is disabled; no CORS filter is installed in that case
  • With both postgres and mysql features enabled, the dashboard picks the backend from the database URL scheme instead of rejecting MySQL URLs. The pool now honours pool_size.
  • Building without a database feature fails with a single clear compile_error!
• cargo-hammerwork (#14):
  • spawn and workflow commands on PostgreSQL treated depends_on/dependents as JSONB, but they are UUID[] (since migration 012); the queries failed with operator does not exist: uuid[] @> jsonb
  • job list compared the integer priority column with a name (integer = text on PostgreSQL), filtered by lowercase status names the library never stores, and failed to decode priority. It now binds parameters instead of interpolating them, maps status names to stored values and priorities to integers.
• EventManager::new panicked when max_buffer_size was 0 (as set by HammerworkConfig::with_events_enabled(false)). A zero buffer now disables event publishing.
• 🔧 MySQL 8 migrations
  • Migration statements now run over the text protocol, so the PREPARE/EXECUTE blocks in migration 010 no longer fail with error 1295
  • Replaced MariaDB-only DROP INDEX IF EXISTS (010) and ADD COLUMN IF NOT EXISTS (011) syntax with MySQL 8 compatible statements
• Library, CLI, web dashboard, integration binaries and examples build without warnings and pass cargo clippy -- -D warnings (#7). spawn_cli_example compiles again, and spawn_example now declares required-features = ["postgres"].