v3.0.0

A major release. Read Upgrading from 2.x to 3.0 before
upgrading; it starts with the order to do things in. In short:

• Dependencies: sqlx 0.9 (your own sqlx must move too) and Rust 1.94.0.
• Features: no default features (add the ones you use, or full); azure-kv is
  azure-kms and test is test-queue.
• API: DatabaseQueue and MigrationRunner are sealed; one dequeue(&DequeueOptions)
  replaces the five claim methods; configuration, statistics and growing enums are
  #[non_exhaustive] with builders; zero rates and other invalid settings are rejected
  by fallible constructors; cargo-hammerwork is binary-only and hammerwork-web exposes
  only the dashboard types.
• Database: migration 021 moves MySQL to DATETIME(6) (no year-2038 limit; MySQL
  8.0.13+), rewriting every table. Stop 1.x workers first: the stale-job reaper now
  relies on leases alone (cargo hammerwork maintenance check --fix repairs rows 1.x
  left behind).

Breaking changes

See Upgrading from 2.x to 3.0 for what to change.

• sqlx 0.9 (#85). Hammerwork now depends on sqlx 0.9 instead of 0.8. sqlx types are
  part of the public API (the pool passed to JobQueue::new, JobQueue<DB: sqlx::Database>,
  DatabaseQueue, the migration runners, sqlx Encode/Decode for JobStatus and
  BatchStatus), so applications must move their own sqlx dependency to 0.9 as well.
  sqlx 0.9 also changes behaviour that applications may notice through their own sqlx
  code: the combined runtime/TLS features are gone (runtime-tokio-rustls is now
  runtime-tokio + tls-rustls-ring-webpki), dynamic SQL strings must be wrapped in
  sqlx::AssertSqlSafe, MySQL RSA password encryption for non-TLS connections needs the
  mysql-rsa feature (the CLI and dashboard enable it; the library does not), and MySQL
  connections no longer force the utf8mb4_unicode_ci collation. No schema migration is
  needed.
• MSRV is now Rust 1.94.0 (was 1.91.1), as required by sqlx 0.9.
• No features are enabled by default (#92). default used to be ["metrics", "alerting", "webhooks"], which pulled warp, reqwest, lettre and prometheus into every build. It is now empty. Enable what you use, or the new full feature (metrics, alerting, webhooks, streaming, tracing, encryption; no database backend, KMS key source, event stream sink or test-queue). Code that uses hammerwork::metrics, hammerwork::alerting or hammerwork::webhooks must now enable those features. cargo-hammerwork and hammerwork-web enable the features they need themselves.
• azure-kv is renamed azure-kms, to match aws-kms, gcp-kms and vault-kms (#92). There is no alias. The same rename applies to the cargo-hammerwork feature.
• test is renamed test-queue, after the TestQueue it enables (#92). There is no alias.
• DatabaseQueue is sealed (#87). It can no longer be implemented outside Hammerwork; its implementations are JobQueue<Postgres>, JobQueue<MySql> and TestQueue, and it stays usable as a bound (impl DatabaseQueue, Q: DatabaseQueue). Use TestQueue (feature test-queue) to test without a database. The default method bodies that existed only for implementations outside the crate are gone (the one-at-a-time fallback for batch claims, and the fallbacks of release_job_run, finish_job_run, complete_job_run_with_children, heartbeat_job, start_claimed_run, requeue_stale_jobs, purge_expired_encrypted_jobs, purge_archived_jobs_in_queue and list_queue_names); every backend implements these methods, so behaviour does not change. queue::RELEASED_JOB_MESSAGE, the error message of the removed release_job_run fallback, is removed.
• One claim method (#89). DatabaseQueue::dequeue(queue, &DequeueOptions) replaces dequeue, dequeue_with_priority_weights, dequeue_leased, dequeue_batch and dequeue_batch_leased. It returns Vec<Job>. The new #[non_exhaustive] queue::DequeueOptions (also hammerwork::DequeueOptions) has priority_weights: Option<PriorityWeights>, lease_duration: Option<Duration> (None is DEFAULT_LEASE_DURATION) and max_jobs (default 1), built with DequeueOptions::new()/default(), DequeueOptions::batch(n), with_priority_weights, with_lease_duration and with_max_jobs. DatabaseQueue::dequeue_one(queue) keeps the common case short: it is the 2.x dequeue(queue) (one job, strict priority, default lease, Option<Job>). Claims behave as before: a max_jobs of 1 claims like the old single-job methods, larger values like the old batch claims.
• Worker::with_batch_processing_enabled is removed (#89). with_batch_size(n) enables batch mode, as it already did; with_batch_size(1) replaces with_batch_processing_enabled(true) (batch statistics, one job per poll). New optional [worker] settings batch_size and batch_concurrency (WorkerConfig::batch_size, WorkerConfig::batch_concurrency) set them from configuration through Worker::with_config; WorkerConfig is now non-exhaustive (see below), so build it with WorkerConfig::default().with_batch_size(n).
• Worker::with_max_retries is renamed with_attempt_cap (#89), since it can only lower a job's own max_attempts. There is no alias.
• MySQL times are stored as DATETIME(6) (#91). Migration 021_mysql_datetime converts every TIMESTAMP column of every Hammerwork table to DATETIME(6), which lifts the year-2038 limit: jobs can be scheduled, retried, leased and archived up to year 9999. The MySQL backend no longer rejects times after 2038-01-19 03:14:07 UTC (and no longer caps leases at it); only times outside years 1000 to 9999 are rejected with HammerworkError::InvalidJobPayload. MySQL rebuilds every table for the conversion, which blocks writes while it runs: run it off-peak. Its UTC column defaults need MySQL 8.0.13 or later. See MySQL timestamps (migration 021).
• Configuration, statistics and record structs are #[non_exhaustive] (#81). Code outside Hammerwork can no longer build them with struct literals (not even with ..Default::default()); fields stay public for reading and assignment. Configuration types are built from default() or a constructor and with_* methods, and every field of the configuration sections now has one: new setters on DatabaseConfig, WorkerConfig, WebhookConfigs, WebhookGlobalSettings, StreamingConfigs, StreamingGlobalSettings, ArchiveConfig, RateLimitingConfig (plus with_queue_throttle), LoggingConfig, PayloadEncryptionConfig (plus with_decryption_key), KeyRotationConfig, EventConfig, webhooks::RetryPolicy, WebhookManagerConfig, StreamRetryPolicy, BufferConfig, StreamManagerConfig, SpawnConfig, StatsConfig and KeyDerivationConfig; HammerworkConfig::with_database/with_worker/... for each section; KeyManagerConfig::with_key_derivation_config; and new constructors JobError::new, JobLifecycleEvent::new (with setters), stats::JobEvent::new (with setters), ExternalKmsConfig::new (with with_auth, with_auth_config, with_region, with_namespace), Migration::new, ArchivedJob::new and SpawnContext::new. stats::QueueStats now implements Default. Also non-exhaustive: Job, ResultConfig, JobBatch, JobGroup, EventFilter, WebhookConfig, StreamConfig, AlertingConfig, SmtpConfig, ArchivalPolicy, ArchivalConfig, EncryptionConfig, KeyManagerConfig, KeyAuditFilter, MetricsConfig, TracingConfig, ThrottleConfig, AutoscaleConfig and JobEventHooks (all already had builders), every *Stats struct, and the records WebhookDelivery, StreamDelivery, StreamedEvent, EncryptionKey, KeyAuditRecord, KmsWrappedKey, EncryptionMetadata, EncryptedPayload, MigrationRecord, QueuePauseInfo, RecordedOutcome, SpawnResult, JobHookEvent, JobResult, BatchResult, Alert and ScheduledArchivalPass. In hammerwork-web, DashboardConfig, AuthConfig and WebSocketConfig are non-exhaustive, with new setters (DashboardConfig::with_pool_size, with_auth_config, with_websocket, with_allowed_origins; every field of AuthConfig and WebSocketConfig; AuthConfig::disabled()). See the upgrade guide for the full list.
• ArchivedJob has workflow_id and workflow_name (#81), filled in by list_archived_jobs on PostgreSQL, MySQL and TestQueue from the archive columns added in migration 019. No migration is needed.
• Dropping an EventSubscription unregisters it from its EventManager (#81), so subscription_count and payload_wanted stop counting it without a call to EventManager::unsubscribe (which still works). The subscription holds a private reference to the manager's registry, so its receiver can no longer be moved out of it: keep the subscription alive while reading. The registry uses a std lock, so dropping works outside a Tokio runtime. The webhook and stream managers' listeners hold their subscriptions and unregister when they stop.
• WebhookStats::failed_attempts and WebhookManagerStats::failed_attempts (#81) count failed HTTP delivery attempts, retries included, next to total_attempts and the per-delivery counters.
• GET /api/system/metrics drops metrics_endpoint, custom_metrics_count and last_scrape (#81). The dashboard serves no /metrics route and keeps no metrics registry, so they were always "/metrics", null and null; only prometheus_enabled remains.
• Growing public enums are #[non_exhaustive] (#86). A match on them outside Hammerwork needs a wildcard arm. Affected: the errors HammerworkError, EncryptionError, CronError, SpawnError and PriorityError; the events JobLifecycleEventType, stats::JobEventType, ArchiveEvent, AlertType and KeyOperation; and the backend and option choices StreamBackend, SerializationFormat, PartitioningStrategy, PartitionField, AlertTarget, SecretSource, WebhookAuth, HttpMethod, KeySource, RetentionPolicy, EncryptionAlgorithm, PayloadEncryptionAlgorithm, KeyPurpose, ArchivalReason, PartialFailureMode, ResultStorage, PrioritySelectionStrategy, RetryStrategy, JitterType, FailurePolicy, JobHandlerType, JobTransition and JobOutcome. JobStatus, JobPriority, BatchStatus, WorkflowStatus, DependencyStatus, KeyStatus, AlertSeverity and SmtpTls stay exhaustive. The CLI refuses to webhook test a webhook whose HTTP method or authentication it does not know, and the dashboard shows an unknown job event as status Unknown and does not forward unknown archive events.
• The stale job reaper relies on leases alone (#90). DatabaseQueue::requeue_stale_jobs(queue_name: Option<&str>) loses its older_than argument and takes an optional queue name (None reclaims every queue; Some(name) reclaims that queue only, atomically with the lease check): the older_than window only reclaimed Running jobs that Hammerwork 1.x had claimed without a lease, and that fallback is gone. A Running job without a lease of its current run is never reclaimed; every 2.x and 3.x claim writes one. WorkerPool::with_stale_job_reaper(interval) loses its older_than argument too. Stop every 1.x worker before upgrading, and fix the jobs they left Running once (cargo hammerwork maintenance check --fix, or the SQL in the upgrade guide).
• CLI: job requeue-stale --older-than-secs and maintenance dead-jobs --stale-hours are removed (#90). Both commands now use the lease rule only: a Running job is stale when its lease expired; lease-less jobs (left by 1.x) are left alone. maintenance check reports Running jobs without a lease, and check --fix gives them a lease that has already expired, so the next reaper pass reclaims them.
• AlertingConfig::email(recipient) is removed (#90), deprecated since 2.0.0 because the target it created could not send. Use email_via_smtp(recipient, SmtpConfig). AlertTarget::Email::smtp is now a required SmtpConfig (was Option<SmtpConfig>), so a configuration file with an email target without an smtp table fails to load with a missing-field error instead of a "no SMTP settings" validation error.
• JobArchiver holds no pool (#90). JobArchiver::new() takes no argument and JobArchiver is no longer generic over the database (JobArchiver<DB> is now JobArchiver); the pool it took was never used, since every operation takes the queue it works on. JobArchiver now implements Default.
• Zero rates are rejected (#90). A rate limit of zero never refilled, so a worker using it waited forever. RateLimit::per_second, per_minute and per_hour now return Result<RateLimit> and fail with HammerworkError::Config for zero. RateLimit's fields are private (read them with rate(), per() and burst_limit()), so a zero rate cannot be built. For an enabled ThrottleConfig with rate_per_minute(0): the new ThrottleConfig::validate and RateLimitingConfig::validate reject it, ThrottleConfig::to_rate_limit returns Result<Option<RateLimit>>, Worker::with_throttle_config and Worker::with_hammerwork_config now return Result<Worker>, JobQueue::set_throttle fails, and HammerworkConfig::from_file and WorkerPool::from_hammerwork_config reject such a rate_limiting section.
• MigrationRunner is sealed (#90). Hammerwork's migrations exist for PostgreSQL and MySQL only, so the trait is implemented by PostgresMigrationRunner and MySqlMigrationRunner only; it stays usable as a bound and as Box<dyn MigrationRunner<DB> + Send + Sync>. The no-op default bodies of acquire_migration_lock and release_migration_lock are removed; both runners implement them.
• Configuration sections follow the features (#130). [webhooks], [streaming], [alerting] and [metrics] are optional in a HammerworkConfig file when their feature is enabled (they were required), so a file saved by a build without the feature loads. Without the feature, HammerworkConfig::from_file now fails with HammerworkError::Config naming the feature to enable, instead of silently ignoring the section.
• config::SimpleEventFilter is removed (#130). It existed only in builds without webhooks and was never used; use events::EventFilter.
• RateLimit::with_burst_limit returns Result<RateLimit> and rejects 0 (it acted as 1) with HammerworkError::Config (#130).
• Worker::with_attempt_cap returns Result<Worker> and rejects a cap below 1 with HammerworkError::Config (it silently acted as 1) (#130).
• WebhookManager::new and new_default return Result<WebhookManager> instead of panicking when the HTTP client cannot be built; WebhookManager::try_new is removed (#130).
• WorkerConfig::validate checks the batch settings (#130): batch_size = 0 (which turned batch mode on), a batch_size above MAX_DEQUEUE_BATCH_SIZE and batch_concurrency = 0 are rejected by HammerworkConfig::from_file/from_env, WorkerPool::from_hammerwork_config and now also Worker::with_hammerwork_config.
• cargo-hammerwork is a binary only (#88). The crate no longer has a library target: cargo_hammerwork::commands, cargo_hammerwork::config, cargo_hammerwork::utils and their glob re-exports are gone. Drive the CLI through the cargo hammerwork executable.
• hammerwork-web exposes only WebDashboard, DashboardConfig, AuthConfig and WebSocketConfig (#88). The api, auth, config, live, security, server and websocket modules are private, so hammerwork_web::server::app_routes and the other building blocks are no longer public API. WebSocketConfig is now re-exported at the crate root (it was hammerwork_web::config::WebSocketConfig). There is no __private re-export of the internals either: the crate's own tests live inside the crate.

Added

• job requeue-stale --queue <name> and --dry-run. --queue reclaims only that queue's stale jobs through requeue_stale_jobs(Some(queue)); --dry-run counts the jobs the lease rule would reclaim without changing anything. Both use bound parameters on PostgreSQL and MySQL.
• migration status lists applied and pending migrations, from hammerwork_migrations against the migrations the library registers, plus applied migrations this version does not know (it used to list the columns of hammerwork_jobs; the column count remains as a summary). New MigrationManager::registered_migrations() returns the registered migrations without touching the database.

Fixed

• monitor health and queue health no longer call a Running job stuck after an hour. A job is stuck when its lease expired (the rule of requeue_stale_jobs, shared with maintenance dead-jobs and job requeue-stale), so a long-running job that keeps heartbeating is healthy. Running jobs without a lease (left by a Hammerwork 1.x worker) are reported on their own Lease-less Running Jobs row; monitor health --format json gains that check.

• Building with encryption but no database backend (and so with full) no longer fails clippy with dead-code and unused-import errors.

• Workers publish lifecycle events to an EventManager (Worker::with_event_manager) without the webhooks feature, so streaming-only builds and plain EventManager subscribers get them (#130). In 2.x this worked only because webhooks was a default feature.

• TestQueue::set_throttle_config rejects an enabled throttle with a zero rate, as the PostgreSQL and MySQL queues do (#130).

• DatabaseQueue::list_queue_names returns the names in byte (str) order without duplicates on every backend; PostgreSQL and MySQL sorted them by the database collation, so the two could disagree with each other and with TestQueue (#130).

• The MySQL key manager writes and compares key times with UTC_TIMESTAMP(6) instead of the session-local NOW(6), now that the columns are DATETIME(6) (#130).

Changed

• hammerwork-web no longer enables hammerwork's metrics feature, which it never used (the dashboard still reports prometheus_enabled: false).
• config/init-mysql.sql and config/init-postgres.sql (docker-compose) no longer create a hammerwork_jobs table and a seed job: the schema comes from cargo hammerwork migration run, so it always matches the library. The MySQL script only grants the user's permissions.
• CI checks the feature combinations full, postgres,full, mysql,full and test-queue.
• The library no longer depends on the rsa crate (RUSTSEC-2023-0071), which sqlx 0.9
  made optional. It stays in the lockfile only through the CLI and dashboard binaries.

Database

• Migration 021_mysql_datetime (#91): on MySQL, converts all 39 TIMESTAMP / TIMESTAMP(6) columns of hammerwork_jobs, hammerwork_jobs_archive, hammerwork_batches, hammerwork_encryption_keys, hammerwork_key_audit_log, hammerwork_kms_data_keys, hammerwork_queue_pause and hammerwork_migrations, and the 3 second-precision DATETIME columns of hammerwork_workflows (created_at, completed_at, failed_at), to DATETIME(6), keeping nullability, values and defaults (CURRENT_TIMESTAMP defaults become CURRENT_TIMESTAMP(6); hammerwork_queue_pause.updated_at keeps ON UPDATE). One ALTER TABLE per table, run under a UTC session time_zone, safe to re-run. A no-op on PostgreSQL.