v2.5.0

A bug-fix release from a third bug hunt (#111 to #114, #118). It is a minor version
because it adds a database migration (020; run it before upgrading workers or archivers,
and off-peak on MySQL, where it rebuilds the archive table) and new public items
(job::MAX_JOB_DEPENDENCIES, job::MAX_JOB_DEPENDENTS, Job::validate_dependencies,
and the defaulted DatabaseQueue::list_queue_names and
DatabaseQueue::purge_archived_jobs_in_queue), and because several fixes change
observable behaviour (see below). No public signatures change.

Behaviour changes to check when upgrading

• cargo hammerwork batch cancel cancels only Pending jobs by default. It used
  to delete Running jobs too, out from under their workers; pass --status running
  to cancel those.
• cargo hammerwork maintenance dead-jobs marks a Running job dead only when it
  was abandoned, by the rule of the library's stale-job recovery: its lease expired, or
  it holds no lease and started more than --stale-hours ago. A long-running job that
  keeps heartbeating is no longer marked dead.
• "Failed" includes TimedOut everywhere in the CLI (one shared definition: status
  Failed, Dead or TimedOut, aged by COALESCE(failed_at, timed_out_at)). queue health counts timed-out jobs as recent failures, job purge --failed deletes them
  (--dead still means Dead only), and batch retry retries them by default and
  accepts --status timed_out.
• webhook add/update --max-retries N now means N retries after the first delivery
  (stored as N + 1 attempts); it used to give N - 1 retries. Webhooks added before keep
  their stored attempts and webhook list shows attempts - 1 as their retries.
• Dashboard queue routes (/api/queues/{name}, .../actions, .../jobs)
  percent-decode the queue name, so email%20jobs is the queue email jobs; a name that
  is not valid UTF-8 once decoded is a 400.
• POST /api/jobs/search reads jobs in every status (failed, timed out, completed,
  running, retrying and jobs scheduled for later, up to 200 per status and queue), not just
  the ready, dead and recurring ones. statuses accepts the same names as the jobs
  listing (failed covers Failed, Dead and TimedOut). The route needs T: JobHistory.
• GET /api/system/metrics: the dashboard serves no /metrics route. The
  metrics_endpoint field is kept for compatibility but documented as not served;
  custom_metrics_count and last_scrape are always null.
• WebhookManager::shutdown and StreamManager::shutdown unsubscribe their listeners
  from the event manager, so EventManager::subscription_count and payload_wanted stop
  counting them. Dropping an EventSubscription does not unregister it: call
  EventManager::unsubscribe(id) when done (now documented).
• Webhook statistics: successful_deliveries, failed_deliveries and success_rate
  count events by final outcome (an event delivered on its third attempt is one successful
  delivery, not two failures and a success); total_attempts still counts every HTTP
  attempt, retries included.
• EventFilter processing-time range rejects events that have no processing time when
  a minimum or maximum is set (it used to let them through).
• HammerworkConfig::save_to_file writes the file with mode 0600 on Unix (through a
  temporary file renamed over the target); from_file validates the [encryption] section
  like JobQueue::from_config.
• Streaming Hash partitioner uses a stable FNV-1a hash, so partition keys change
  (they were a character sum, which collided for anagrams).
• RetryStrategy::Linear starts at base. The first retry (calculate_delay(1))
  now waits base, then base + increment, base + 2 × increment and so on
  (base + (attempt - 1) × increment), like Fixed, Exponential and Fibonacci
  since 2.3.0 and as the variant's documentation said. It used to wait
  base + increment first, so every linear delay is now one increment shorter.
• A job can depend on at most 1000 jobs on every backend. enqueue,
  enqueue_batch, enqueue_workflow, enqueue_cron_job and spawned children
  (complete_job_run_with_children) refuse a job with more (or with more than 10000
  cached dependents) with HammerworkError::Workflow, before writing anything.
  PostgreSQL already refused them through a CHECK constraint (with a database error);
  MySQL and TestQueue accepted them. JobBatch::validate and JobGroup::validate
  run the same check; it is public as Job::validate_dependencies, with the limits in
  job::MAX_JOB_DEPENDENCIES and job::MAX_JOB_DEPENDENTS.
• get_workflow_status computes completed_jobs, failed_jobs and status from the
  workflow's jobs (archived ones included) when it is read, as get_batch_status
  does, instead of returning the stored counters. A workflow left running by the
  archive bug below reads as finished. get_workflow_jobs (and JobGroup::jobs)
  also lists archived members, with status Archived.
• Spawned children count towards their workflow. A workflow whose job spawns
  children now has a larger total_jobs and completes after them (see Fixed).
• JobArchiver::run_scheduled_pass applies per-queue policies (set_policy): each
  queue is archived and purged with its own policy, every other queue with the default
  one, and a disabled policy leaves its queues alone. A pass with per-queue policies
  works through the queues one at a time, using the new (defaulted) DatabaseQueue
  methods list_queue_names and purge_archived_jobs_in_queue.
  WorkerPool::with_archival (a single policy) is unchanged.
• TestQueue matches the database backends in four places (see Fixed), so tests
  written against it may need their expectations updated: get_batch_status reports
  Pending for a batch none of whose jobs has finished (it said Processing) and
  Failed when every job failed (it said PartiallyFailed), with error_summary worded
  "N of M jobs failed"; get_ready_jobs returns jobs by priority, then scheduled_at;
  and a restored job starts over with no attempts. (#112)

Database

• Migration 020 (020_archive_retry_strategy_and_workflow_index) indexes
  hammerwork_jobs_archive(workflow_id) on PostgreSQL and MySQL. On MySQL it also
  widens hammerwork_jobs_archive.retry_strategy from VARCHAR(50) to TEXT and turns
  the archive's timestamps into TIMESTAMP(6), like hammerwork_jobs. It is idempotent.
  Run it before upgrading: on MySQL, archiving a job with a retry strategy needs it.
  On MySQL the column changes rebuild hammerwork_jobs_archive, which takes a while on a
  large archive; run it off-peak (purge old archived jobs first to make it quicker).
• Migration 012 (PostgreSQL) no longer runs its own BEGIN;/COMMIT;: the runner
  already wraps each migration in a transaction, and the script's COMMIT ended it
  early. MySQL migrations 010, 018 and 019 no longer use double-quoted string literals,
  which ANSI_QUOTES reads as identifiers. Re-running them is unaffected.

Fixed

• Archiving measures job age on the database clock. archive_jobs compared the
  database-written completed_at/failed_at/timed_out_at against a cutoff taken from
  the application clock, so clock skew could archive jobs early or late. The cutoff is now
  read from the database inside the archive transaction (#118).
• config set and config reset no longer write environment variables into
  config.toml. They used to save the merged configuration, copying DATABASE_URL
  (with its password) and the HAMMERWORK_* variables into the file. They now change
  the file alone, and write it through a temp file with mode 0600 renamed into place.
• config set refuses to overwrite a config file that cannot be parsed (it used to
  replace it with the defaults plus the environment). Fix or remove the file, or run
  config reset --confirm.
• config show's Source column tells Environment, Config File and Default
  apart for every setting (it said "Config File" for environment values and defaults).
• archive run --dry-run reports how many jobs match the policy, using the same
  predicate as the real run, instead of only echoing the flags.
• webhook: the webhook file is created with mode 0600 from the start, not
  chmod-ed after it was written.
• backup restore --skip-existing is hidden (it is a no-op: existing jobs are always
  skipped), and archive resolves the database URL like the other commands.
• A deserialized CronSchedule is rebuilt from its expression and timezone, so it fires
  (it used to have no schedule and never matched); an invalid expression or timezone is a
  deserialization error.
• Concurrent checks of the same alert reserve the cooldown under the write lock, so the
  alert is sent once instead of once per concurrent caller.
• The streaming Hash partitioner no longer panics on overflow in debug builds.
• MySQL archival stopped for good on jobs with a retry strategy. The archive's
  retry_strategy VARCHAR(50) column was too short for nearly every serialized
  strategy, so strict MySQL rejected the insert (error 1406), the batch rolled back, and
  since candidates are taken oldest first every later pass, including
  WorkerPool::with_archival, failed on the same jobs. Migration 020 widens it. (#111)
• MySQL archive timestamps were rounded to the second. (#111)
• A workflow never completed once one of its finished jobs was archived: its
  progress counted only hammerwork_jobs. Workflow counters and status (every terminal
  transition, get_workflow_status) now count archived member jobs with the status they
  were archived with, on PostgreSQL, MySQL and TestQueue. (#111)
• Children spawned by a workflow job joined the workflow without raising its
  total_jobs, so the workflow could complete before they ran, and a child's later
  failure never failed it. complete_job_run_with_children now raises the total of each
  workflow its children join, in the same transaction, on both backends and
  TestQueue. (#111)
• The dependency limit differed between backends (see the behaviour change
  above). (#111)
• JobArchiver::run_scheduled_pass ignored per-queue policies although
  set_default_policy documents that the default applies only to queues without a
  policy of their own. (#111)
• RetryStrategy::Linear documentation and behaviour disagreed (see the behaviour
  change above). (#111)
• Cleanups: JobStatus::Retrying is documented as never written by Hammerwork, the
  batch-counter comment in both backends is accurate again, get_priority_stats decodes
  priorities with JobPriority::from_i32, and a worker no longer reads a batch's status
  twice after a job's final failure. (#111)
• TestQueue diverged from PostgreSQL and MySQL (#112):
  • restore_archived_job only set the job back to Pending; it now resets attempts,
    times, error and result like the backends (scheduled at the mock clock's now), so a
    restored dead job no longer dies after one run.
  • get_batch_status tallies with the backends' rule and counts archived members with
    the status they were archived with (it ignored them).
  • get_ready_jobs returned jobs still waiting on their dependencies (outside
    workflows) and in no particular order; it now filters on dependency_status
    (none or satisfied) and orders by priority, then scheduled_at.
  • cancel_workflow skipped Retrying jobs, left the dependency status of waiting
    jobs at waiting, kept their leases and overwrote the workflow's failed_at; it
    now fails every unfinished job as the backends do (waiting becomes failed) and
    keeps an earlier failed_at.