Failure Detection and Retries

Failure Detection and Retries

This page documents how yacron2 decides whether a job run failed (failsWhen), the exact precedence order of failure reasons, the retry mechanism with exponential backoff (onFailure.retry), and when each of the three report hooks (onFailure, onPermanentFailure, onSuccess) fires.

Overview

After a job process exits, yacron2 computes a single failure reason from the run's exit code and captured output. If the reason is non-empty the run is failed; otherwise it succeeded. Failure triggers onFailure reporting and, if a retry is configured and not yet exhausted, schedules another run after a backoff delay. When retries are exhausted (or none was configured) onPermanentFailure reporting fires. Success cancels any pending retry and fires onSuccess reporting.

Determining failure: failsWhen

failsWhen is a per-job (or per-defaults) block of four booleans. It is evaluated by RunningJob.fail_reason (yacron2/job.py) after the process exits and its streams have been read.

Option	Type	Default	Description
producesStdout	Bool	false	If true, any captured standard output marks the run as failed.
producesStderr	Bool	true	If true, any captured standard error marks the run as failed.
nonzeroReturn	Bool	true	If true, an exit code other than 0 marks the run as failed.
always	Bool	false	If true, the run is always considered failed regardless of exit code or output.

In the strictyaml schema (yacron2/config.py), only producesStdout is required within a failsWhen map; producesStderr, nonzeroReturn, and always are Opt(...). Defaults come from DEFAULT_CONFIG["failsWhen"] and are merged in before a failsWhen block is applied, so a partial failsWhen block inherits the defaults for the keys it omits.

Output detection considers both retained and discarded lines. A stream is treated as non-empty if it has saved content or if any lines were discarded (saveLimit exhausted, or saveLimit: 0). See Output Capturing for how captureStdout/captureStderr and saveLimit govern what is captured. If a stream is not captured, it cannot produce a failure reason: producesStderr only fires when captureStderr is enabled, and producesStdout only fires when captureStdout is enabled.

Precedence order

fail_reason returns the first matching condition, in this fixed order, and None if none match:

1. always is true -> "failsWhen=always".
2. nonzeroReturn is true and retcode != 0 -> "failsWhen=nonzeroReturn and retcode={retcode}".
3. producesStdout is true and stdout is non-empty or any stdout lines were discarded -> "failsWhen=producesStdout and stdout is not empty".
4. producesStderr is true and stderr is non-empty or any stderr lines were discarded -> "failsWhen=producesStderr and stderr is not empty".

The first match wins; later conditions are not evaluated. The resulting string is exposed to report templates as the fail_reason variable and to the shell reporter as YACRON2_FAIL_REASON. The boolean failed is simply fail_reason is not None.

Special exit codes

Two synthetic exit codes are set by the runtime rather than the child process:

• 127: the subprocess could not be launched at all (e.g. the command does not exist, or the argv could not be encoded). RunningJob sets start_failed and, in wait(), assigns retcode = 127 so the run is treated as an ordinary failure rather than raising an internal error. With the default nonzeroReturn: true, this is a failure.
• -100: the run exceeded its executionTimeout and was cancelled. wait() sets retcode = -100 before terminating the process. With the default nonzeroReturn: true, this is a failure. See Concurrency and Timeouts.

A run cancelled to make way for a newer instance (concurrencyPolicy: Replace) is marked replaced and is not evaluated for failure, reported, or retried.

Example

jobs:
  - name: strict-job
    command: ./run.sh
    schedule: "*/5 * * * *"
    captureStdout: true
    captureStderr: true
    failsWhen:
      producesStdout: false
      producesStderr: true
      nonzeroReturn: true
      always: false

Retries: onFailure.retry

Retries are configured under onFailure.retry. Retry orchestration lives in yacron2/cron.py (launch_scheduled_job, handle_job_failure, schedule_retry_job, cancel_job_retries); per-job backoff state is JobRetryState in yacron2/job.py. Retry state is in-memory by default, so a pending retry dies with the process; with a state: section configured it also survives daemon restarts (see "Restart-surviving retries" below), and on a shared store under leader election a ladder can even move to the node that now owns the job (see "Cross-node retry resume" below).

Option	Type	Default	Description
maximumRetries	Int	0	Number of retries after the initial failed run. 0 disables retrying. -1 retries forever.
initialDelay	Float	1	Delay in seconds before the first retry.
maximumDelay	Float	300	Upper bound in seconds on the backoff delay.
backoffMultiplier	Float	2	Factor the delay is multiplied by after each retry.

In the schema, all four keys are required within a retry map (no Opt(...)), but the entire retry map is optional and the DEFAULT_CONFIG values above are merged in, so a job that omits retry entirely gets these defaults. If a retry map is given, strictyaml requires all four keys to be present (a partial retry block is a validation error). Numeric ranges are validated in JobConfig._validate_numeric_ranges and raise ConfigError on violation:

• maximumRetries >= -1
• initialDelay >= 0
• maximumDelay > 0
• backoffMultiplier > 0

Exponential backoff

JobRetryState.next_delay() returns the current delay, then advances it for the next retry:

delay      = current delay (returned, used to sleep)
next delay = min(current delay * backoffMultiplier, maximumDelay)

The first retry waits initialDelay; each subsequent retry waits the previous delay times backoffMultiplier, capped at maximumDelay. With initialDelay: 1, backoffMultiplier: 2, maximumDelay: 30, the delay sequence is 1, 2, 4, 8, 16, 30, 30, ... seconds. The retry counter (count) increments on each next_delay() call.

The ladder is a pure function of the retry config and the attempt number, which is what lets a durable pending retry be re-armed at its exact position after a daemon restart (see "Restart-surviving retries" below).

Retry lifecycle

• A retry state is created only when maximumRetries is truthy (non-zero). With maximumRetries: 0 no state is created and a failed run goes straight to permanent failure.
• launch_scheduled_job calls cancel_job_retries(name) before starting a scheduled run, then creates a fresh JobRetryState. A scheduled run therefore resets any in-progress retry sequence for that job. A manually triggered run (POST /jobs/{name}/start, see HTTP Control API) goes through maybe_launch_job directly and does not reset or create retry state; it reuses whatever retry state currently exists.
• On each failed run, handle_job_failure fires onFailure reporting, then: if no retry state exists or it was cancelled, fires onPermanentFailure and stops; otherwise, if count >= maximumRetries and maximumRetries != -1, cancels the retry state and fires onPermanentFailure; otherwise schedules the next retry after next_delay() seconds.
• A success (handle_job_success) calls cancel_job_retries and fires onSuccess, ending the sequence.
• If a job is removed from the configuration while a retry is pending, schedule_retry_job logs a warning, discards the stale retry state, and skips the run cleanly (no exception).
• When leader election is enabled (cluster.electLeader), schedule_retry_job re-checks the cluster gate before relaunching. A transient fail-closed condition (lost quorum, a detected conflict, a rebuilt gossip manager's still-converging view, a backend read error) does not end the sequence: the retry state is kept and the gate is re-checked after another delay of the same length (floored at one second; the first deferral of a wait is logged at INFO, repeats at DEBUG), so a keep-alive job survives the blip. Only when another node is positively identified as the job's owner does the pending retry leave this node, and what happens then depends on the store. When cross-node retry resume is active (a shared-topology state store under leader election, see "Cross-node retry resume" below) the ladder is handed off: the local retry state is cancelled, a handoff record supersedes the durable pending one, a WARNING is logged, and no cancelled run-history record is written -- the attempt is not ending, it is moving, and the new owner resumes the same attempt from its durable record. Otherwise the pending retry is abandoned: the retry state is cancelled and discarded, a WARNING is logged, the abandonment is recorded in the run history as cancelled, and the failed attempt is not re-run elsewhere: the new owner only picks up the job's future scheduled firings. Neither path fires onPermanentFailure. An @reboot ladder is anchored to its host's boot and is never handed off, and an @reboot one-shot has no future firing either (its boot run is already recorded), so an abandoned @reboot keep-alive ends cluster-wide even when resume is active; EveryNode ladders stay strictly per-node. See Clustering and Leader Election.
• On shutdown, all pending retries are cancelled before yacron2 exits. Without a state: section that ends the sequence for good; with one, the graceful-shutdown cancellation deliberately does not settle the durable pending record, so the next start re-arms it (see below).

Restart-surviving retries

Everything in this subsection activates only when a state: config section is present (the durable state store). Without one, the lifecycle above is the whole story and a pending retry dies with the process. (The store is server-side, on state.path; it is unrelated to the web dashboard's browser-side IndexedDB run ledger.)

With state: configured, every job with a non-zero maximumRetries gets a durable retry ladder alongside the in-memory one:

• When a retry is armed, a pending record is appended (fire-and-forget) to the job's durable retry stream, carrying the attempt number, the absolute notBefore deadline, and the job's per-job config digest (yacron2.fingerprint.job_digest). A write that never lands loses only the durability: the retry dies with the process, exactly the stateless behaviour. No durable record is written for a ladder that never scheduled a retry, so a retry-armed job that keeps succeeding costs no store writes.
• Every way a ladder can resolve appends a settled record on top, so the next boot finds nothing pending. The settle reasons are launched (the settle-before-launch write below), succeeded (the run succeeded), superseded (a fresh scheduled fire reset the sequence), cancelled (e.g. a run cancelled from the dashboard), exhausted (maximumRetries reached), owner-moved (the cluster abandonment described above -- on a shared store the ladder is handed off instead of settled, see "Cross-node retry resume" below), superseded-by-run (a claim scan found a durable run newer than the ladder), and job-removed (the job disappeared from a reloaded config while the retry slept), plus the boot-time invalidation reasons below.
• Just before a retry launches, its pending record is settled with reason launched -- record-before-run, so a crash right after the launch cannot re-arm the attempt that already ran. This is the at-most-once bias. The one caveat: under the default onStoreUnavailable: degrade, a settle write that cannot land launches anyway (at-least-once -- a crash in that narrow window could replay that one attempt after a restart); under onStoreUnavailable: fail-closed the launch is deferred and re-checked instead, exactly like a closed cluster gate.
• A graceful shutdown does not settle: the shutdown drain cancels the in-process retry tasks but leaves the pending record on top of the stream, and that record is exactly what the next boot re-arms.
• On boot, a job whose newest retry record is pending has its ladder re-armed at the persisted position: the retry counter and the next backoff delay are replayed to the recorded attempt, and the task sleeps only the time remaining until the absolute notBefore deadline -- zero if it passed while the daemon was down, in which case the retry is due immediately. The re-armed task is the ordinary schedule_retry_job, so the cluster-gate re-check, job-removed cleanup, and shutdown behaviour are identical to a never-restarted ladder. Live activity outranks the ledger: a job already retrying or running when the store comes up is left alone.
• A pending record is settled instead of re-armed (invalidation) when: the job's per-job config digest changed -- the digest is per-job, stricter than the whole-set job-set id, so editing an unrelated job does not drop the retry; the job was removed or disabled; the recorded attempt already exhausts maximumRetries; the record is older than the job's startingDeadlineSeconds (when set); or, for an @reboot job, the machine actually rebooted, so the fresh boot run supersedes the stale ladder. Any ambiguous case also settles: the bias is always no-run over double-run.
• @reboot keep-alive continuity: an @reboot job with maximumRetries: -1 whose durable boot marker shows its boot run already happened during this OS boot gets its pending retry re-armed rather than a fresh boot run, so a keep-alive "supervisor" (see below) survives daemon restarts.

Cross-node retry resume

Restart-surviving retries have a cluster-wide half: when the state store's resolved topology is shared (see Durable State), leader election is configured (cluster.electLeader), and the cluster manager is running, a durable retry ladder can move to the node that now owns the job instead of ending with the old one. Only Leader/PreferLeader ladders that are not @reboot are eligible: EveryNode ladders stay strictly per-node (a foreign pending on the shared stream is another node's live ladder), and an @reboot ladder is anchored to its host's boot, so it never moves. This is the operator's-altitude view; the record-level mechanics live in Durable State's "Restart-surviving retries".

• Graceful moves hand off. When the owning node itself observes the ownership move (the abandonment check under "Retry lifecycle" above), it appends a handoff record carrying the attempt, the job digest, and a now-due deadline instead of settling the ladder dead, and writes no cancelled run-history record on this path: the attempt is moving, not ending. The WARNING reads "handed off: the cluster moved ownership of it to another node; the new owner resumes the ladder from its durable record (cross-node retry resume)".
• Crashed owners are claimed after a grace. A crashed owner simply leaves its pending record newest on the stream. The new owner's claim scan (spawned from the housekeeping pass about once a minute) claims a handoff immediately (the owner positively relinquished), but a foreign pending only once it is stale 30 seconds past due: a live owner fires within moments of its deadline, so the grace only tolerates a slightly late fire. It cannot cover a gate-deferred owner, whose re-check cadence is its own ladder delay; the consume-time re-check below is what protects that case.
• Claims serialize on a lease. A claim validates the record (digest match, job enabled, retry budget, startingDeadlineSeconds, no locally-known newer run), acquires the per-job retry-claim/<name> lease (TTL 30 seconds), re-reads the newest record under the lease (it must be unchanged), and checks the durable run ledger for a run newer than the ladder -- a newer run settles the record superseded-by-run instead of claiming it (no-run beats double-run). Only then does the claimer append its own pending, wait for that write to land before releasing the lease, and re-arm the local ladder exactly as a restart would: absolute deadline, only the remaining delay is slept. The claim is logged at INFO: "claimed pending retry #N from host H (cross-node retry resume); due in S seconds".
• The consume re-checks ownership. With resume active, a due retry's launch decision serializes on the same claim lease and re-checks that the newest ladder record still belongs to this host. A foreign newest record (a claimer's pending, or its launched settle after it already fired) aborts the local ladder silently -- no settle is written, so the claimer's record stays newest -- with a WARNING: "dropped: another node claimed this retry ladder (cross-node retry resume); it fires there". Read or acquire failures follow onStoreUnavailable: degrade proceeds unserialized, fail-closed defers the launch.
• The honest contract is at-least-once, not exactly-once: an owner that is gate-deferred or cut off from the store can still fire an attempt a claimer also fires. Record ordering (newest-wins) and lease expiry compare wall clocks across hosts, so the clock-skew requirement on Durable State (run NTP on every node sharing the mount) covers resume too. In a mixed-version fleet, older builds treat the unknown handoff kind as not-pending and skip it, which is safe.

Retry example

jobs:
  - name: flaky-job
    command: ./flaky.sh
    schedule: "*/10 * * * *"
    captureStderr: true
    onFailure:
      report:
        mail:
          from: cron@example.com
          to: ops@example.com
          smtpHost: 127.0.0.1
      retry:
        maximumRetries: 10
        initialDelay: 1
        maximumDelay: 30
        backoffMultiplier: 2

Restart a long-running process

A schedule of @reboot runs the job once at yacron2 startup. Combined with maximumRetries: -1, this re-launches the process whenever it exits with a failure, indefinitely: a way to keep a long-running process alive under yacron2.

jobs:
  - name: keep-alive
    command: ./long-running-server
    schedule: "@reboot"
    onFailure:
      retry:
        maximumRetries: -1
        initialDelay: 1
        maximumDelay: 30
        backoffMultiplier: 2

By default the keep-alive lasts only as long as the yacron2 process: a daemon restart runs the @reboot job afresh and any pending retry is lost. With a state: section configured, both halves become durable: the boot run is deduplicated to once per OS boot, and a pending retry is re-armed across daemon restarts, so the supervisor pattern survives them (see "Restart-surviving retries" above).

See Schedules and Timezones for @reboot semantics.

Report hooks

Each hook has its own independent report block (Sentry, mail, shell, webhook), defaulted from _REPORT_DEFAULTS (deep-copied per hook so they do not alias). All four reporters in a block run for the relevant outcome; reporting errors are logged and do not abort the others. See Reporting (Mail, Sentry, Shell, Webhook) for the report block options.

Hook	Fires when	Frequency
onFailure.report	Every failed run.	Once per failed attempt (including each retry that fails).
onPermanentFailure.report	Retries are exhausted, or no retry was configured, or the retry state was cancelled.	Once, at the end of a failing sequence.
onSuccess.report	The run succeeded (fail_reason is None).	Once per successful run.

With no retry configured, a single failed run fires both onFailure.report (always) and then onPermanentFailure.report (because there is no retry state). To report only after all retries are exhausted, leave onFailure.report empty and configure onPermanentFailure.report instead, as in the example below.

jobs:
  - name: eventually-consistent
    command: ./run.sh
    schedule: "*/10 * * * *"
    captureStderr: true
    onFailure:
      retry:
        maximumRetries: 10
        initialDelay: 1
        maximumDelay: 30
        backoffMultiplier: 2
    onPermanentFailure:
      report:
        mail:
          from: cron@example.com
          to: ops@example.com
          smtpHost: 127.0.0.1

A note on mail reporting: an onSuccess mail whose rendered body is empty (after strip()) is suppressed (no email is sent). This applies only to success reports.

Notes

• nonzeroReturn checks retcode != 0, so both the synthetic 127 (launch failure) and -100 (timeout) codes count as non-zero failures under the default.
• The failsWhen evaluation runs once per completed run, including each retried run, so a retry that still produces stderr (with producesStderr: true) fails again and continues the backoff sequence.
• Output-based failure (producesStdout/producesStderr) depends on stream capturing; without captureStdout/captureStderr the corresponding condition can never trigger because nothing is captured.
• During shutdown, handle_job_failure returns early if the stop event is set: a job that finishes failing while yacron2 is shutting down is not reported (onFailure/onPermanentFailure do not fire) and is not retried. A job that finishes successfully during shutdown still cancels its retries and fires onSuccess.