Skip to content

Metrics with Prometheus

Jump to bottom
ptweezy edited this page Jul 8, 2026 · 2 revisions

Metrics with Prometheus

yacron2 natively exposes Prometheus metrics at GET /metrics on the HTTP web API. This is the pull-side sibling of the per-job statsd push metrics: a Prometheus server scrapes cumulative counters (run outcomes, durations, retries), live gauges (running instances, next scheduled run, last-run outcome), and the full cluster health surface on demand. The exposition is generated by yacron2 itself -- no exporter sidecar and no extra dependency, so it works on every architecture and platform yacron2 ships for, including Windows.

Both exposition formats are supported: the classic Prometheus text format (text/plain; version=0.0.4) by default, and OpenMetrics 1.0 (application/openmetrics-text) when the scraper advertises it in its Accept header, as modern Prometheus servers do.

Enabling the endpoint

The endpoint is on by default whenever the web API is enabled; a web.listen block is all it takes:

web:
  listen:
    - http://127.0.0.1:8080
$ curl http://127.0.0.1:8080/metrics
# HELP yacron2_info yacron2 build information.
# TYPE yacron2_info gauge
yacron2_info{version="1.0.13"} 1
...

The web.metrics option tunes or disables it. It accepts a boolean shorthand or a map:

web:
  listen:
    - http://127.0.0.1:8080
  # metrics: false            # shorthand: disable the endpoint entirely
  metrics:
    enabled: true             # default
    public: false             # default; see Authentication below
    durationBuckets:          # histogram bounds override (seconds)
      - 1
      - 10
      - 60
      - 600

Options

The schema for the block is Bool() | Map({Opt("enabled"): Bool(), Opt("public"): Bool(), Opt("durationBuckets"): Seq(Float())}) -- every key is optional.

Option Type Default Description
enabled bool true Serve GET /metrics. metrics: false / metrics: true are shorthands for {enabled: ...}.
public bool false Exempt /metrics (and only /metrics) from web.authToken bearer-token authentication, for scrapers that cannot send credentials.
durationBuckets list of float 0.1, 0.5, 1, 5, 15, 60, 300, 900, 3600 Upper bounds (seconds) of the yacron2_job_duration_seconds histogram. Must be finite, positive and strictly increasing; a +Inf bucket is always appended automatically. Changing the bounds on a reload restarts the histograms from zero (an ordinary counter reset to Prometheus).

Scraping and authentication

When web.authToken is configured, /metrics requires the bearer token like every other data endpoint. Prometheus supports this directly in its scrape configuration:

scrape_configs:
  - job_name: yacron2
    static_configs:
      - targets: ["cron-host:8080"]
    authorization:
      type: Bearer
      credentials: "the-secret-token"

If your scraper cannot send credentials, set web.metrics.public: true to exempt /metrics from authentication while everything else stays gated. The metrics reveal job names, schedules and outcomes (but never command lines, output, or secrets); treat public accordingly.

Two response details differ from the other endpoints:

  • web.headers are stamped onto the response as usual, except Content-Type, which is the exposition format's contract and always wins.
  • Content negotiation is on the Accept header: any request advertising application/openmetrics-text gets OpenMetrics (terminated by # EOF); everything else gets the text format.

Metric reference

All metrics are prefixed yacron2_. Per-job metrics carry the job name in a job_name label (not job, which would collide with Prometheus's own target label). Counters are cumulative since process start; they reset when the daemon restarts, which Prometheus's rate()/increase() handle natively (with a durable state store configured, the per-job counters are re-seeded across restarts; see Semantics and guarantees).

Daemon

Metric Type Description
yacron2_info{version} gauge (info) Build information; always 1.
yacron2_start_time_seconds gauge Unix time this yacron2 process started.
yacron2_job_set_info{job_set_id} gauge (info) The job-set fingerprint of the currently loaded configuration; always 1. Compare across a fleet to spot config drift.
yacron2_jobs{state} gauge Number of configured jobs, state="enabled" / "disabled".
yacron2_config_last_reload_successful gauge 1 if the most recent configuration parse succeeded, 0 if the config on disk is currently broken (the daemon keeps running the previous good config). Absent until the first parse.
yacron2_config_last_reload_success_timestamp_seconds gauge Unix time of the last successful configuration parse.

Per job

Metric Type Description
yacron2_job_runs_total{job_name, status} counter Finished runs by outcome: status="success" / "failure" / "cancelled", exactly as recorded in the run history. Zero-filled for every configured job from the first scrape. Runs replaced under concurrencyPolicy: Replace are not runs and are not counted, matching the dashboard.
yacron2_job_duration_seconds histogram Duration of finished runs (bucket bounds from durationBuckets). Cancelled-without-starting records carry no duration and are not observed.
yacron2_job_retries_total{job_name} counter Retry attempts actually launched (onFailure.retry); deferred or abandoned retries are not counted.
yacron2_job_permanent_failures_total{job_name} counter Failures with no retry remaining -- exactly when onPermanentFailure reporting fires.
yacron2_job_start_failures_total{job_name} counter Runs whose command could not be launched at all (also recorded as failures with the conventional exit code 127).
yacron2_job_last_success_timestamp_seconds{job_name} gauge Unix time of the last successful finish. Absent until the first success.
yacron2_job_last_failure_timestamp_seconds{job_name} gauge Unix time of the last failed finish. Absent until the first failure.
yacron2_job_info{job_name, schedule, cluster_policy} gauge (info) Static configuration facts; always 1.
yacron2_job_enabled{job_name} gauge 1 if the job is enabled in the loaded config.
yacron2_job_running{job_name} gauge Number of currently running instances (can exceed 1 under concurrencyPolicy: Allow).
yacron2_job_next_run_timestamp_seconds{job_name} gauge Unix time of the next scheduled run. Absent for disabled jobs and @reboot schedules.
yacron2_job_last_run_timestamp_seconds{job_name} gauge Unix time the most recent run finished.
yacron2_job_last_run_duration_seconds{job_name} gauge Duration of the most recent finished run.
yacron2_job_last_run_exit_code{job_name} gauge Exit code of the most recent finished run.
yacron2_job_last_run_success{job_name} gauge 1 if the most recent finished run succeeded, else 0 (a cancelled run counts as 0).
yacron2_job_cpu_seconds_total{job_name, mode} counter CPU time consumed by monitorResources runs, mode="user" / "system", summed over the job's monitored runs. Emitted only once the job has a monitored run (an unmonitored job exports no CPU series).
yacron2_job_peak_rss_bytes{job_name} gauge Highest resident-set size (bytes) observed across all of the job's monitored runs. Absent until the first monitored run.
yacron2_job_last_run_cpu_seconds{job_name} gauge Total CPU time of the most recent monitored run. Absent unless that run had monitorResources on.
yacron2_job_last_run_max_rss_bytes{job_name} gauge Peak resident-set size (bytes) of the most recent monitored run. Absent unless that run had monitorResources on.

The four resource families above require the job to opt into monitorResources; they are sampled and best-effort. Like the run counters, yacron2_job_cpu_seconds_total and yacron2_job_peak_rss_bytes are made restart-durable by a state: store (see Semantics and guarantees).

State backend

Emitted when a durable state store is configured (a state: section); without one yacron2 is stateless and these families are absent. They observe the store itself -- the job families above are unchanged, except that the store makes the per-job counters restart-durable (see Semantics and guarantees). Families and label values appear as they first become relevant, so a store that never contends a lock or is never throttled does not export frozen zero series forever.

Metric Type Description
yacron2_state_info{backend, topology} gauge (info) Static store facts: the backend implementation (filesystem) and the effective topology (single-node / shared, after topology: auto resolves); always 1.
yacron2_state_ops_total{op} counter Store operations by kind: op="append" / "list" / "derive-max" / "prune" / "lease-acquire" / "lease-renew" / "lease-release" / "lease-read" / "start" / "gc". Each op value appears once that operation has first run.
yacron2_state_op_errors_total{op} counter Store operations that raised, same op label.
yacron2_state_op_seconds_total{op} counter Seconds spent inside store operations; divide by yacron2_state_ops_total (e.g. rate() over rate()) for mean in-store latency per operation.
yacron2_state_lock_acquisitions_total counter Advisory-lock acquisitions by the store. Emitted once nonzero.
yacron2_state_lock_wait_seconds_total counter Seconds spent waiting to acquire the store's advisory locks -- the lock-contention signal. Emitted alongside yacron2_state_lock_acquisitions_total.
yacron2_state_throttled_ops_total counter Store operations delayed by the state.maxOpsPerSecond token bucket. Emitted once nonzero.
yacron2_state_throttle_wait_seconds_total counter Seconds those operations spent queued behind the rate limit. Emitted alongside yacron2_state_throttled_ops_total.
yacron2_state_dropped_writes_total{kind} counter Durable writes that failed and were dropped (with a logged warning) -- writes are fire-and-forget under either onStoreUnavailable policy -- by kind="run-record" / "checkpoint" / "retry" / "reboot-marker" / "counters" / "manifest". Emitted once nonzero -- the single most alertable state signal.

If reading the store's stats fails during a scrape, the state families are omitted from that scrape (with the error logged) rather than failing it -- the same degradation contract as the cluster block below. yacron2_state_dropped_writes_total is the exception: it is accumulated by the scheduler as it drops writes, not read from the store, so it keeps being served even when the store itself cannot be read.

Cluster

Emitted when a cluster is configured; the values mirror the pre-derived fields of GET /cluster, so the same alert rules documented there apply.

Metric Type Description
yacron2_cluster_enabled gauge 1 when a leadership backend is running on this node (0 also covers a configured backend that failed to start).
yacron2_cluster_info{backend, node_name, distribution} gauge (info) Backend (gossip/kubernetes/etcd), this node's name, and the ownership distribution; always 1.
yacron2_cluster_size / yacron2_cluster_quorum gauge Effective cluster size N and the quorum threshold (lease backends report 1/1).
yacron2_cluster_quorate gauge Whether this node is part of a quorum -- the single most alertable cluster signal.
yacron2_cluster_is_leader gauge Whether this node holds scheduled-job leadership. Always 0 under distribution: spread (ownership is per job there), mirroring GET /cluster.
yacron2_cluster_leader_info{leader} gauge (info) The observed leader's name; absent when there is none.
yacron2_cluster_conflict{kind} gauge The three fail-closed conflict gates, kind="nodename" / "size" / "policy". Any 1 means Leader jobs are standing down cluster-wide.
yacron2_cluster_peers{status} gauge Gossip backend only: configured peers by observed status (agreed, syncing, drifted, unreachable, untrusted, conflict, self, unknown), zero-filled so alert series never vanish.
yacron2_cluster_leader_transitions_total counter Times this node acquired or lost leadership. Observed at scheduler cadence (once per minute), so a flap shorter than a scheduler tick may be missed. Emitted only when electLeader is on (an observe-only cluster has no leadership to transition).
yacron2_cluster_quorum_transitions_total counter Times this node joined or left quorum (same cadence and electLeader caveats).

If a backend read fails during a scrape, the cluster block degrades to yacron2_cluster_enabled alone (with the error logged by the prometheus logger) rather than failing the whole scrape.

Semantics and guarantees

  • Metrics never affect scheduling. Every hook is a synchronous in-memory increment on the scheduler's event loop; there is nothing to time out, retry, or crash on. This is the same contract the statsd writer has, without the UDP.
  • Counters agree with the API. yacron2_job_runs_total increments at the exact point a run enters the run history (GET /jobs/{name}/runs), so the two surfaces can never disagree on outcomes.
  • Counters survive reloads -- and, with durable state, restarts. The accumulators live on the daemon (not the web app), so a config reload -- even one that restarts the web server or the cluster manager -- keeps them. Without a state: section a process restart resets them, which is normal for Prometheus counters. With a durable state store configured, the per-job counters (runs by outcome, retries, permanent and start failures, the duration histogram, and the last success/failure timestamps) are snapshotted durably -- at most one write per 15 seconds while running, plus a final snapshot on clean shutdown -- and seeded back on boot for the jobs still in the config (histogram state only if the durationBuckets bounds are unchanged). A restart then reads as, at worst, a small ordinary counter reset covering the events since the last snapshot; a crash forfeits at most those last few seconds. yacron2_start_time_seconds still resets per process by design -- that is how you see the restart itself.
  • Removed jobs disappear. On every successful reload, series for jobs no longer in the config are dropped; a renamed job starts over at zero.
  • @reboot keep-alives look "always running". A long-running @reboot job shows yacron2_job_running 1 with zero finished runs -- that is the correct reading, not missing data.

Example alerts

groups:
  - name: yacron2
    rules:
      # a job's most recent run failed
      - alert: YacronJobFailed
        expr: yacron2_job_last_run_success == 0
        for: 5m
      # a job that should run hourly has not succeeded for 2 hours
      - alert: YacronJobStale
        expr: time() - yacron2_job_last_success_timestamp_seconds > 7200
      # the config file on disk is broken; the daemon is running stale config
      - alert: YacronConfigBroken
        expr: yacron2_config_last_reload_successful == 0
        for: 10m
      # the cluster cannot elect a leader: Leader jobs are not running anywhere
      - alert: YacronClusterNotQuorate
        expr: yacron2_cluster_quorate == 0
        for: 5m
      # more than one node believes it is the leader (split-brain)
      - alert: YacronMultipleLeaders
        expr: sum(yacron2_cluster_is_leader) > 1

The Clustering monitoring guide discusses the cluster signals in depth.

See also

This wiki documents yacron2. See the README and the changelog.

yacron2 is a fork of gjcarneiro/yacron.

yacron2 Wiki

Clone this wiki locally