Metrics with Prometheus

Metrics with Prometheus

yacron2 natively exposes Prometheus metrics at GET /metrics on the HTTP web API. This is the pull-side sibling of the per-job statsd push metrics: a Prometheus server scrapes cumulative counters (run outcomes, durations, retries), live gauges (running instances, next scheduled run, last-run outcome), and the full cluster health surface on demand. The exposition is generated by yacron2 itself -- no exporter sidecar and no extra dependency, so it works on every architecture and platform yacron2 ships for, including Windows.

Both exposition formats are supported: the classic Prometheus text format (text/plain; version=0.0.4) by default, and OpenMetrics 1.0 (application/openmetrics-text) when the scraper advertises it in its Accept header, as modern Prometheus servers do.

Enabling the endpoint

The endpoint is on by default whenever the web API is enabled; a web.listen block is all it takes:

web:
  listen:
    - http://127.0.0.1:8080

$ curl http://127.0.0.1:8080/metrics
# HELP yacron2_info yacron2 build information.
# TYPE yacron2_info gauge
yacron2_info{version="1.0.13"} 1
...

The web.metrics option tunes or disables it. It accepts a boolean shorthand or a map:

web:
  listen:
    - http://127.0.0.1:8080
  # metrics: false            # shorthand: disable the endpoint entirely
  metrics:
    enabled: true             # default
    public: false             # default; see Authentication below
    durationBuckets:          # histogram bounds override (seconds)
      - 1
      - 10
      - 60
      - 600

Options

The schema for the block is Bool() | Map({Opt("enabled"): Bool(), Opt("public"): Bool(), Opt("durationBuckets"): Seq(Float())}) -- every key is optional.

Option	Type	Default	Description
enabled	bool	true	Serve GET /metrics. metrics: false / metrics: true are shorthands for {enabled: ...}.
public	bool	false	Exempt /metrics (and only /metrics) from web.authToken bearer-token authentication, for scrapers that cannot send credentials.
durationBuckets	list of float	0.1, 0.5, 1, 5, 15, 60, 300, 900, 3600	Upper bounds (seconds) of the yacron2_job_duration_seconds histogram. Must be finite, positive and strictly increasing; a +Inf bucket is always appended automatically. Changing the bounds on a reload restarts the histograms from zero (an ordinary counter reset to Prometheus).

Scraping and authentication

When web.authToken is configured, /metrics requires the bearer token like every other data endpoint. Prometheus supports this directly in its scrape configuration:

scrape_configs:
  - job_name: yacron2
    static_configs:
      - targets: ["cron-host:8080"]
    authorization:
      type: Bearer
      credentials: "the-secret-token"

If your scraper cannot send credentials, set web.metrics.public: true to exempt /metrics from authentication while everything else stays gated. The metrics reveal job names, schedules and outcomes (but never command lines, output, or secrets); treat public accordingly.

Two response details differ from the other endpoints:

• web.headers are stamped onto the response as usual, except Content-Type, which is the exposition format's contract and always wins.
• Content negotiation is on the Accept header: any request advertising application/openmetrics-text gets OpenMetrics (terminated by # EOF); everything else gets the text format.

Metric reference

All metrics are prefixed yacron2_. Per-job metrics carry the job name in a job_name label (not job, which would collide with Prometheus's own target label). Counters are cumulative since process start; they reset when the daemon restarts, which Prometheus's rate()/increase() handle natively.

Daemon

Metric	Type	Description
yacron2_info{version}	gauge (info)	Build information; always 1.
yacron2_start_time_seconds	gauge	Unix time this yacron2 process started.
yacron2_job_set_info{job_set_id}	gauge (info)	The job-set fingerprint of the currently loaded configuration; always 1. Compare across a fleet to spot config drift.
yacron2_jobs{state}	gauge	Number of configured jobs, state="enabled" / "disabled".
yacron2_config_last_reload_successful	gauge	1 if the most recent configuration parse succeeded, 0 if the config on disk is currently broken (the daemon keeps running the previous good config). Absent until the first parse.
yacron2_config_last_reload_success_timestamp_seconds	gauge	Unix time of the last successful configuration parse.

Per job

Metric	Type	Description
yacron2_job_runs_total{job_name, status}	counter	Finished runs by outcome: status="success" / "failure" / "cancelled", exactly as recorded in the run history. Zero-filled for every configured job from the first scrape. Runs replaced under concurrencyPolicy: Replace are not runs and are not counted, matching the dashboard.
yacron2_job_duration_seconds	histogram	Duration of finished runs (bucket bounds from durationBuckets). Cancelled-without-starting records carry no duration and are not observed.
yacron2_job_retries_total{job_name}	counter	Retry attempts actually launched (onFailure.retry); deferred or abandoned retries are not counted.
yacron2_job_permanent_failures_total{job_name}	counter	Failures with no retry remaining -- exactly when onPermanentFailure reporting fires.
yacron2_job_start_failures_total{job_name}	counter	Runs whose command could not be launched at all (also recorded as failures with the conventional exit code 127).
yacron2_job_last_success_timestamp_seconds{job_name}	gauge	Unix time of the last successful finish. Absent until the first success.
yacron2_job_last_failure_timestamp_seconds{job_name}	gauge	Unix time of the last failed finish. Absent until the first failure.
yacron2_job_info{job_name, schedule, cluster_policy}	gauge (info)	Static configuration facts; always 1.
yacron2_job_enabled{job_name}	gauge	1 if the job is enabled in the loaded config.
yacron2_job_running{job_name}	gauge	Number of currently running instances (can exceed 1 under concurrencyPolicy: Allow).
yacron2_job_next_run_timestamp_seconds{job_name}	gauge	Unix time of the next scheduled run. Absent for disabled jobs and @reboot schedules.
yacron2_job_last_run_timestamp_seconds{job_name}	gauge	Unix time the most recent run finished.
yacron2_job_last_run_duration_seconds{job_name}	gauge	Duration of the most recent finished run.
yacron2_job_last_run_exit_code{job_name}	gauge	Exit code of the most recent finished run.
yacron2_job_last_run_success{job_name}	gauge	1 if the most recent finished run succeeded, else 0 (a cancelled run counts as 0).

Cluster

Emitted when a cluster is configured; the values mirror the pre-derived fields of GET /cluster, so the same alert rules documented there apply.

Metric	Type	Description
yacron2_cluster_enabled	gauge	1 when a leadership backend is running on this node (0 also covers a configured backend that failed to start).
yacron2_cluster_info{backend, node_name, distribution}	gauge (info)	Backend (gossip/kubernetes/etcd), this node's name, and the ownership distribution; always 1.
yacron2_cluster_size / yacron2_cluster_quorum	gauge	Effective cluster size N and the quorum threshold (lease backends report 1/1).
yacron2_cluster_quorate	gauge	Whether this node is part of a quorum -- the single most alertable cluster signal.
yacron2_cluster_is_leader	gauge	Whether this node holds scheduled-job leadership. Always 0 under distribution: spread (ownership is per job there), mirroring GET /cluster.
yacron2_cluster_leader_info{leader}	gauge (info)	The observed leader's name; absent when there is none.
yacron2_cluster_conflict{kind}	gauge	The three fail-closed conflict gates, kind="nodename" / "size" / "policy". Any 1 means Leader jobs are standing down cluster-wide.
yacron2_cluster_peers{status}	gauge	Gossip backend only: configured peers by observed status (agreed, syncing, drifted, unreachable, untrusted, conflict, self, unknown), zero-filled so alert series never vanish.
yacron2_cluster_leader_transitions_total	counter	Times this node acquired or lost leadership. Observed at scheduler cadence (once per minute), so a flap shorter than a scheduler tick may be missed. Emitted only when electLeader is on (an observe-only cluster has no leadership to transition).
yacron2_cluster_quorum_transitions_total	counter	Times this node joined or left quorum (same cadence and electLeader caveats).

If a backend read fails during a scrape, the cluster block degrades to yacron2_cluster_enabled alone (with the error logged by the prometheus logger) rather than failing the whole scrape.

Semantics and guarantees

• Metrics never affect scheduling. Every hook is a synchronous in-memory increment on the scheduler's event loop; there is nothing to time out, retry, or crash on. This is the same contract the statsd writer has, without the UDP.
• Counters agree with the API. yacron2_job_runs_total increments at the exact point a run enters the run history (GET /jobs/{name}/runs), so the two surfaces can never disagree on outcomes.
• Counters survive reloads, not restarts. The accumulators live on the daemon (not the web app), so a config reload -- even one that restarts the web server or the cluster manager -- keeps them. A process restart resets them, which is normal for Prometheus counters.
• Removed jobs disappear. On every successful reload, series for jobs no longer in the config are dropped; a renamed job starts over at zero.
• @reboot keep-alives look "always running". A long-running @reboot job shows yacron2_job_running 1 with zero finished runs -- that is the correct reading, not missing data.

Example alerts

groups:
  - name: yacron2
    rules:
      # a job's most recent run failed
      - alert: YacronJobFailed
        expr: yacron2_job_last_run_success == 0
        for: 5m
      # a job that should run hourly has not succeeded for 2 hours
      - alert: YacronJobStale
        expr: time() - yacron2_job_last_success_timestamp_seconds > 7200
      # the config file on disk is broken; the daemon is running stale config
      - alert: YacronConfigBroken
        expr: yacron2_config_last_reload_successful == 0
        for: 10m
      # the cluster cannot elect a leader: Leader jobs are not running anywhere
      - alert: YacronClusterNotQuorate
        expr: yacron2_cluster_quorate == 0
        for: 5m
      # more than one node believes it is the leader (split-brain)
      - alert: YacronMultipleLeaders
        expr: sum(yacron2_cluster_is_leader) > 1

The Clustering monitoring guide discusses the cluster signals in depth.

See also

• HTTP Control API: the web server /metrics is part of -- listeners, authToken, headers, reload behavior.
• Metrics with statsd: the push-side alternative; both can be enabled at once.
• Clustering and Leader Election: what the cluster gauges mean and when they change.
• Failure Detection and Retries: how run outcomes, retries, and permanent failures are determined.
• Configuration Reference: the full web section.