Skip to content

HTTP API

Jump to bottom
ptweezy edited this page Jul 8, 2026 · 4 revisions

HTTP Control API

yacron2 exposes an optional aiohttp REST control API, enabled by adding a top-level web section to the configuration. It serves endpoints for querying the daemon version, inspecting job status, starting and cancelling jobs on demand, reading per-job run history, tailing captured job output live, exposing Prometheus metrics, and (when a cluster section is configured) reporting the cluster and fleet views. This page documents the configuration schema, every endpoint, bearer-token authentication, Unix-socket permissions, and lifecycle behavior.

The interface is inherited from upstream yacron; web.authToken and web.socketMode are yacron2 additions.

Looking for the browser UI? The same HTTP interface also serves the built-in Web Dashboard at / on every http:// listener (enabled by default; disable it with ui: false). This page documents the REST endpoints; the Web Dashboard page is the visual tour.

Enabling the API

Add a web section with at least one listen URL:

web:
  listen:
    - http://127.0.0.1:8080
    - unix:///tmp/yacron2.sock

Windows: unix:// listeners are not supported on Windows. On Windows such a listen URL is skipped with the warning Ignoring web listen url <url>: unix-socket listeners are not supported on this platform (aiohttp's UnixSite needs create_unix_server, which the Windows Proactor loop lacks); use an http:// listener instead. The http:// listener and the entire HTTP control API otherwise behave identically on Windows. See Running on Windows.

The server is created only when web.listen is non-empty. There must be exactly one web block across the whole configuration: a duplicate web block in an included file or a second file in a config directory raises a ConfigError. See Includes, Defaults, and Multi-File Config.

Configuration reference

The web section is parsed by the strictyaml CONFIG_SCHEMA in yacron2/config.py. listen is required; the rest are optional (strictyaml Opt(...)).

Option Type Default Description
listen sequence of strings (required) List of URLs to bind. Each is http://host:port or unix:///path. An empty list disables the server.
headers map of string→string (none) Extra HTTP headers added to every 200 success response (all routes, including /cluster and /job-set-id) and to the 409 conflict body, but not the 404 or 401.
authToken map (value/fromFile/fromEnvVar) (none) When set, requires bearer-token authentication on all routes (see Authentication).
socketMode string (octal) (none) File mode applied via chmod to unix:// listen sockets (see Unix socket permissions). Applies only to unix:// sockets, so it is irrelevant on Windows (where unix-socket listeners are unsupported and skipped with a warning).

listen URL forms

Scheme Form Requirements
http http://host:port Both host and port are required. An http URL missing either is logged as a warning (Ignoring web listen url ...: http url needs host and port) and skipped.
unix unix:///path/to/socket Binds an aiohttp UnixSite at the given filesystem path. POSIX-only: on Windows UnixSite is unavailable (no create_unix_server on the Proactor loop), so such a URL is skipped with the warning Ignoring web listen url <url>: unix-socket listeners are not supported on this platform. Use an http:// listener instead.

Any other scheme is logged (scheme ... not supported) and skipped. Binding maps to web.TCPSite for http and web.UnixSite for unix (web_site_from_url in yacron2/cron.py).

https is not a recognized scheme. To serve the API over TLS, bind to a loopback http address or a unix socket (POSIX-only; on Windows use a loopback http address) and terminate TLS in a reverse proxy.

Endpoints

All routes are registered in start_stop_web_app:

Method Path Handler Success status
GET /version _web_get_version 200
GET /job-set-id _web_job_set_id 200
GET /cluster _web_get_cluster 200
GET /fleet _web_get_fleet 200
GET /node _web_get_node 200
GET /node/history _web_node_history 200
GET /status _web_get_status 200
GET /jobs _web_list_jobs 200
GET /jobs/{name}/runs _web_job_runs 200
GET /jobs/{name}/resources _web_job_resources 200
GET /jobs/{name}/trends _web_job_trends 200
POST /jobs/{name}/start _web_start_job 200
POST /jobs/{name}/cancel _web_cancel_job 200
GET /jobs/{name}/logs _web_job_logs 200 (SSE stream)
GET /dags _web_list_dags 200
GET /dags/{name}/runs _web_dag_runs 200
GET /dags/{name}/runs/{run_key} _web_dag_run 200
GET /dags/{name}/runs/{run_key}/xcom _web_dag_xcom 200
GET /dags/{name}/runs/{run_key}/tasks/{taskkey}/logs _web_dag_task_logs 200 (SSE stream)
POST /dags/{name}/trigger _web_dag_trigger 200
POST /dags/{name}/backfill _web_dag_backfill 200
POST /dags/{name}/runs/{run_key}/tasks/{taskkey}/decision _web_dag_decision 200
GET /state _web_state 200
GET /state/documents _web_state_documents 200
GET /state/records _web_state_records 200
GET /metrics _web_metrics 200 (Prometheus exposition; omitted when metrics: false)
GET / _web_index 200 (dashboard page; omitted when ui: false)

The /dags/... routes are documented under DAG endpoints and the /state... routes under State inspector endpoints.

The configured headers map is applied to every 200 success response across all routes (including /cluster and /job-set-id) and to the 409 conflict bodies of /jobs/{name}/start and /jobs/{name}/cancel. The 404 (unknown job) and 401 (authentication failure) responses are raised without it.

The same interface serves the Web Dashboard at /; that page is the visual tour of the UI these endpoints feed.

GET /version

Returns the yacron2 version as text/plain (the value of yacron2.version.version).

$ http get http://127.0.0.1:8080/version
HTTP/1.1 200 OK
Content-Type: text/plain; charset=utf-8

1.0.13

GET /status

Returns the status of every configured job. The response format depends on the request's Accept header:

  • If Accept is exactly application/json, the response is a JSON array (application/json).
  • Otherwise the response is text/plain, one job per line.

Each job has one of three statuses, determined in this order:

Status When Fields
running One or more instances are currently running. job, status, pid (list of the PIDs of running instances whose subprocess has been started, i.e. runjob.proc is not None).
disabled The job is not running and enabled: false. job, status.
scheduled The job is not running and is enabled. job, status, scheduled_in.

For scheduled jobs, scheduled_in is the number of seconds until the next run (a float, computed from the job's crontab in the job's timezone). For an @reboot schedule, scheduled_in is the literal string "@reboot".

The disabled status is reported honestly instead of an inapplicable scheduled (in N seconds).

Text form:

$ http get http://127.0.0.1:8080/status
HTTP/1.1 200 OK
Content-Type: text/plain; charset=utf-8

test-01: scheduled (in 14 seconds)
test-02: running (pid: 12345)
test-03: disabled

In the text form, scheduled_in is rendered as a human-readable relative time (in N seconds / minutes / hours / days), running jobs show running (pid: <comma-separated pids>), and disabled jobs show disabled.

JSON form:

$ http get http://127.0.0.1:8080/status Accept:application/json
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8

[
    {"job": "test-01", "status": "scheduled", "scheduled_in": 6.16588},
    {"job": "test-02", "status": "running", "pid": [12345]},
    {"job": "test-03", "status": "disabled"}
]

GET /cluster

Returns this node's cluster view as JSON. When no cluster section is configured, it returns {"enabled": false, "peers": []}. When a cluster section is present it returns enabled: true plus a backend field naming the active leadership backend (gossip, kubernetes, etcd, or filesystem) and the node's view: its node_name and job_set_id, the computed cluster_size and quorum, whether elect_leader is on, the distribution mode (single-leader or spread), whether any conflict is pausing Leader jobs (conflict, the umbrella flag), whether this node is quorate, the elected leader (null when this node is not quorate, and always null in spread mode) and is_leader (always false in spread mode), and a peers array (each with host, status, node_name, job_set_id, last_seen, last_error, mismatch_streak, and node_stats). Under distribution: spread, per-job owners instead appear as a clusterOwner field on each leader-gated job in GET /jobs.

A top-level node_stats object carries this node's own live CPU/memory (the same shape as GET /node's resources), always present (it is local and free). Each peer's node_stats is its last-shared load — null unless the cluster shares node stats via cluster.observability. The dashboard's cluster panel renders this node's load in the summary and a per-peer Load column when peers share.

The umbrella conflict flag is set by any of three triggers, each with its own detail list, and all three stand Leader jobs down: a duplicate nodeName (the offending names in conflict_names); an agreeing peer declaring a different cluster size (size_conflict: true, the divergent sizes in conflicting_sizes); and a quorate peer advertising a different distribution or elect_leader setting, a coordination-policy conflict surfaced as policy_conflict: true with the differing descriptors in conflicting_policies.

The lease backends (kubernetes / etcd / filesystem) have no static peer set, so their view is lease-shaped: backend names the backend, peers is empty, cluster_size/quorum are 1, elect_leader is true, distribution is single-leader, all three conflict flags (conflict, size_conflict, policy_conflict) are always false, and an extra lease block carries the backend-specific detail. This is an endpoint-reference view; the field semantics (what quorate means for a lease backend, the lease block contents, and the expiry rules below) are documented once in Clustering and Leader Election. For kubernetes the block is {name, namespace, identity, holder, expiry}, for etcd {electionName, identity, holder, leaseId, expiry}, and for filesystem {path, electionName, identity, holder, fence, expiry}. fence is the store's monotonic takeover counter: it bumps every time the lease changes hands (a renew by the same holder keeps it). For kubernetes and etcd the expiry is populated only while this node holds the lease: a follower reports expiry: null (for kubernetes it is the local lease deadline while this node leads; for etcd the current lease deadline). For filesystem it is the written expiry of the last lease this node observed, follower included. quorate is whether the node has a fresh successful read of the lease store (stale → Leader fails closed; the never-skip PreferLeader default then runs the job anyway).

$ http get http://127.0.0.1:8080/cluster Accept:application/json
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8

{
    "enabled": true,
    "backend": "gossip",
    "node_name": "node-a",
    "job_set_id": "v1:…",
    "cluster_size": 3,
    "quorum": 2,
    "elect_leader": true,
    "distribution": "single-leader",
    "conflict": false,
    "conflict_names": [],
    "size_conflict": false,
    "conflicting_sizes": [],
    "policy_conflict": false,
    "conflicting_policies": [],
    "quorate": true,
    "leader": "node-a",
    "is_leader": true,
    "peers": [
        {"host": "yacron-b.internal:8443", "status": "agreed", "node_name": "node-b", "job_set_id": "v1:…", "last_seen": "2026-06-23T19:00:00+00:00", "last_error": null, "mismatch_streak": 0}
    ]
}

A lease backend (here kubernetes) returns the lease-shaped view instead:

{
    "enabled": true,
    "backend": "kubernetes",
    "node_name": "yacron2-0",
    "job_set_id": "v1:…",
    "cluster_size": 1,
    "quorum": 1,
    "elect_leader": true,
    "distribution": "single-leader",
    "conflict": false, "conflict_names": [],
    "size_conflict": false, "conflicting_sizes": [],
    "policy_conflict": false, "conflicting_policies": [],
    "quorate": true,
    "leader": "yacron2-0",
    "is_leader": true,
    "peers": [],
    "lease": {"name": "yacron2-leader", "namespace": "default",
              "identity": "yacron2-0", "holder": "yacron2-0",
              "expiry": "2026-06-24T19:00:14.000000Z"}
}

The per-peer status values (agreed, syncing, drifted, unreachable, untrusted, self, conflict, unknown) are documented in Clustering and Leader Election.

The separate GET /peer attestation endpoint is not part of this web API. It is served only on the cluster's own mutual-TLS listen address (default port 8443), never on the public web listeners. When node stats are shared, each /peer response carries the node's live load as an X-Yacron2-Node-Stats response header (on 200 and 304 responses alike, never in the body), so sharing preserves that exchange's conditional 304 optimisation. See Clustering and Leader Election.

GET /fleet

Returns the cluster-wide per-job run view that backs the dashboard's fleet view: one entry per node, each carrying that node's per-job run summaries. It is answered entirely from state this node already holds. Every gossip node piggybacks a compact summary of its own jobs (running / enabled / seconds to next fire / last finished run) on its mutual-TLS /peer response, so the summaries arrive with the peer polls this node is already making; serving /fleet triggers no peer traffic. Any node can therefore serve the whole fleet's picture, at most one gossip interval stale per peer.

When no cluster section is configured, or the backend is a lease backend (kubernetes / etcd / filesystem, which carry only a lease and know nothing about what other nodes run), it returns {"enabled": false, "nodes": []}.

For the gossip backend it returns enabled: true, the serving node's node_name, the distribution and elect_leader policy, the gossip interval in seconds (the peer-data freshness bound), and a nodes array. The serving node is always first (self: true, status self, as_of stamped at request time); each configured peer follows with the status and host from the peer table and the summaries absorbed from its last successful poll (as_of = last_seen). Self-listings are skipped and two addresses that answer as the same process are deduplicated.

Per node: jobs maps each job name to {running, enabled, scheduled_in, last}, where last is {outcome, finished_at, duration, exit_code} or null for a job that has not run since that node started. jobs: null (as opposed to {}) means no snapshot is held for that node at all: it was never reached, or it runs an older build that does not gossip summaries. Each node also carries node_stats — its whole-node CPU/memory (the same shape as GET /node's resources) — when the cluster shares node load via cluster.observability; null when that node shares none. truncated: true flags a node with more jobs than the per-payload cap (512), whose advertised set is the sorted-name prefix. A briefly unreachable peer keeps its last-known summaries with the old as_of rather than being blanked, so stale data is visibly stale instead of silently missing.

$ http get http://127.0.0.1:8080/fleet Accept:application/json
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8

{
    "enabled": true,
    "backend": "gossip",
    "node_name": "node-a",
    "distribution": "spread",
    "elect_leader": true,
    "interval": 30,
    "nodes": [
        {"node_name": "node-a", "host": null, "self": true, "status": "self",
         "as_of": "2026-06-23T19:00:02+00:00", "truncated": false,
         "jobs": {"backup": {"running": false, "enabled": true, "scheduled_in": 1042.5,
                             "last": {"outcome": "success", "finished_at": "2026-06-23T18:00:01+00:00",
                                      "duration": 12.4, "exit_code": 0}}}},
        {"node_name": "node-b", "host": "yacron-b.internal:8443", "self": false, "status": "agreed",
         "as_of": "2026-06-23T18:59:45+00:00", "truncated": false,
         "jobs": {"backup": {"running": true, "enabled": true, "scheduled_in": null, "last": null}}}
    ]
}

The summaries are observability data only: they never feed leader election or any run/skip decision, and a malformed or hostile peer payload degrades to "no data for that node" rather than poisoning the view.

GET /node

The serving node's live CPU and memory, sampled fresh per request (this is what drives the dashboard header's node meter). Returns the node identity and a resources object:

$ http get http://127.0.0.1:8080/node
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8

{
    "node_name": "node-a",
    "resources": {
        "cpu_percent": 37.2,
        "cpu_count": 8,
        "mem_percent": 61.4,
        "mem_used_bytes": 5284167680,
        "mem_total_bytes": 8589934592,
        "proc_rss_bytes": 58720256,
        "proc_cpu_percent": 1.1
    }
}

node_name is the cluster node name when clustered, else the hostname. cpu_percent / mem_percent are whole-host utilisation; proc_rss_bytes / proc_cpu_percent are the yacron2 daemon's own footprint (best-effort, may be absent if the platform denies the per-process read). resources is null when host sampling is unavailable (psutil could not read the host), and the dashboard then hides the meter. CPU percentages are measured since the previous sample, so the first request after startup reads a priming 0.

Containers. When the daemon runs under a cgroup v2 limit (Docker/Kubernetes memory or CPU limits, or a systemd slice with MemoryMax/CPUQuota), these numbers describe its own slice rather than the whole host: mem_total_bytes is the effective memory limit, mem_used_bytes is the slice's usage with reclaimable page cache excluded (the same accounting docker stats shows), cpu_count is the CPU quota rounded up, and cpu_percent is utilisation of that quota. Memory and CPU switch over independently — a container with only a memory limit still reports host-wide CPU. Unlimited cgroups, cgroup v1 hosts, and non-Linux platforms report whole-host numbers as before; the response shape never changes.

POST /jobs/{name}/start

Launches the named job immediately, regardless of its schedule. {name} is the job's name.

Condition Response
No job with that name. 404 Not Found.
The job exists but has enabled: false. 409 Conflict, body job '<name>' is disabled.
Otherwise. 200 OK, empty body; the job is launched via the normal launch path.

The 409 is deliberate: a disabled job behaves as if it is not there, so the API refuses to launch it manually rather than overriding the config.

Manual launch goes through maybe_launch_job, so the job's concurrencyPolicy applies. If an instance is already running, Allow starts another, Forbid does not start a new one (the 200 still returns), and Replace cancels the running instance(s) first. See Concurrency and Timeouts.

$ http post http://127.0.0.1:8080/jobs/test-02/start
HTTP/1.1 200 OK
Content-Length: 0

POST /jobs/{name}/cancel

Terminates every currently-running instance of the named job, using the same graceful terminate-then-kill sequence yacron2 uses elsewhere (honoring the job's killTimeout; see Concurrency and Timeouts). Instances are cancelled concurrently, so a job with several running instances costs at most one killTimeout, not one per instance.

Condition Response
No job with that name. 404 Not Found.
The job exists but no instance is running. 409 Conflict, body job '<name>' is not running.
Otherwise. 200 OK, empty body; all running instances are cancelled.

A run cancelled this way is recorded in the job's history with the outcome cancelled. Cancellation is a deliberate operator action, not a job failure, so it is not reported (onFailure does not fire) and does not trigger retries.

$ http post http://127.0.0.1:8080/jobs/test-03/cancel
HTTP/1.1 200 OK

GET /jobs

Returns a JSON array describing every job: its schedule and timezone, whether it is enabled and running, the time until its next scheduled run, a summary of its most recent finished run, and a compact tail of recent outcomes. This is the endpoint the Web Dashboard polls.

Field Meaning
name, enabled, schedule, command The job's name and enabled flag, its schedule as a crontab string, and its command (argv lists are joined for display).
captureStdout, captureStderr Which output streams the job captures, and therefore which are available from /jobs/{name}/logs.
utc, timezone The schedule's reference frame: utc (default true) and the IANA timezone name, or null.
running, pids Whether any instance is currently running, and the PIDs of running instances whose subprocess has started.
running_resources Present only while a monitorResources job has a running instance: the live CPU/memory of the running instance(s), summed — {cpu_percent, cpu_seconds, rss_bytes, instances}. Omitted otherwise. cpu_percent is usage since the last sample and can exceed 100 across cores.
scheduled_in Seconds until the next scheduled run (a float), or null when not applicable (disabled, currently running, or a one-off @reboot schedule).
last_run The most recent finished run (outcome, exit_code, started_at, finished_at, duration, fail_reason), or null if the job has not run yet.
history Compact oldest-first tail of recent runs (outcome and duration only), sized for the dashboard's inline sparkline. Full per-run detail comes from /jobs/{name}/runs.
clusterPolicy, clusterOwner Present only when leader election is configured: the job's cluster policy, and, under distribution: spread for leader-gated jobs, the node that currently owns the job (null when there is no quorum). 
$ http get http://127.0.0.1:8080/jobs
[
    {
        "name": "test-01",
        "enabled": true,
        "schedule": "*/5 * * * *",
        "command": "echo foobar",
        "captureStdout": true,
        "captureStderr": true,
        "utc": true,
        "timezone": "UTC",
        "running": false,
        "pids": [],
        "scheduled_in": 42.1,
        "last_run": {
            "outcome": "success",
            "exit_code": 0,
            "started_at": "2026-06-21T12:00:00+00:00",
            "finished_at": "2026-06-21T12:00:01+00:00",
            "duration": 1.02,
            "fail_reason": null
        },
        "history": [
            {"outcome": "success", "duration": 0.98},
            {"outcome": "failure", "duration": 1.21},
            {"outcome": "success", "duration": 1.02}
        ]
    }
]

GET /jobs/{name}/runs

Returns the job's retained run history (oldest first, bounded, and held in memory -- though with a durable state store configured it is rehydrated from the durable run ledger after a restart) together with aggregate statistics. Returns 404 Not Found for an unknown job.

Each entry in runs carries the same fields as last_run in GET /jobs (outcome, exit_code, started_at, finished_at, duration, fail_reason, and resources). resources is null unless the job opted into monitorResources, in which case it is {cpu_user_seconds, cpu_system_seconds, cpu_total_seconds, max_rss_bytes, samples} for that run. Besides success, failure, and cancelled, outcome can be unknown: a crash-reconciled run, recorded when the daemon exited or lost the state store mid-run so no completion was ever written. It is a non-verdict: excluded from success_rate, counted only in total, with no started_at or duration (fail_reason explains the interruption). stats summarizes them:

stats field Meaning
total, success, failure, cancelled Counts by outcome over the retained history.
success_rate Success rate over runs that ran to completion. Cancellations are user-initiated, not a verdict on the job, so they are excluded; null when no run has completed.
avg_duration, min_duration, max_duration, last_duration Duration aggregates in seconds, over runs with a recorded duration; null when there are none.
avg_cpu_seconds, max_cpu_seconds, last_cpu_seconds CPU-time aggregates over the monitorResources runs in the window; null when none were monitored.
avg_rss_bytes, max_rss_bytes, last_rss_bytes Peak-RSS aggregates (bytes) over the monitored runs; null when none were monitored. 
$ http get http://127.0.0.1:8080/jobs/test-01/runs
{
    "name": "test-01",
    "runs": [
        {
            "outcome": "success",
            "exit_code": 0,
            "started_at": "2026-06-21T12:00:00+00:00",
            "finished_at": "2026-06-21T12:00:01+00:00",
            "duration": 1.02,
            "fail_reason": null
        }
    ],
    "stats": {
        "total": 1,
        "success": 1,
        "failure": 0,
        "cancelled": 0,
        "success_rate": 1.0,
        "avg_duration": 1.02,
        "min_duration": 1.02,
        "max_duration": 1.02,
        "last_duration": 1.02
    }
}

GET /jobs/{name}/resources

Chart-grade CPU/RSS time series for one job — the heavyweight sibling of the summary numbers that ride GET /jobs and GET /jobs/{name}/runs. The dashboard fetches it lazily when a job's Resources tab is opened, never on the poll loop. Returns 404 Not Found for an unknown job.

$ http get http://127.0.0.1:8080/jobs/test-01/resources runs==5
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8

{
    "name": "test-01",
    "monitored": true,
    "interval": 1.0,
    "live": [
        {
            "started_at": "2026-07-07T12:00:00+00:00",
            "pid": 4242,
            "current": {"cpu_seconds": 8.4, "cpu_percent": 96.2, "rss_bytes": 21430272},
            "series": [[1751889600.0, 0.0, 20971520], [1751889601.0, 98.1, 21430272]]
        }
    ],
    "runs": [
        {
            "outcome": "success",
            "started_at": "2026-07-07T11:00:00+00:00",
            "finished_at": "2026-07-07T11:00:42+00:00",
            "duration": 42.1,
            "exit_code": 0,
            "fail_reason": null,
            "resources": {
                "cpu_user_seconds": 39.2,
                "cpu_system_seconds": 1.1,
                "cpu_total_seconds": 40.3,
                "max_rss_bytes": 22020096,
                "samples": 42,
                "series": [[1751886000.0, 0.0, 20971520], [1751886001.0, 97.4, 22020096]]
            }
        }
    ]
}

A series is a list of [t, cpu_percent, rss_bytes] points, oldest first, with t in epoch seconds. Points are recorded every monitorResources.interval seconds and downsampled in place once a run exceeds its configured history cap (mean CPU%, peak RSS per merged bucket, so spikes survive), so a series is bounded no matter how long the run. live carries the run-so-far series of each currently-running monitored instance plus its current instantaneous readings; runs the recorded series of recent finished monitored runs (oldest first, unmonitored runs are omitted), capped by the runs query parameter (default 20, clamped to the retained history). With a durable state store, run series survive restarts inside the run ledger records. monitored: false with empty lists means the job never opted into monitorResources — distinguishable from "monitored but no data yet".

GET /node/history

The serving node's retained CPU/memory history — the time-series companion to GET /node's live snapshot, driving the dashboard's node chart (click the header meter). Sampled in the background per web.nodeHistory (every 5s, last hour, by default), independent of whether anyone is polling.

$ http get http://127.0.0.1:8080/node/history
HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8

{
    "node_name": "node-a",
    "enabled": true,
    "interval": 5.0,
    "points": [[1751889600.0, 37.2, 61.4], [1751889605.0, 35.9, 61.5]]
}

points is oldest-first [t, cpu_percent, mem_percent] (epoch seconds; the same cgroup-aware percentages GET /node reports). A gap between consecutive points much wider than interval means the daemon was down, not idle. The ring is in-memory only and resets on restart. enabled: false (with empty points) means the sampler is off — web.nodeHistory: false, or psutil cannot read this host.

GET /jobs/{name}/trends

The long-horizon sibling of GET /jobs/{name}/runs: the same stats object, computed per time window over the durable run ledger, which survives restarts and -- on a shared mount -- merges every node's runs. Returns 404 Not Found for an unknown job.

The response carries the job name, a source field, a generated_at timestamp, and a windows map with keys 1h, 24h, 7d, 30d, and all, each holding the stats object documented under GET /jobs/{name}/runs for the runs that finished inside that window:

$ http get http://127.0.0.1:8080/jobs/test-01/trends
{
    "name": "test-01",
    "source": "durable",
    "generated_at": "2026-07-04T12:00:00+00:00",
    "windows": {
        "1h":  { "total": 4, "success": 4, "...": "..." },
        "24h": { "total": 96, "success": 95, "...": "..." },
        "7d":  { "total": 672, "success": 668, "...": "..." },
        "30d": { "...": "..." },
        "all": { "...": "..." }
    }
}

source is "durable" when the aggregates were computed over the durable ledger (the horizon is then bounded by state.maxRunsPerJob retention, and by the 5000 newest records per request on an unbounded-retention store) and "memory" when the endpoint degraded to the in-memory run history -- because no state: section is configured, or the store could not be read in time. The endpoint always answers rather than erroring on store trouble. See Durable State for the ledger this reads.

GET /jobs/{name}/logs

A Server-Sent Events stream of a job's captured output. Returns 404 Not Found for an unknown job. The stream begins with the lines already buffered (from the most recent running instance, or else the last finished run's retained output), then follows a running job's new lines live, and finishes with an end event:

  • Each line arrives as an event: line whose data is a JSON object {"stream": "stdout"|"stderr", "line": "..."}.
  • When the run finishes, the server sends event: end with data {}. If the job has no captured output at all yet, the stream ends immediately with event: end and data {"reason": "no-output"}.
  • During quiet periods the server writes an SSE comment (: ping) every 15 seconds as a keep-alive, which also detects disconnected clients.

Only the streams a job captures (captureStdout / captureStderr) appear here; see Output Capturing. The response carries X-Accel-Buffering: no so reverse proxies (e.g. nginx) do not buffer the stream.

$ curl -N http://127.0.0.1:8080/jobs/test-01/logs
event: line
data: {"stream": "stdout", "line": "foobar"}

event: end
data: {}

DAG endpoints

The orchestration DAGs (the dags: section) are introspected and controlled here. All are token-gated like the job endpoints.

GET /dags

The configured DAGs and their tasks:

[{"name": "nightly-etl", "enabled": true, "scheduled": true,
  "tasks": [{"id": "extract", "type": "task", "dependsOn": []},
            {"id": "load", "type": "task", "dependsOn": ["extract"]}]}]

GET /dags/{name}/runs

Recent dag_runs (newest first), each with its state and a per-state task count:

{"dag": "nightly-etl", "runs": [
  {"runKey": "2026-07-04T02:00:00_00:00", "runId": "…", "state": "success",
   "kind": "scheduled", "logicalDate": "2026-07-04T02:00:00+00:00",
   "taskStates": {"success": 3}}]}

404 if the DAG is not configured.

GET /dags/{name}/runs/{run_key}

One run's full durable document -- every task's state, attempt, timing, XCom expansion (mapped), and approval decisions. 404 if the run is unknown.

POST /dags/{name}/trigger

Create and start a manual run now; returns {"dag": …, "runKey": …}. 404 if the DAG is not configured; a run that could not be durably recorded (the state backend is unavailable) surfaces as a 500 error rather than a runKey for a run that does not exist.

POST /dags/{name}/backfill

Replay a scheduled DAG across a historical range. Body: {"from": "<ISO>", "to": "<ISO>"}. Idempotent (create-if-absent per date) and bounded. Returns {"ok": true, "created": <N>}; 400 on a bad range.

POST /dags/{name}/runs/{run_key}/tasks/{taskkey}/decision

Approve or reject an approval gate. Body: {"decision": "approve"|"reject", "by": "<who>"}. 200 on success, 400 on a bad decision value, 409 if the task is not awaiting a decision.

GET /dags/{name}/runs/{run_key}/xcom

The XCom outputs the run's tasks published, as a flat list of entries (task, key, sha256, size, timestamp) with small text values inlined and larger ones metadata-only; truncated flags a run with more entries than the cap. 404 if the DAG or run is unknown.

GET /dags/{name}/runs/{run_key}/tasks/{taskkey}/logs

An SSE stream of a running task instance's live captured output, in the same event shape as GET /jobs/{name}/logs. A finished instance's buffer is not retained, so the stream then ends immediately with event: end and {"reason": "no-output"}. 404 if the DAG is not configured.

State inspector endpoints

The dashboard's durable state inspector is fed by three metadata-only routes: record payloads, KV values, and archived output never cross this surface. All are token-gated like the job endpoints.

GET /state

Store health and topology plus an inventory: per-prefix stream and document counts, capped scope lists, active leases, the quarantine count, and this node's live retry ladders and held concurrency slots. Returns {"enabled": false} when no state: section is configured; an unreadable store degrades to health-only rather than erroring.

GET /state/documents?ns=<namespace>

The documents of one kv/, cursor/, or idem/ namespace (400 for any other namespace). KV values are redacted to a valueSize / valueType summary; cursor watermarks and idempotency claim metadata are returned verbatim.

GET /state/records?stream=<stream>&limit=<n>

The newest records of one stream, newest first (default 100, max 500). Archived-output logs/ streams are refused with 403: they carry raw job output, which the metadata-only stance keeps off this surface.

GET /job-set-id

Returns this instance's job-set id: the order-independent fingerprint of every job's effective configuration that replicas compare to confirm they hold the same set of jobs (see the job-set id foundation). The response is text/plain by default; with Accept: application/json it is a JSON object that also carries the job count.

$ http get http://127.0.0.1:8080/job-set-id
v1:b834d7565aee0da50cd017f666651a5ba3b2e6b161daf0cb6e430f23f51ce90b

$ http get http://127.0.0.1:8080/job-set-id Accept:application/json
{"job_set_id": "v1:b834d7…51ce90b", "jobs": 3}

GET /metrics

Exposes yacron2's native Prometheus metrics: daemon info, per-job run counters and duration histograms, live per-job gauges, and (when a cluster section is configured) the cluster health series that mirror GET /cluster. The exposition is generated by yacron2 itself, with no exporter sidecar and no extra dependency. This section covers the endpoint mechanics; the full metric reference lives in Metrics with Prometheus.

The response format depends on the request's Accept header:

  • By default the response is the classic Prometheus text format (text/plain; version=0.0.4).
  • If Accept advertises application/openmetrics-text, the response is OpenMetrics 1.0 (terminated by # EOF), as modern Prometheus servers request.

The configured web.headers are applied to the response as on every other route, except Content-Type, which this endpoint owns: the exposition format's contract always wins over an operator-configured header.

The endpoint is enabled by default whenever the web API is on. The web.metrics option tunes or disables it, accepting either a boolean shorthand (metrics: false disables the endpoint) or a map:

Sub-option Type Default Description
enabled bool true Serve GET /metrics.
public bool false Exempt /metrics (and only /metrics) from web.authToken bearer-token authentication (see Authentication).
durationBuckets sequence of floats 0.1, 0.5, 1, 5, 15, 60, 300, 900, 3600 Upper bounds (seconds) of the yacron2_job_duration_seconds histogram. Bounds must be finite, positive, and strictly increasing; anything else raises a ConfigError.

The metric registry is owned by the daemon rather than the web app, so counters survive config reloads (including ones that restart the web server) and reset only when the process restarts; series for jobs removed by a reload are pruned.

$ curl http://127.0.0.1:8080/metrics
# HELP yacron2_info yacron2 build information.
# TYPE yacron2_info gauge
yacron2_info{version="1.0.13"} 1
# HELP yacron2_jobs Number of configured jobs by enablement state.
# TYPE yacron2_jobs gauge
yacron2_jobs{state="enabled"} 2
yacron2_jobs{state="disabled"} 1
# HELP yacron2_job_runs_total Finished job runs by outcome, as recorded in the run history.
# TYPE yacron2_job_runs_total counter
yacron2_job_runs_total{job_name="test-01",status="success"} 12
yacron2_job_runs_total{job_name="test-01",status="failure"} 1
yacron2_job_runs_total{job_name="test-01",status="cancelled"} 0
...

GET / (the dashboard page)

Serves the single-page Web Dashboard. Set ui: false in the web section to disable the page and expose only the REST endpoints. The page is served with secure default headers (a strict Content-Security-Policy, anti-clickjacking, and nosniff) with any operator web.headers merged on top, so a deliberately-set operator header wins. When web.authToken is enabled, the page itself loads without a token (it holds no data); it prompts for the token in the browser and authenticates every data request with it (see Authentication).

Response headers

The web.headers map (merged upstream but never released in yacron 0.19) is a string→string map applied to every 200 success response across all routes (/version, /status, /cluster, /job-set-id, the job routes, and the 200 of /jobs/{name}/start) and to the 409 conflict bodies of /jobs/{name}/start and /jobs/{name}/cancel. It is not applied to the 404 (unknown job) or 401 (authentication failure) responses, which are raised without the configured headers. Example:

web:
  listen:
    - http://127.0.0.1:8080
  headers:
    X-Frame-Options: DENY
    Cache-Control: no-store

Authentication

By default the API is unauthenticated; anyone who can reach a listen address can call every endpoint. Restrict access at the network or socket level, or enable bearer-token authentication with web.authToken.

authToken resolves the token from exactly one source, in this precedence order (_resolve_web_token):

Sub-option Type Description
value string or null Literal token value.
fromFile string or null Path to a file; the token is the file contents with surrounding whitespace stripped.
fromEnvVar string or null Name of an environment variable holding the token.

When authToken is set, an aiohttp middleware (_make_auth_middleware) requires Authorization: Bearer <token> on every route:

  • The auth scheme is compared case-insensitively (Bearer, bearer, etc.) per RFC 7235.
  • The presented token is compared against the configured token in constant time via hmac.compare_digest.
  • A missing/malformed Authorization header, a wrong scheme, or a non-matching token returns 401 Unauthorized.

The one configurable exception: setting web.metrics.public: true exempts /metrics (and only /metrics) from the bearer token, for scrapers that cannot send credentials; every other route stays gated (see GET /metrics).

web:
  listen:
    - http://127.0.0.1:8080
  authToken:
    fromEnvVar: YACRON2_WEB_TOKEN
$ http get http://127.0.0.1:8080/status "Authorization:Bearer s3cr3t"
$ curl -H "Authorization: Bearer s3cr3t" http://127.0.0.1:8080/status

Fail-closed behavior

If authToken is configured but resolves to an empty token, yacron2 raises a ConfigError and refuses to start the web server, rather than silently serving the control API with no authentication. This happens when:

  • value, fromFile, and fromEnvVar are all empty/absent;
  • fromEnvVar names a variable that is unset (resolves to "");
  • fromFile points to a file that is empty or contains only whitespace.

If fromFile cannot be read (OSError), yacron2 also raises a ConfigError (web.authToken.fromFile could not be read: ...).

Unix socket permissions

Windows: this entire feature is POSIX-only. It depends on unix:// sockets (which Windows does not support) and on chmod. On Windows socketMode has no effect and unix:// listen URLs are skipped with a warning; use an http:// listener. See Running on Windows.

web.socketMode is an octal-string file mode applied with chmod to each unix:// listen socket after it starts (_apply_socket_mode):

web:
  listen:
    - unix:///run/yacron2/yacron2.sock
  socketMode: "0660"
  • The mode is parsed as base-8 (int(socketMode, 8)).
  • It is applied only to unix:// sockets; non-unix addresses are ignored.
  • An invalid mode (not an octal integer, raising ValueError) or a chmod failure (OSError) is logged as a warning (web: could not set socketMode <mode> on <path>: ...) and does not abort startup.

When using a Unix socket on a read-only-root container, point the socket at a small writable volume. See Production and Container Deployment.

Lifecycle and reload behavior

The control API lifecycle is driven by start_stop_web_app, called on each config reload from the scheduler loop:

  • If a server is running and the new web config is absent or differs from the running one, the running server is stopped (web_runner.cleanup()) before any new one is started. A change to any web field (including headers, authToken, or socketMode) thus triggers a restart of the server on reload.
  • The server is (re)started only when web is present, listen is non-empty, and no server is currently running.
  • Each listen address is bound independently. A bad URL (ValueError) or a bind failure (OSError, e.g. address already in use) on one address is logged as a warning (web: could not listen on <addr>: ...) and skipped; the remaining addresses still bind, and the config update is not aborted.
  • The web: started listening on <addr> log line is emitted only after the bind succeeds.
  • On shutdown, the running server is stopped after currently running jobs finish. On Windows, graceful shutdown is triggered with Ctrl-C or Ctrl-Break (rather than SIGTERM as on POSIX); yacron2 still finishes currently-running jobs before stopping the server, identical to POSIX behavior. See Running on Windows.

A ConfigError raised while resolving authToken (empty token or unreadable fromFile) propagates out of start_stop_web_app and is caught by the reload loop's dedicated web-app handler, which logs Error in the web configuration, so not starting the web API and leaves the web API down until a later reload fixes it. The rest of the new configuration (jobs, cluster, logging) is still applied: the web app starts under its own error handling, after the cluster manager, so a web misconfiguration cannot skip the cluster gate (which would otherwise fail open and run every Leader job on every node).

Job-facing state endpoints (loopback)

Separate from the web control API above, yacron2 can run a second, loopback-only HTTP server that hands its durable state store to the jobs it runs. It binds 127.0.0.1 on an OS-assigned ephemeral port (or the address in state.jobApi.listen), and the daemon injects its base URL and a per-run bearer token into every job's environment, so a job's command line can read and write durable state, coordinate through a fleet-wide lock, or fetch a run-scoped secret. The yacron2 state|cursor|lock|artifact|idempotent|secret CLI commands are thin clients of this endpoint; the same primitives are also reachable offline against the store directly, so this server is a front-end, not a second source of truth. The surface is served by yacron2/jobapi.py; the primitives themselves live in yacron2/jobstate.py.

Enabling the endpoint

The endpoint is stood up only when the configuration has a state: section with jobApi.enabled (default true). A stateless install, or one with a state: section but jobApi.enabled: false, never starts it and injects nothing.

Option Type Default Description
enabled bool true Serve the loopback endpoint and inject the YACRON2_* environment into every job.
listen string (ephemeral) Address to bind, as host:port or http://host:port. When omitted it binds 127.0.0.1 on an OS-assigned port; an explicit port must be in 0-65535 (0 = OS-assigned), and a non-loopback host needs allowNonLoopbackBind.
maxValueBytes int 1048576 (1 MiB) Reject a KV or cursor value larger than this many bytes (JSON-encoded) with 413. 0 means no limit.
maxArtifactBytes int 67108864 (64 MiB) Reject an artifact payload larger than this many bytes with 413. 0 means no limit.
lockTtlSeconds float 30 Default lease TTL for a job lock (floored at 5); the daemon renews it at a third of the TTL while the job holds it.
allowNonLoopbackBind bool false Explicit opt-in for a non-loopback listen host; without it such a host is a ConfigError (the endpoint serves per-run tokens and staged secrets over plaintext HTTP).

The server's own transport-level body cap is derived from these limits (the larger of the two, plus envelope headroom), so an oversized request body is refused with 413 rather than silently truncated; setting either limit to 0 lifts the transport cap too -- genuinely unlimited.

Injected environment

On launch the daemon injects these variables into every job's process. All are strings; an unknown scheduled time is the empty string rather than absent, so a job can test the variable instead of guessing whether it was set.

Variable Meaning
YACRON2_STATE_URL The loopback base URL, e.g. http://127.0.0.1:54321.
YACRON2_STATE_TOKEN A per-run bearer token, revoked the instant the run ends.
YACRON2_RUN_ID A unique id for this run.
YACRON2_JOB_NAME The job's name (also its default scope).
YACRON2_ATTEMPT The retry attempt number (0 on the first fire).
YACRON2_SCHEDULED_AT The scheduled fire time (ISO-8601), or empty.
YACRON2_HOST The host name.

Authentication and errors

Every request must carry Authorization: Bearer $YACRON2_STATE_TOKEN. The token is matched in constant time against the live run set, so a missing, malformed, forged, or stale token returns 401 Unauthorized before any state is touched. Other outcomes:

Status When Body
400 A caller error: a missing required field, or a body that is not a JSON object. {"error": "..."}
409 A cursor advanced with a value not comparable to its stored one (a type clash). {"error": "..."}
410 An artifact record survives but its payload blob was garbage collected. {"error": "..."}
413 A value or artifact larger than the configured maxValueBytes / maxArtifactBytes. {"error": "..."}
404 A get for a key, cursor, artifact, or secret that is not set. (empty)
503 The state store is unavailable or a backend call timed out. {"error": "..."}

Scopes

Every KV, cursor, idempotency, and artifact call acts in a scope: a namespace that defaults to the calling job's own name (defaultScope in GET /v1/run), so one job cannot read another's state by accident. Omit scope for that private namespace, or pass scope=global (any shared name works) for deliberate cross-job coordination.

Routes

All routes are under /v1/ and registered in JobStateAPI._routes:

Method Path Body / query Success response
GET /v1/run -- {runId, job, attempt, scheduledAt, host, defaultScope}
GET /v1/kv/get ?scope=&key= {value, updatedAt}, or 404
POST /v1/kv/set {scope?, key, value} {ok, updatedAt}
POST /v1/kv/delete {scope?, key} {existed}
GET /v1/kv/list ?scope= {scope, keys: [{key, value, updatedAt}]}
GET /v1/cursor/get ?scope=&name= {value, updatedAt}, or 404
POST /v1/cursor/advance {scope?, name, value, force?} {value, advanced}
POST /v1/idempotency/claim {scope?, key, ttl?} {fresh, claimedAt}
POST /v1/idempotency/release {scope?, key} {released}
POST /v1/artifact/put ?scope=&name=, raw body {sha256, size}
GET /v1/artifact/get ?scope=&name= raw bytes plus X-Yacron2-Sha256 / X-Yacron2-Size, or 404
GET /v1/artifact/list ?scope= {scope, artifacts: [{name, sha256, size, at}]}
POST /v1/lock/acquire {scope?, name, permits?, ttl?, wait?, blockSeconds?} {acquired, token?, slot?, fence?, ttl?}
POST /v1/lock/release {token} {released}
GET /v1/secret/get ?name= {value}, or 404
GET /v1/secret/list -- {names: [...]}

cursor/advance is monotonic by default: the stored value only ever moves to max(current, value), so a replayed or out-of-order batch cannot walk a watermark backwards, and two nodes racing to advance the same cursor converge on the larger value. advanced is false when the given value was not greater than the current one (a no-op that is not even a write), and force: true sets the value unconditionally (a deliberate rewind). idempotency/claim is a fleet-wide create-if-absent: the first caller gets fresh: true, every later caller fresh: false, so a retried run can tell "already did this" from "first time"; a positive ttl bounds the dedupe window (0 is a permanent claim).

The lock is a fleet-wide mutex (or a semaphore, with permits > 1) held as a TTL lease that the daemon renews for as long as the job holds it and releases the instant the job releases it or the run ends -- so a job that crashes or forgets to unlock never leaks a lock. wait: true retries for up to blockSeconds before giving up; without it the call makes a single pass over the permits and returns {acquired: false} when they are all taken. Like every yacron2 coordination primitive the lock is at-least-once, not exactly-once (the fence token in the reply is there for a job that needs true fencing). Run-scoped secrets are staged in memory by the daemon per run and vanish when the run ends; they never touch the store, so only the daemon holds them, and there is no scope on the secret routes -- a run sees only its own.

# Inside a job, using the injected env directly.
$ curl -s -H "Authorization: Bearer $YACRON2_STATE_TOKEN" \
    "$YACRON2_STATE_URL/v1/run"
{"runId": "…", "job": "nightly-etl", "attempt": 0, "scheduledAt": "2026-07-04T02:00:00+00:00", "host": "node-a", "defaultScope": "nightly-etl"}

# Advance the job's private ETL cursor to the last row processed.
$ curl -s -H "Authorization: Bearer $YACRON2_STATE_TOKEN" \
    -H 'Content-Type: application/json' \
    -d '{"name": "rows", "value": 20482}' \
    "$YACRON2_STATE_URL/v1/cursor/advance"
{"value": 20482, "advanced": true}

In practice a job reaches for the job-facing CLI rather than raw curl; those commands read the injected environment for you.

See also

This wiki documents yacron2. See the README and the changelog.

yacron2 is a fork of gjcarneiro/yacron.

yacron2 Wiki

Clone this wiki locally