Background Jobs Developer Guide

🛠️ Background jobs - Developer Guide

How FreeITSM records every run of every background job, and how it decides whether each job is healthy. Built for #182; the user guide is Background jobs and the design, with the rules as agreed on the issue, is Background job health - design.

The one-line version: every job calls JobRun::start() / source() / finish() from includes/job_runs.php; includes/job_health.php reads job_runs and job_run_sources on demand and applies four rules; the registry in includes/job_registry.php says what each job is expected to do. Nothing here schedules anything.

---

1. The files

🗄️ schema · 📖 read · ✏️ write · 🔗 cross-module · 🔌 REST · 🖥️ UI / help · 🧪 tests

File	🎨	What happens there
database/freeitsm.sql, includes/db_verify_schema.php	🗄️	job_runs, job_run_sources. Indexes are generated into includes/db_verify_indexes.php - run php scripts/gen_db_verify_indexes.php, never hand-edit
api/system/db_verify.php	🗄️	Seeds job_runs_recording_since, job_runs_retention_months (13) and lms_reminder_cron_token
includes/job_registry.php	📖	The list of jobs. Per job: command, web address or not, interval, grace, longest normal run, needed and sources callables. Also jobHealthConfig() (the admin's changes, one JSON setting job_health_config) and jobHealthRetentionMonths()
includes/job_runs.php	✏️	JobRun - the recorder: start, source, finish, fail, skipped, wrap, refused, clean, prune, current
includes/job_health.php	📖	jobHealthOne() / jobHealthAll() / jobHealthSummary() - the rules; jobHealthMayView() / jobHealthMayConfigure() - the gates
cron/*.php, scripts/documents_maintenance.php, scripts/directory_sync.php	✏️	The 13 jobs, each recording its own runs
includes/domains/scheduler.php, includes/lms_reminders.php, includes/projects/alerts.php	✏️	The page-load runs (trigger page), and domainJobFinish() / lmsRemindersJobFinish() shared with the cron
api/domains/settings.php, api/lms/reminder_settings.php, api/assets/proxmox_sync.php, api/assets/vcloud_sync.php, api/system/run_directory_sync.php	✏️	The Run now paths (trigger manual)
includes/workflow_scheduled.php	✏️	workflowScheduledRun() now reports its three parts, which become the job's sources
reporting/settings/manifest.php, includes/capabilities.php	🔗	Cap::REPORTING_JOB_HEALTH, Cap::REPORTING_JOB_SETTINGS - everyday-work permissions with restrict => null (always apply)
api/reporting/job_health.php	📖✏️	The tab's API: `view=health
reporting/logs/index.php, lang/en/reporting.php (jobs.*)	🖥️	The Background jobs tab (?tab=jobs)
watchtower/index.php, system/index.php	🖥️	The card and the notice - server-rendered from jobHealthSummary()
assets/js/mobile.js	🖥️	The two job tables as cards on a phone (data-log-type="jobs" / "job-runs")
api/v1/resources/jobs.php, api/v1/lib/routes.php, api/v1/lib/permissions.php, api/v1/spec.json	🔌	GET /jobs/health, permission job_health.read
reporting/help.php, system/help/roles.php	🖥️	In-app help
includes/feature_bingo/cards/reporting.php	🖥️	Three cards
tests/job-health.php	🧪	The rules and the recorder, plus REST with a base URL

What you do not touch to add a job: the tab, the API, Watchtower or the REST address. They read the registry.

---

2. Recording a run

require_once __DIR__ . '/../includes/job_runs.php';
$trigger = JobRun::scheduledTrigger();            // 'cron' from the CLI, 'web' over HTTP

// Turned away at the door - NOT a run:
JobRun::refused($conn, 'my_job', 'wrong token');

// Did no work - recorded, but never read when deciding whether a job is failing:
JobRun::skipped($conn, 'my_job', $trigger, 'Minimum interval: last run 12s ago.');

$runId = JobRun::start($conn, 'my_job', $trigger);
foreach ($sources as $s) {
    // ... work ...
    JobRun::source($conn, $runId, 'my_job', 'thing:' . $s['id'], $s['name'], $ok, $message);
}
JobRun::finish($conn, $runId, null, ['done' => $n], "Did {$n} things.");   // null outcome = work it out from the sources

• finish() with a null outcome: every source failed is failed, some failed is partial, otherwise ok. Pass an outcome to override (the SLA job passes partial when some emails failed).
• JobRun::wrap($conn, $job, 'manual', fn($runId) => ..., $analystId) does start/finish/fail around a closure - for page and manual paths.
• Every method swallows its own errors. An install that has not run Database Verification has no job_runs table and its SLA emails must still go.
• A run that dies stays running. A shutdown handler marks it failed where PHP gets the chance, and the next finished run of the same job marks any older running row past max_runtime as failed ("Never finished") - so a crash becomes a recorded failure, and stuck only shows while nothing has finished since.
• JobRun::current() is the newest run started in the request. TaskHistory::record() stores it on events written by the system, so an occurrence the scheduler creates says which run made it (#181).

Triggers

trigger_kind	Set by	Counts for freshness?
cron	JobRun::scheduledTrigger() under CLI	✅
web	JobRun::scheduledTrigger() over HTTP (the token address)	✅
page	the opportunistic page-load runs	❌
manual	Run now, Sync now, one directory run by hand	❌

---

3. The health rules, and where each lives

All in jobHealthOne(). In order of the issue's clarifications:

Rule	Implementation
1. Freshness reads only scheduled runs	last_scheduled_success = newest ok/partial run with trigger_kind IN ('cron','web'). Never seen if none; overdue if older than interval + grace (grace = max(registry grace, interval / 2)).
2. Failure reads only real runs	"Failing" = the newest run whose outcome is ok, partial or failed - skipped rows are never in the query.
3. Recovery is per source	The newest job_run_sources row per source_key; a failed one counts while the key is in the registry's sources($conn) list (gone or switched off = no longer expected). A job with no sources callable uses its newest real run's partial outcome instead.
4. Not used here	job_health_config[job].not_used - returns not_used before anything else, with who and when.
Stuck	Newest running row older than max_runtime, with no real run started after it.
Not needed	The registry's needed($conn) - the same test the feature uses (e.g. repeating tasks: an active mode='schedule' series).

A job can have several problems (failing and overdue); problems lists them, and status is the most severe (stuck > failing > overdue = never seen > partial).

---

A failed lookup is "unknown", never "not needed" (Dr4inder's review, 20 Oct)

Every lookup the health check makes used to turn an error into false / empty: a database that could not be read made every job Not needed and jobHealthSummary() said healthy - reproduced exactly as reported, with a PDO subclass whose every read throws. A monitor reading ?strict=1 would have stayed green through the very failure it watches for.

Now only a missing table (SQLSTATE 42S02 - the feature has never been set up on this install) is an answer. Any other failure throws JobHealthUnknown (includes/job_registry.php), and the job's status is unknown - severity 6, the worst, and in JOB_HEALTH_UNHEALTHY, so the Watchtower card, the System notice and ?strict=1 (503) all react to it.

Lookup	Helper	problems[0].what
Whether the job is needed	jobNeedOne() / jobSettingStrict() in each needed callable	requirement / setting
Which sources still exist	jobSourceKeys() in each sources callable	sources
Each source's newest result	the job_run_sources read in jobHealthJudge()	source health
Run history	the job_runs reads	history
Intervals and not used here	jobHealthConfig($conn, strict: true) in jobHealthAll()	configuration (every job)

jobHealthUnknownResult() builds the result. It names what could not be read, never the database's message - that goes to the server log, not to a monitor or a page. Not needed and source removed now require a lookup that succeeded and said so.

🔴 TRAP: never catch-and-default in a needed or sources callable. Use jobNeedOne(), jobSourceKeys() and jobSettingStrict(); they draw the missing-table line for you. jobSetting() (lenient) is for display only.

4. Adding a background job

1. Registry - add it to jobRegistry() with its command, interval, grace, max_runtime and a needed callable that survives a missing table (use jobNeedOne()). Add sources if it works through several things.
2. Strings - reporting.jobs.name_<key> and reporting.jobs.affects_<key> in lang/en/reporting.php.
3. Record - refused on a token refusal, skipped for lock / interval / switched off / dry run, start / source / finish around the work, fail in the catch. Exit non-zero on failure.
4. Page and Run now paths - record them too, as page / manual. If you skip this the history is incomplete, but freshness is unaffected.
5. Docs - Scheduled tasks (all four tables: the list, Windows, Linux, Docker).
6. Test - php tests/job-health.php; run the job once by hand and check the tab.

---

5. The REST address

apiJobsHealth() returns status, checked_at, unhealthy (keys) and jobs (apiJobHealthRow(): no command lines, no internal ids). ?strict=1 makes it apiRespond($data, 503) while unhealthy.

🔒 Install-wide, so the gate is too: the key's company_scope must be null or cover every tenant, and its analyst must be an administrator or reach every company. A key for one company is refused even with job_health.read.

---

6. Traps

• TRAP: the old "last run" settings are not health. webhook_cron_last_run, workflow_cron_last_run and friends are written before the work, for the minimum-interval guard. A crash after that line looks like success. They are kept for the guards; health reads job_runs only.
• TRAP: trigger is a reserved word in MySQL. The column is trigger_kind.
• TRAP: summaries are shown to people and returned by the REST address. Never pass a raw exception with a URL or a credential straight through - finish() / source() / fail() run JobRun::clean(), which blanks user:pass@, query strings, password=..., and long hex or base64 runs. Do not bypass them with SQL.
• TRAP: a wrong-token call is not a run. Recording one per request would let anyone fill the table; refused() overwrites one setting per job.
• TRAP: needed must mirror the feature's own test. Calendar pull is needed only where two-way sync (mode='push') is on - an install that only hands out subscription links used to be warned for nothing.
• TRAP: a page-load run must never refresh freshness. That is the whole of Dr4inder's first clarification: "healthy, but only page loads have run it" let a status-code monitor stay green while cron was gone.
• TRAP: Run now on one Proxmox / vCloud server or one directory records ONE source. It can clear that source's failure (it processed it), and correctly says nothing about the others.

---

7. Tests

php tests/job-health.php [base-url] - uses a made-up job key, zz_test_job, with its own definition, so it never touches a real job's health:

• rule 1 with positive controls: page-only is never seen; a scheduled success 2 minutes ago is healthy; 2 hours ago is overdue although Run now and a page ran since;
• rule 2: failed then two skips is still failing; a real success clears it;
• rule 3: A failed, a later run processed only B, still partial naming A; A succeeding clears it; A no longer expected clears it;
• stuck, the "never finished" sweep, clean();
• with a base URL: /jobs/health 200, ?strict=1 503 on an unhealthy install, a key without the permission 403. The temporary key is deleted.

---

• Failed lookups (tests/job-health.php, last section): Dr4inder's reproduction (every read throws: now unhealthy, 13 unknown), and a SelectiveFailPDO that fails only chosen reads on the real database - a requirement lookup, the source-health read, the source list, the job settings, the run history - each unknown, with controls that a missing table (42S02) still reads as not needed and that nothing failing means nothing unknown.

Related pages

• Background jobs · Background job health - design (#182) · Scheduled tasks
• Repeating task evidence - Developer Guide - which reads this history
• Roles - Developer Guide · REST API · Developer tests