Background Job Health Design

Background job health - design (#182)

Status: proposal - nothing here is built yet. This page sets out what I plan to build for issue #182, and why, written before any code so the approach can be checked first. Comments on the issue are welcome. The open decisions are in section 7. It is designed alongside #181, and section 4 says how the two fit.

The request, from issue #182 (raised by @Dr4inder):

"A scheduler outage can otherwise leave the application looking quiet while expected work, notifications or synchronisation never occur."

In plain words

Several parts of FreeITSM rely on background jobs that the server runs on a timer - a cron entry on Linux, a scheduled task on Windows: sending SLA warnings, delivering webhooks, creating repeating tasks, syncing calendars and directories, checking domains. If the timer stops - a server rebuild, a moved install, a Docker container with no scheduler - nothing tells you. No error appears; things just quietly stop happening. The first sign is usually somebody asking why an SLA warning never came.

The request is for one place that shows every background job: which ones this install needs, when each last worked, whether any is overdue, failing or stuck - plus a health address an outside monitoring tool can check, so an alarm can go off without relying on FreeITSM itself.

---

1. The principle: measure, never schedule

• FreeITSM does not become a scheduler. The jobs keep running exactly as they do now - from cron, Windows Task Scheduler, or a monitoring tool calling their web address. This work only records each run and reads those records.
• "Last success" means the work finished. A job that starts and crashes, is turned away by its lock or minimum interval, fails its token check, or is a dry run, never refreshes it. A run that finds nothing to do is a success.
• Health is worked out when you look. Whether a job is overdue is calculated at the moment the page or the health address is opened - so a stopped job shows as overdue because it has stopped, not because it ran again to say so.
• Honest about what it cannot see. FreeITSM cannot inspect the server's crontab. A job with no recorded run is "never seen" - not "the cron entry is missing", which it cannot know.

---

2. What FreeITSM records today

Mapped from the code (3.9.0). There are 13 background jobs and no shared way of recording them:

Job	What depends on it	What it records about its own runs
SLA breach check	SLA warning and breach emails	✅ A proper run log (sla_cron_runs): start, finish, outcome, counts
Webhook deliveries	Outbound webhooks	⚠️ A "last run" time written before the work
Scheduled workflows	Contract and warranty expiry workflows, planned maintenance, project alerts and digests	⚠️ Written before the work; failures in its parts are swallowed and it still says OK
Integration poll	Jira / tracker status and comments	⚠️ Written before the work; errors land on each linked ticket, not the connection
Repeating tasks	Tasks on a fixed schedule	⚠️ Written before the work
Attachment reading	Searching inside attachments	⚠️ Written before the work
Calendar sync pull	Changes made in analysts' calendars	❌ Nothing at job level
Domains	Lookups, checks, alerts	⚠️ One time, written by the cron and by page loads and Run now alike
LMS reminders	Training reminder emails	⚠️ Same - cron, page loads and Run now look identical
Proxmox sync	Proxmox inventory	⚠️ Per server only
VMware Cloud Director sync	vCloud inventory	⚠️ Per server only
Documents maintenance	Document search, file tidy-up	❌ Exit code only
Directory sync	LDAP / AD / CardDAV people	✅ A run log per provider (directory_sync_runs)

The useful pieces that already exist - the SLA run log, the directory-sync per-provider log, the webhook page's running / stale / down judgement, the per-server sync status columns - are the model for what follows. Reporting → System logs describes itself as covering "background processes, scheduled tasks", but no job writes to it yet.

Docker: the image runs no scheduler at all, so on a fresh Docker install every job is "never seen" until the host is set up to run them - worth saying plainly on the page and in the docs.

Not a background job: fetching email into tickets is done by the browser while the ticket list is open, not by a scheduled task. It has its own mailbox health warnings already, and it stays outside this work (section 8).

---

3. The proposal, in six parts

Part A - One run recorder for every job

A new job_runs table and a small shared helper that every job calls at the start and the end of its work. One row per run:

Column	Holds
job	which job (a fixed key such as task_recurrence)
trigger	cron (command line), web (its token address), page (run in passing by a page load) or manual (Run now)
started / finished	UTC times, and the duration
outcome	ok, partial (some sources failed), failed, skipped (lock, minimum interval or token refused), or running while it works
counts	what it did - "12 sent, 0 failed", "3 occurrences created"
summary	a short message written by the job, never a raw error that could carry a password or token

Jobs that work through several sources - tracker connections, directory providers, analysts' calendars, Proxmox and vCloud servers, domains - also write one source result each (job_run_sources: which source, ok or failed, a short message). That is the issue's point that "partial failures should remain visible rather than being hidden by an overall successful invocation": one failing directory provider makes the run partial, named, even though the other three worked.

The SLA job's existing log becomes the first user of the shared recorder; its own table stays for its own screen. The existing "last run" settings stay too, so nothing that reads them breaks - but health no longer reads them, because five of them are written before the work starts.

Retention: ok, partial and failed runs are kept for 13 months by default, long enough for a yearly audit and for #181's report (section 4); skipped runs, which can arrive every minute, for 30 days.

Part B - What each job is expected to do

A job registry - one file, like the currency list - naming every job with:

• what depends on it, in plain words;
• when it is needed: the same test the job itself uses - repeating tasks only when a series runs on a fixed schedule, the tracker poll only when a connection is active, LMS reminders only when they are switched on;
• its expected interval (from the docs: SLA 5 minutes, webhooks 1 minute, repeating tasks daily...) and a grace period;
• its longest normal run, beyond which a run still marked running is stuck.

An administrator can change a job's expected interval, because installs schedule differently, and can mark a job not used here on purpose.

Each job's health is then worked out on reading:

Health	Means
Not needed	The feature it serves is not in use on this install
Not used here	An administrator has said so
Never seen	Needed, but no run has ever been recorded
Healthy	Last success within its interval plus grace
Overdue	Last success older than that
Failing	Its most recent run failed (the last success is still shown)
Partial	Its most recent run worked for some sources and not others, which are named
Stuck	A run started longer ago than its longest normal run and never finished

Page loads are not proof of a schedule. Domains, LMS reminders and project alerts also run in passing when somebody opens their pages, which can keep a job looking alive while its cron entry has gone. So a job whose recent runs were all page or manual shows "Healthy - but only page loads have run it; no scheduled run seen since {date}".

Part C - The page

Reporting → System logs → Background jobs - a third tab beside User logins and Email imports, where the logs already live.

• At the top, the health of every job: name, what depends on it, health, last success, last run and its outcome, expected interval, and how it was triggered. Unhealthy jobs first.
• Below, the run history: every run, filterable by job, outcome and period, each opening to its counts, message and source results.
• For "never seen" and "overdue", a short note on what to check - with a link to Scheduled tasks, which says how to set each one up on Linux, Windows and Docker.

Who can open the tab is a decision for section 7.

Part D - Being told

A log page only helps somebody who looks at it, and the point of the issue is that a stopped job is silent. So:

• a Watchtower card, shown only while a job is unhealthy, naming the jobs and what they affect;
• a notice on the System page for administrators, in the same way as the other System warnings.

Both are worked out when the page is opened, so neither depends on the scheduler being alive.

Part E - A health address for outside monitoring

GET /api/v1/jobs/health on the REST API, using an API key that holds a new "Background job health - read" permission. It returns, for every job: its health, last success, last run and outcome, expected interval, and the summary and failing sources - and an overall status.

• It never runs a job and never returns passwords, tokens or connection details.
• For simple uptime monitors, ?strict=1 answers HTTP 503 whenever anything needed is unhealthy, and 200 otherwise - so a tool that only looks at status codes still raises the alarm.
• If FreeITSM or its database is down, the address fails - an error, never a cheerful "healthy" - which is itself the alarm (acceptance criterion 6).
• Job health is install-wide, so a key that can see only some companies is not enough: the key's analyst must be an administrator or reach every company.

Part F - Fixes found while mapping it

These would make the health page wrong, so they come first:

• Five jobs write "last run" before doing any work (webhooks, scheduled workflows, the tracker poll, repeating tasks, attachment reading) - so a crashed or empty run looks healthy. The recorder replaces this for health.
• The scheduled workflows job swallows failures in its parts (planned maintenance, project alerts, digests) and still says OK. Each part becomes a source result.
• The SLA job records a run that sent some emails but hit errors as plain success. It becomes partial.
• LMS reminders can never be run by web address: its token is never created, and is not read back correctly. Fixed, and added to the token tool.
• Tracker poll errors are written to every linked ticket rather than to the connection, and System → Integrations does not show when a connection last polled.
• Calendar sync's health treats "every pull is failing" the same as "never run", and warns on installs that only use subscription links.
• Two warnings say mail is collected by a scheduled task, which does not exist.
• Calendar sync, Proxmox and vCloud jobs have no protection against overlapping runs.
• The docs disagree on how many jobs there are ("nine", "four") and on intervals; Scheduled tasks is missing four jobs and has no Docker section.
• Jobs that fail exit with success (webhooks, workflows, tracker poll), so a scheduler cannot tell either. They will exit non-zero.

---

4. How this fits with #181

The issue makes the point itself: "a healthy scheduler does not prove that a control was completed, and a completed task does not prove that the scheduler is healthy." The two designs are built to answer each other:

• #181's report can say why. It shows a repeating task's occurrence as Missing or Created late. With #182's run history for the repeating-tasks job, it can add the cause: "Missing - the repeating-tasks job did not run between 2 and 9 March", or "Created late - caught up on 9 March after the job was overdue". That is the auditor's next question, answered on the same page.
• #181's history links to the run. When the scheduler creates an occurrence, #181's task history records created by the repeat - on schedule / catch-up; it now also records which job run created it.
• Retention is set for audits. Run history is kept for 13 months so a yearly evidence report can still read it.

#181's design has been updated with these three points. Build order: #182's recorder (Part A) should land before #181's report, so the report has run history to read.

---

5. The issue's acceptance criteria

#	Criterion	Where
1	Identify the required jobs, their cadence and last successful processing	Parts B, C
2	A stopped repeating-task scheduler turns unhealthy after its interval and grace, with no new record	Part B (worked out on reading)
3	Never seen, overdue, failed, partial, stuck and intentionally inactive are distinguishable	Part B
4	Skipped and failed runs keep the previous success; empty successful runs count	Parts A, B
5	Diagnostics name the job or source without secrets	Part A (summaries written by the job, source results)
6	An outside monitor can detect unhealthy jobs on its own	Part E
7	Existing cron, command-line and web paths keep working; no second scheduler, no duplicate runs	Section 1, Part A

---

6. The order to build it in

1. Record - the recorder and registry (Parts A, B) and the fixes (Part F). Every job starts writing proper history from this release.
2. See - the Background jobs tab (Part C), the Watchtower card and System notice (Part D).
3. Monitor - the health address and its permission (Part E).

Phase 1 also unblocks #181's report (section 4).

---

7. Open decisions

#	Decision	My recommendation
1	Where the page lives	Reporting → System logs → Background jobs, beside the existing log tabs - with the Watchtower card and System notice so nobody has to go looking.
2	Who can open it	A new permission, "View background job health", rather than everyone with Reporting: job messages name mailboxes, directories and servers. It also fits #181's Auditor role.
3	Expected intervals	Built-in defaults from the docs, changeable per job by an administrator, plus not used here.
4	Telling somebody by email or the bell when a job turns unhealthy	Not in the first version. Sending that message would itself need a background job - the very thing that may have stopped. The health address and an outside monitor are the reliable alarm; Watchtower and the System notice cover people already in FreeITSM.
5	Retention	13 months for real runs, 30 days for skipped ones - long enough for a yearly audit.
6	Docker	Document the host's cron for Docker installs in Scheduled tasks. A scheduler inside the container could come later, as its own change.

---

8. Deliberately out of scope

• No second scheduler and no new way to run jobs - the existing Run now buttons stay as they are.
• No monitoring service, database or job framework - the issue asks for none, and outside tools consume the health address.
• Email fetching stays browser-driven with its existing mailbox health warnings; it is not a background job today.
• Nothing here proves that the work a job should have produced was done - that is #181's job, for repeating tasks.

---

Related pages

• Scheduled tasks - how each job is set up
• Recurring task evidence - design (#181)
• Webhooks · REST API