Background Jobs

⏱️ Background jobs

Reporting → System logs → Background jobs shows every scheduled job FreeITSM relies on, whether each one is doing its work, and every run it has made. It answers one question that nothing else in FreeITSM could: has anything quietly stopped?

Why this exists. A background job that stops does not announce itself. Nothing errors and nothing appears in a log - SLA warnings just stop arriving, webhooks sit in a queue, repeating tasks stop appearing. The first sign is usually somebody asking why. This page, a card on Watchtower and a notice on the System page make that visible, and a health address lets an outside monitoring tool raise the alarm on its own. Requested in #182; the design is Background job health - design.

• Setting the jobs up is a different page: Scheduled tasks lists every job, what it needs and how to schedule it on Windows, Linux and Docker.
• Developers: Background jobs - Developer Guide.

---

1. The health of every job

The top half of the tab lists the jobs, the ones that need attention first. Each shows what stops without it, its health, the last time its schedule ran it successfully, its last run of any kind, and how often it is expected to run.

Health	Means	What to do
Healthy	Its schedule ran it successfully within its expected interval plus some slack	Nothing
Overdue	No scheduled run has succeeded for longer than that	Check the job is still scheduled - see Scheduled tasks
Failing	Its most recent real run failed - the message says why	Read the message; the run history has the detail
Partial	Part of its work failed and has not succeeded since - one tracker connection, one directory, one server, one calendar. It is named	Fix that source; the job recovers when that source next works
Stuck	A run started longer ago than any normal run takes, and never finished	Look for a hung process; the next finished run records the stuck one as failed
Unknown	FreeITSM could not read something it needs to judge the job - whether it is needed, its settings, its run history or how its sources last ran. Usually the database being unavailable. Shown as a problem, and the strict health address answers 503, rather than assumed fine	Check the server's error log
Never seen	The job is needed here, but no scheduled run has ever been recorded	Schedule it. The tab shows the command to schedule
Not needed	The feature it serves is not in use on this install - no fixed-schedule repeats, no tracker connections, LMS reminders switched off	Nothing
Not used here	Somebody has said this install does not run it, on purpose	Nothing - who said so, and when, is shown

Worked out when you open the tab. Nothing has to run for a job to turn overdue - time passing is enough. That is the point: a job that has stopped cannot tell you it has stopped.

After upgrading, every job you need shows Never seen until its schedule next runs it, because run history starts on the day this was added. A job you run every few minutes turns healthy within minutes; a daily one, by tomorrow.

---

2. The rules, in plain words

Three rules decide health. They were agreed on the issue, and they are what make the page trustworthy rather than reassuring.

1. Only the schedule proves the schedule. Some jobs also run when somebody opens a page (Domains, LMS reminders, project alerts) or presses Run now. Those runs do the work, and they appear in the history - but they never keep a job healthy. A job whose cron entry has gone turns overdue however often somebody opens its page, and the tab says "It has run since, but only by page load".
2. Nothing but success clears a failure. A run that did no work - turned away by its lock or minimum interval, switched off, or a dry run - is shown as Skipped, and never makes a failing job look healthy, however recent it is.
3. A failing source stays failing until that source works. If one of four directories fails, the job is Partial and names that directory until a run processes that directory successfully - not merely until some other run succeeds. Switching the source off, or deleting it, also clears it: it is no longer expected.

A call turned away at the door - the job's web address called with the wrong token - is not a run at all, because anyone can make one. The job shows "Last call turned away: wrong token" instead, which is the clue when a scheduler is still calling an old address after the token changed.

---

3. The run history

The bottom half lists every run, newest first, filtered by job, outcome, what started it and period. Click a run to see its counts, its message and the result for each source it worked through.

Started by	Means
Schedule (command line)	cron or Windows Task Scheduler running the script
Schedule (web address)	a scheduler or monitoring tool calling the job's token address
Page load	the job ran in passing when somebody opened its page
Run now	somebody pressed Run now (or Sync now, or ran one directory by hand)

Outcome	Means
OK	It worked. A run that found nothing to do is OK
Partial	Some sources worked and some did not
Failed	It did not do its work
Skipped	It did no work: its lock or minimum interval turned it away, the feature is off, or it was a dry run
Running	It is running now

Kept for 13 months, so a yearly audit can still read it - repeating task evidence uses it to explain missing occurrences. Skipped runs, which can arrive every minute, are kept for 30 days.

Messages never contain secrets. A job's summary is written by the job, and anything that looks like a password, a token or a URL's query string is blanked before it is stored.

---

4. Being told

A page only helps somebody who looks at it. So while anything needs attention:

• Watchtower shows a Background jobs card above the others, naming each job and what it affects. It appears only while something is unhealthy, and only for people allowed to see job health.
• The System page shows an amber notice to administrators.

Both are worked out when the page is opened, so neither depends on any job running.

Why no email or bell notification? Sending one would itself need a background job - the very thing that may have stopped. The reliable alarm is an outside monitor (below); Watchtower and the System notice cover people who are already in FreeITSM.

---

5. Outside monitoring - the health address

GET /api/v1/jobs/health
GET /api/v1/jobs/health?strict=1

Call it with an API key holding the Background job health - read permission (System → API). It returns every job's health, last scheduled success, last run and failing sources, and an overall healthy or unhealthy. It never runs a job and never returns passwords, tokens or command lines.

• ?strict=1 answers HTTP 503 while any needed job is unhealthy, and 200 otherwise - so a monitoring tool that only looks at the status code still raises the alarm.
• If FreeITSM or its database is down, the address fails - an error, never a cheerful "healthy" - which is itself the alarm.
• Install-wide, so the key must be too. Job health covers every company, so the key, and the analyst it acts as, must reach every company (or the analyst must be an administrator).

Examples - a check every five minutes from Uptime Kuma, Zabbix, Nagios or a plain script:

curl -fsS -H "Authorization: Bearer fitsm_..." "https://itsm.example.com/api/v1/jobs/health?strict=1" > /dev/null \
  || echo "FreeITSM background jobs need attention"

The full reference is on REST API.

---

6. Telling it what to expect

Edit on a job (with the permission below) does two things:

• Expected to run every - how often your scheduler actually runs it. The default comes from Scheduled tasks: the SLA check every 5 minutes, repeating tasks daily, and so on. If you run the SLA check every 15 minutes, say so, or it will show overdue between runs. The slack before it turns overdue grows with the interval.
• Not used here - for a job this install deliberately does not run. Its health is no longer checked, and the tab records who marked it and when, because it silences that job's alarm.

---

7. Who can see it

Permission (System → Roles)	Lets someone	Notes
View background job health	Open the tab, see the Watchtower card	Also needs Reporting module access and access to every company - job messages name directories, servers and tracker connections that belong to all of them
Change what background jobs are expected to do	Edit intervals and mark jobs not used here	Sensitive: it can silence an alarm

Administrators hold both. Nobody else sees the tab until a role grants it, so upgrading changes nothing for them.

---

8. Docker

The FreeITSM image runs no scheduler of its own, so on a Docker install every job shows Never seen until the host runs them. Schedule them from the host with docker exec - see the Docker section of Scheduled tasks.

---

Related pages

• Scheduled tasks - setting each job up
• Background jobs - Developer Guide
• Background job health - design (#182)
• Repeating task evidence - which reads this history
• Reporting · Watchtower · REST API · Roles and permissions