Background Services

Background Services

Some of what HybridCore does can't happen while a visitor waits for a page. Querying your game servers takes seconds; sending an email can fail and need a retry; resizing an avatar is slow. All of it runs in the background instead, which means two processes have to be running on your server — plus two more if you turn on live updates or server-side rendering.

Without them the site still loads — and quietly stops being current.

On this page: What they do · What breaks without them · Setting them up · Checking they work · After deploying · Troubleshooting

Service	Needed	Fails by
Scheduler	Always	Data going stale
Queue worker	Always	Email never arriving
Websocket server	Only with live updates	Updates waiting for a page load
Renderer (SSR)	Only with SSR enabled	Pages just getting slower, silently

What they do

The scheduler

Runs jobs on a clock — every minute it checks what is due:

Job	How often	What it does
Server queries	Every minute	Asks each game server for its player count, map and ping
Uptime tracking	Every minute	Records whether each server answered, building the uptime history
Bridge command prune	Daily	Deletes delivered and expired bridge commands
Bridge event prune	Daily	Drops telemetry older than 30 days so the table stops growing
Email digest	Weekly	Sends the news and server summary to subscribers

The queue worker

Picks up jobs the moment something creates them:

Job	Created when
Sending email	Someone registers, resets a password, receives a message
Avatar and image processing	A user uploads a picture
Bridge event dispatch	A game server reports telemetry
Extension jobs	Giveaway draws, vote rewards, store deliveries

The scheduler decides when; the worker does the work. They are separate because a job that takes 30 seconds must not delay the minute tick.

The websocket server (optional)

Only if you use live updates. It keeps a connection open to every visitor's browser so notifications, unread counts and the typing indicator in messages arrive without the page asking. Turned off, those still work — they just wait for the next page load instead.

The renderer (optional)

Only if you turn on server-side rendering. Without it the browser has to download and run the site's JavaScript before anything appears; with it the server sends finished HTML and the page shows up roughly 700ms sooner on a phone. Nothing else changes — the page behaves identically once loaded.

This one fails differently from the others: if it stops, pages still render correctly, just more slowly, and nothing anywhere says so. See Troubleshooting for how to spot it.

What breaks without them

This is the part worth understanding, because nothing announces itself as broken:

No scheduler:

• Player counts freeze at whatever they were — the server browser shows stale data
• Uptime history has a gap for the whole outage
• Bridge tables grow without limit
• No weekly digest goes out

No queue worker:

• No email is ever sent. Registration confirmations, password resets — all queue up and wait. Users see "check your inbox" and nothing arrives.
• Uploaded avatars stay unprocessed
• Bridge telemetry is stored but never handled — events accumulate in the database and no extension ever sees them, so stats and rewards silently stop
• Giveaway draws don't run

No websocket server (if you enabled it): notifications and unread counts stop updating live. They still appear on the next page load.

No renderer (if you enabled SSR): pages get slower. Nothing else — no error, no log line, no broken page.

The email one catches most people out: everything looks fine until someone tries to reset their password.

Setting them up

HybridCore generates the service files for you, filled in with your real paths and user:

cd /path/to/hybridcore
php artisan hybridcore:systemd

It prints a unit for each service your install actually needs — the websocket server and the renderer only appear if you have switched them on, so you can paste everything it gives you without thinking about it. Add --all to see the ones it skipped.

Paste each unit into /etc/systemd/system/ under the filename it names, then run the systemctl lines it prints at the end. They look like:

sudo systemctl daemon-reload
sudo systemctl enable --now hybridcore-scheduler hybridcore-worker

enable makes them start on boot; --now starts them immediately.

Run the command as your normal user, not root. It reports the user it sees, and that user goes into the unit file. If the services run as root, everything they write into storage/ becomes root-owned and the web server can no longer write there — which breaks the site in a way that is genuinely hard to trace.

Why systemd and not cron

Older Laravel guides tell you to add schedule:run to a crontab and run the worker under Supervisor. That works, but:

• A crontab entry that fails fails silently — nothing tells you
• It needs two different tools for two similar jobs
• Neither restarts cleanly after a crash or a reboot

The scheduler service runs schedule:work, a foreground process that fires the scheduler every minute itself. systemd restarts it if it dies, starts it at boot, and gives you systemctl status and journalctl for both processes. If you had a crontab entry from an older setup, remove it — otherwise the scheduler runs twice.

Checking they work

The scheduler and worker write a heartbeat every minute. Admin → Health shows them, along with a row for the websocket server:

• Green — heard from within the last minute
• Amber — stale, the process has stopped or is stuck
• Missing — never started

From the shell:

systemctl status hybridcore-scheduler hybridcore-worker
journalctl -u hybridcore-worker -f     # live log

The renderer has no heartbeat. Check it by viewing the page source of your home page — view-source:https://your-domain.com. If SSR is working you will see the real page markup; if it is down you get an empty <div id="app"> and the browser builds the page itself.

After deploying new code

Restart the worker. A worker holds your code in memory from the moment it started, so it keeps running the old version indefinitely:

sudo systemctl restart hybridcore-worker

Restart the renderer too, if you use it. It holds the built JavaScript bundle in memory the same way, so a deploy without a restart serves the previous build's markup into the new page.

The scheduler picks up changes on its own, but restarting everything is simpler to remember:

sudo systemctl restart hybridcore-scheduler hybridcore-worker hybridcore-ssr

Forgetting this is the usual reason a fix "didn't work" after a deploy.

Troubleshooting

Service won't start

journalctl -u hybridcore-worker -n 50

Usually the PHP path or the working directory in the unit is wrong. Re-run php artisan hybridcore:systemd and compare.

Health shows a heartbeat but jobs don't run

The worker is running against a different queue connection than the app. Check QUEUE_CONNECTION in .env and restart the worker.

Emails still don't arrive with the worker running

The jobs are being processed but the delivery is failing. Check the failed jobs:

php artisan queue:failed

Jobs run twice

You have both a crontab entry and the scheduler service. Remove the crontab entry.

The renderer service won't start

Almost always the runtime. The service runs node against the built bundle, and systemd gives it a much barer PATH than your shell — a node installed through nvm is not on it. php artisan hybridcore:systemd writes the absolute path into the unit for you, so regenerate it and compare. Failing that, set it yourself in .env:

INERTIA_SSR_RUNTIME=/usr/bin/node

Pages are slow and view-source shows an empty <div id="app">

The renderer is down, or INERTIA_SSR_ENABLED is not true. The site is not broken — it has fallen back to letting the browser build the page, which is what it did before you enabled SSR.

systemctl status hybridcore-ssr
journalctl -u hybridcore-ssr -n 50

A deploy changed the page but the old version still shows in view-source

The renderer is still holding the previous bundle:

sudo systemctl restart hybridcore-ssr

Worker keeps restarting

Look at the log. A job throwing on every attempt will exhaust --tries and land in failed_jobs; a worker dying repeatedly is usually running out of memory, which the hourly --max-time recycle is there to prevent.

---

See also: Installation · Updating & Maintenance · Game-Server Bridge