Troubleshooting

Known problems with server-prep, grouped by area: what you see, why it happens, and the command that fixes it.

Before you start

• Most fixes re-run one step from your saved answers. That file has no secrets:

  sudo ./server-prep.sh --config /etc/server-prep/server-prep.conf --only <step> --yes

  --list-steps prints the step names. Every step is safe to run again: it only changes what is missing.
• Do not use --yes for the ssh step. Today --yes skips the SSH safety check and stops the 10-minute auto-rollback (#65). Run it without --yes and answer the questions.
• Keep a console open the first time: your provider's web console on a VPS, or IPMI / KVM / a screen on a dedicated server.
• After a fix, run the audit again. It changes nothing:

  sudo ./server-audit.sh

  A prepared server shows about 43 PASS and 0 FAIL.

Read the log and the report

Each run writes a log and a report. Both are readable by root only.

What	Where
Log of every command	/var/log/server-prep/run-<date>.log
Report (Markdown)	/root/server-prep-reports/server-prep-report-<host>-<date>.md, plus a copy in your home
Report (JSON)	the same path ending in .json, plus a copy in your home
Backups and rollback.sh	/var/lib/server-prep/backups/<date>/
Saved answers	/etc/server-prep/server-prep.conf

List the logs, newest first:

sudo ls -t /var/log/server-prep/

Show only the errors and warnings of the last run:

sudo sh -c 'grep -n -i -E "error|fail|warn" "$(ls -t /var/log/server-prep/run-*.log | head -1)"'

The report has a Still to do by hand section. Read it first: it lists what server-prep could not do for you.

Warning

Logs written by version 1.1.1 and older can contain the Cloudflare tunnel token (#64). Never paste the whole log in an issue or a discussion. Copy only the lines you need, and check them for tokens, keys, IPs and host names.

Roll back

SSH changes undo themselves. When the ssh step changes the config, it starts a 10-minute timer. If you don't confirm that a new login works, the timer restores the old SSH config. You can check the timer while it runs:

systemctl list-timers 'server-prep-ssh-rollback-*'

You see one line while the timer is waiting, and none once you confirmed or it ran.

To restore only the SSH config by hand, from the console:

sudo bash /var/lib/server-prep/backups/<date>/rollback-ssh.sh

Full rollback. Each run writes rollback.sh next to its backups. It copies back every file the run changed and deletes the files it created. When SSH goes back to its old port, it first allows that port in UFW from the same addresses as the run's SSH rule; when that port is 22, it also stops Endlessh and deletes the Endlessh rule (endlessh tarpit), which allowed port 22 from anywhere. When the run's only SSH source was the Cloudflare Tunnel, UFW gets no rule and the Endlessh rule stays: the script says so. It restarts SSH only when sshd -t accepts the restored files; otherwise it stops there and SSH keeps running as it is. It does not remove packages, undo service changes or turn UFW off: when it starts, it prints what it leaves in place (the timezone, the swap file, the boot target, the AIDE and rkhunter baselines...). Read it before you run it:

sudo less /var/lib/server-prep/backups/<date>/rollback.sh

Caution

Run rollback.sh from the console only, never over SSH: SSH restarts with its old port and settings, and your panel loses its SSH access. A rollback.sh written by v1.1.1 or older does not stop Endlessh or open the old port in UFW (#71), so with SSH going back to 22, run sudo systemctl disable --now endlessh first, then allow the port, for example sudo ufw allow from 192.168.1.50 to any port 22 proto tcp. After the rollback, delete the Endlessh rule, which leaves port 22 open to everyone: sudo ufw status numbered, then sudo ufw delete <number> for each endlessh tarpit line, highest number first.

Reaching a new server

From the first setup with server-preconfig.sh (see server-preconfig.md):

Symptom	Cause	Fix
Connection refused on SSH	openssh-server is not installed	at the console: apt install -y openssh-server
Permission denied as root	Debian refuses root password login over SSH	log in with your own user
Could not open lock file … are you root?	in sudo a && b, only a runs as root	sudo a && sudo b
Authentication failed at [sudo: authenticate]	a token or the next command was pasted at sudo's password prompt	type the server password, and run commands one at a time
sudo: unable to resolve host <name>	the hostname changed before /etc/hosts	harmless: the script updates /etc/hosts first
No internet or SSH after a power cut	the IP was set by hand and never saved	sudo ./server-preconfig.sh --static-ip current
Some peers are advertising routes but --accept-routes is false	another device shares a subnet	ignore it; don't turn on --accept-routes on a LAN server
Server not visible in Tailscale	logged in with another account, or waiting for approval	tailscale switch --list, or approve it in the admin console under Machines
/etc/resolv.conf says "generated by tailscale"	MagicDNS (Tailscale's DNS) is on	normal; the old DNS is in /etc/resolv.pre-tailscale-backup.conf
WARN Tailscale key expires <date>	Tailscale keys expire by default	Tailscale admin console → Machines → the server → … → Disable key expiry

To check that the server will come back after a reboot, without changing anything:

sudo ./server-preconfig.sh --check

Each line starts with OK, WARN or FAIL. Fix every FAIL before you reboot.

Packages and apt

Symptom	Cause	Fix
Install fails with "dpkg lock held"	a fresh server runs apt updates in the background at boot	server-prep waits up to 10 minutes. If it still fails, wait for the boot updates to end, then re-run the step
CrowdSec 1.4.x installed (an old version)	Ubuntu Pro ESM offers an old crowdsec package	server-prep pins CrowdSec to its official repo at priority 900 and upgrades it. Re-run --only ips

To see what is still running apt:

pgrep -a -f 'apt|dpkg'

No output means apt is free. To see where CrowdSec comes from:

apt-cache policy crowdsec

The installed version should come from packagecloud.io, not from esm.

Firewall and Docker

Symptom	Cause	Fix
Audit FAIL "Docker vs firewall" right after a full run (80/443 reachable even though UFW blocks them)	before v1.1.1, the ufw-docker rules were skipped when Docker was already running	update server-prep, then re-run --only firewall
Firewall step FAILED: "ufw-docker rules not loaded: Docker ports still bypass UFW"	ufw-docker could not be downloaded from raw.githubusercontent.com, its checksum did not match, or its rules did not load	read the log named in the report, fix the cause, then re-run --only firewall
docker ps can't reach ~/.docker/desktop/docker.sock	a Docker Desktop context is active	docker context use default, or remove Docker Desktop

After the firewall step, check that Docker's ports go through UFW:

sudo iptables -S DOCKER-USER | head -3

The output must show -j ufw-user-forward.

Kernel settings and services

Symptom	Cause	Fix
log_martians back to 0 after a reboot	/etc/ufw/sysctl.conf overrides it at boot	the sysctl step rewrites that file: re-run --only sysctl
fs.suid_dumpable=2 after a reboot	apport (Ubuntu's crash reporter) forces it	apport is in DISABLE_SERVICES: re-run --only services
A re-run with --config didn't disable a newly added service	saved configs keep their old DISABLE_SERVICES list	edit DISABLE_SERVICES in /etc/server-prep/server-prep.conf, or disable it by hand: sudo systemctl disable --now <service>

Check the two values after a reboot:

sysctl net.ipv4.conf.all.log_martians fs.suid_dumpable

Expect log_martians = 1 and suid_dumpable = 0.

Passwords and sudo

Symptom	Cause	Fix
passwd: "BAD PASSWORD: ... error loading dictionary"	cracklib-runtime was skipped, so the password dictionary is missing	re-run --only auth. Last resort: sudo apt install --reinstall cracklib-runtime then sudo update-cracklib
No /var/log/sudo.log on Ubuntu 26.04	Ubuntu 26.04 uses sudo-rs, which has no log file option	nothing to fix: it logs to the journal and the audit counts it as PASS

To read sudo-rs logs:

journalctl _COMM=sudo

CrowdSec

Symptom	Cause	Fix
Port 8080 conflict with the panel's Traefik dashboard (Coolify, Dokploy)	CrowdSec's local API uses 8080 by default	keep CROWDSEC_LAPI_PORT=8091
cscli console enroll returns 401 while the connection is fine	a problem on CrowdSec's side	retry later, or enroll from the CrowdSec web console

To check CrowdSec's connection to its central service:

sudo cscli capi status

Panels

Symptom	Cause	Fix
Coolify loses the server right after you set up cloudflared	the server-level Cloudflare Tunnel option moved Coolify's SSH to the tunnel	turn it off, set the server's IP back to its Tailscale IP, and run cloudflared as a Coolify Service for apps

More on this in Panels and docs/cloudflare-tunnel.md.

Monitoring

From docs/monitoring.md:

Symptom	Cause	Fix
Telegraf --test shows "permission denied" on disk mounts	some mounts (Docker, gvfs, netns) can't be read	harmless; you can extend ignore_fs in [[inputs.disk]]
Telegraf logs "Wrote batch" but the bucket stays empty	Cloudflare Access refused the service token and sent Telegraf to its login page, which answers OK	pass the /ping test with the token first (expect 204), then check the bucket in Data Explorer
Telegraf doesn't start at all	[[inputs.docker]] can't reach docker.sock	sudo usermod -aG docker telegraf && sudo systemctl restart telegraf, or delete that block on a server without Docker
Coolify's InfluxDB starts empty with a public Get Started page	its template uses INFLUXDB_INIT_*, which the image ignores	rename them to DOCKER_INFLUXDB_INIT_* (with ORG and BUCKET set) before the first deploy

Laptops and desktops used as servers

Symptom	Cause	Fix
Self-hosted GitHub runners use all the memory and the machine freezes	nothing limits the runners' memory	sudo ./extras/github-runners/limit-runners-memory.sh caps them at 40% of RAM
Audit WARN: many running services	normal on a desktop with GNOME	nothing to fix, or boot without the GUI with HEADLESS=yes

Audit noise

Some WARNs are normal on a prepared server: pending regular (non-security) updates, and "reboot pending" right after an upgrade. The SUID scan skips /var/lib/docker and /var/lib/containerd, so container files don't flood it.

Still stuck

Ask in Discussions with your OS, your server-prep version (first line of ./server-prep.sh --help) and the audit summary (see Providers and hardware for a command that leaves out your host name and IP). Report bugs in Issues. Report security problems privately, as SECURITY.md explains.