Operations and Troubleshooting

Operations and Troubleshooting

Routine health checks

Run from the controller:

ssh -i ~/.ssh/id_ed25519 -p 22571 bootstraper@203.0.113.10 \
  'sudo systemctl is-active ssh.socket crowdsec crowdsec-firewall-bouncer'

For a Traefik host:

ssh -i ~/.ssh/id_ed25519 -p 22571 bootstraper@203.0.113.10 \
  'sudo systemctl is-active docker && sudo docker ps'

Check persistent firewall syntax:

ssh -i ~/.ssh/id_ed25519 -p 22571 bootstraper@203.0.113.10 \
  'sudo iptables-restore --test /etc/iptables/rules.v4'

Safe rerun policy

The setup and application playbooks are intended to be idempotent. Rerun the same playbook after correcting configuration, rather than making manual changes on the server.

uv run ansible-playbook --ask-vault-pass playbooks/setup.yaml
uv run ansible-playbook --ask-vault-pass playbooks/apps.yaml

Bootstrap has one intentional non-idempotent behavior: it initializes each managed user's password only once.

SSH fails after a port change

1. Use provider console or serial access. Do not keep guessing against the public endpoint.
2. Check upstream security-group/NAT rules for the new port.
3. On the server, inspect the socket and local rules:

sudo systemctl status ssh.socket ssh.service
sudo ss -lntp | grep ssh
sudo iptables -S TCP_IN

4. Confirm the config validates:

sudo sshd -t

5. Fix inventory or variables, then rerun setup from the currently reachable port.

Host key verification fails

First verify whether the host was rebuilt or its IP was reassigned. If the identity is expected to have changed, remove only that host entry:

ssh-keygen -R '[203.0.113.10]:22571'
ssh-keyscan -p 22571 203.0.113.10 >> ~/.ssh/known_hosts

Do not blindly remove host keys after an unexpected warning.

Traefik certificate does not issue

Check these in order:

1. The DNS A record points to the server's public IPv4 address.
2. TCP 80 reaches Traefik from the public Internet.
3. The host belongs to traefik_hosts so local firewall rules allow 80/443.
4. traefik_acme_email is nonempty.
5. Read logs:

sudo docker logs traefik --tail 200

CrowdSec sees no events

sudo cscli metrics
sudo cscli collections list
sudo ls -la /etc/crowdsec/acquis.d
sudo tail -n 50 /var/log/auth.log
sudo tail -n 50 /opt/traefik/logs/access.log

Baseline hosts need sshd.yaml; Traefik hosts also need traefik.yaml.

Application is unavailable

For every application, check its compose project, labels, and Traefik logs:

sudo docker compose -f /opt/<application>/compose.yaml ps
sudo docker logs <container-name> --tail 100
sudo docker logs traefik --tail 100

Confirm the hostname resolves to the proxy and the host is in both its app group and traefik_hosts.

What not to do

• Do not manually edit managed files under /etc/ssh, /etc/iptables, /etc/docker, /etc/crowdsec, or /opt/<application> and expect the change to survive setup.
• Do not expose an application container with ports: just to make it work.
• Do not add secrets to tracked inventory or group-variable files.
• Do not disable firewall rules to debug. Use console access and inspect the active rules instead.