AppSec listeners can die permanently after SIGHUP reload while LAPI keeps running

[sueess86]

What happened?

What happened

After sending SIGHUP to reload the configuration (the documented way to apply
appsec-rules changes, per Rules Deployment),
all AppSec runners shut down and did not come back:

level=info msg="Shutting down Appsec server"
level=error msg="Appsec Runner is dying"

Afterwards:

• Every AppSec port answers connection refused (verified both from a peer
  container and via wget http://127.0.0.1:7422 inside the CrowdSec container).
• The CrowdSec process itself keeps running and looks healthy: LAPI responds,
  CPU/memory normal, no crash, no further error logs.
• The listeners never recover on their own. Only a full container restart
  (docker restart) brings AppSec back.

The failure is racy: after a single SIGHUP the listeners usually recover within
~5–9 seconds. With overlapping/multiple reloads in a short window they die
permanently. For a fail-closed WAF deployment this silently turns into a full
outage (every request rejected) while the process still reports healthy.

What did you expect to happen?

Expected behavior

Either the AppSec listeners reliably come back after SIGHUP, or a failed re-bind
is retried / escalated (crash or health signal) instead of leaving the process
running in a half-dead state that no health check on the LAPI can detect.

How can we reproduce it (as minimally and precisely as possible)?

Analysis

This looks like a follow-up to #3881 ("CrowdSec crashes if it cannot bind AppSec
Port") and its fix #3882 ("waf: return if we cannot create the TCP listener",
merged for v1.7.3):

• Before waf: return if we cannot create the TCP listener #3882, a bind failure caused a panic (visible crash → supervisor restart).
• Since waf: return if we cannot create the TCP listener #3882, listenAndServe in pkg/acquisition/modules/appsec/run.go sends
  the bind error to serverError and returns — there is no retry / re-bind
  loop, so the runner goroutine dies silently while the rest of the process
  keeps running.
• On SIGHUP, cmd/crowdsec/serve.go kills and restarts the acquisition tombs
  with short timeouts. If the old HTTP server's Shutdown() is slow to release
  the port (e.g. open keep-alive connections from the proxy), the new listener's
  bind races against the old one, fails once, and — because of the point above —
  never tries again. With many AppSec listeners the chance that at least one
  (or all, on repeated reloads) ends up dead grows quickly.

Anything else we need to know?

Suggested fix

Add a bounded retry with backoff around the TCP bind in listenAndServe
(and the UDS variant), or wait for the old listener to be fully closed before
starting the new one during the SIGHUP acquisition restart. Alternatively,
treat a permanently failed AppSec bind as fatal again (as pre-#3882) so process
supervisors can restart cleanly — silent partial death is the worst variant for
fail-closed deployments.

Workaround

We stopped using SIGHUP for appsec-rules changes entirely and do a full,
orchestrated restart instead (with the proxy told about the maintenance window).
That works, but contradicts the documented reload flow and costs ~20–30 s of
inspection downtime per rules change.

Crowdsec version

Details

$ cscli version
 v1.7.8 (Mai 2026)

OS version

Details

# On Linux:
$ cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 13 (trixie)"
NAME="Debian GNU/Linux"
VERSION_ID="13"
VERSION="13 (trixie)"
VERSION_CODENAME=trixie
DEBIAN_VERSION_FULL=13.5
ID=debian
</details>


### Enabled collections and parsers

<details>

```console
$ cscli hub list -o raw
# paste output here

Acquisition config

Details

# On Linux:
$ cat /etc/crowdsec/acquis.yaml /etc/crowdsec/acquis.d/*
# paste output here

# On Windows:
C:\> Get-Content C:\ProgramData\CrowdSec\config\acquis.yaml
# paste output here

Config show

Details

$ cscli config show
# paste output here

Prometheus metrics

Details

$ cscli metrics
# paste output here

Related custom configs versions (if applicable) : notification plugins, custom scenarios, parsers etc.

Details

[github-actions]

@sueess86: Thanks for opening an issue, it is currently awaiting triage.

In the meantime, you can:

1. Check Crowdsec Documentation to see if your issue can be self resolved.
2. You can also join our Discord.
3. Check Releases to make sure your agent is on the latest version.

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[blotus]

Hello,

Thanks for the issue.

Do you have any error logs when this occur ? (i'd expect at least some errors mentioning a failure to bind the socket)

There's no retry by design for this (if you have another process listening on the same port, it's something unlikely to resolve itself quickly), but this relies on the assumption that the previous server has already been properly shutdown.