Block Access for IP after n 404 errors

[vsatmydynipnet]

I am looking for a way to stop bots the tries 100++ requests creating lot of 404 errors.

Thinking about

• 5 404 in a row
• challenge to their bot

I already search since 2 or 3 days, but found no solution.

Any help would be greatly appreciated.

Thank you

[Xe]

This would be a feature request. I don't think this is currently possible.

[quicoto]

Look into fail2ban, which allows you to setup exactly what you're looking for.

Not a replacement but a side to side protection with Anubis

[vsatmydynipnet]

I am sorry fail2ban is nice for things where you have some time but it does not help against thousands of IPs running100 request to test exploits each. I think real time only anubis would help to stop them after e.g. 5 404.

[quicoto]

I don't get it, fail2ban is exactly to do that. You do not need to specify any IP's. The banning is based on behavior in your filter config (for example: 404 errors), then in your jail config you say how many tries (you say 5), then set the ban time in hours, or forever (-1).

[vsatmydynipnet]

fail2ban reads a log file and in case of the mass requests this bots do to check if they find a exploit, it is to slow in blocking this IP. and most of these IPs only come 1 time, so a "forever" rules mostly does not help. I already collected 150k IP adresses out of the logfiles which I automatically distribute to my pfsense as URL IP list. The list is daily growing.

But I am open for anything new if I see it wrong how fail2ban works.

[vshulcz]

The reason it can't be done as stated is structural: Anubis decides before your backend has answered, so it never learns that a request turned into a 404. And there is no cross-request state to count with. The whole CEL environment is one request:

cel.Variable("remoteAddress", cel.StringType),
cel.Variable("contentLength", cel.IntType),
cel.Variable("host", cel.StringType),
cel.Variable("method", cel.StringType),
cel.Variable("userAgent", cel.StringType),
cel.Variable("path", cel.StringType),
cel.Variable("query", cel.MapType(cel.StringType, cel.StringType)),
cel.Variable("headers", cel.MapType(cel.StringType, cel.StringType)),
cel.Variable("load_1m", cel.DoubleType),
cel.Variable("load_5m", cel.DoubleType),
cel.Variable("load_15m", cel.DoubleType),

plus weight inside thresholds. No status code, no counters, no history.

But for the traffic you're describing, counting is the wrong tool anyway. Those 404s come from paths your app doesn't serve, and the scanner asks for them on request one. You can act on that instead of waiting for the fifth, and it needs no state, which is what makes it work against 150k single-shot IPs:

bots:
  - name: probe-paths
    action: WEIGH
    expression:
      any:
        - path.endsWith(".php")
        - path.contains("/wp-")
        - path.contains("/.env")
        - path.contains("/.git/")
        - path.contains("/vendor/")
        - path.contains("/phpmyadmin")
    weight:
      adjust: 20

  - name: my-real-paths
    action: WEIGH
    expression:
      any:
        - path == "/"
        - path.startsWith("/static/")
    weight:
      adjust: -5

thresholds:
  - name: obvious-probe
    expression: weight >= 25
    action: DENY

DENY is fine in a threshold. The only action thresholds reject is WEIGH:

ErrThresholdCannotHaveWeighAction = errors.New("config.Threshold: a threshold cannot have the WEIGH action")

Two things to watch. Thresholds are skipped for requests that already matched a terminal bot rule with CHALLENGE, per the note in the thresholds docs, so keep the probe rules as WEIGH and let the threshold decide. And tune the path list against your own access log, because a DENY on a path a legitimate client uses is a real outage.

If you want the pressure to depend on how bad the wave is, load_1m is in that variable list too, so a threshold can get stricter only while the box is actually being hammered.

None of this replaces fail2ban at the network layer, it just moves the decision to before the first 404 rather than after the fifth.