Troubleshooting Cloudflare action

[neil-verus]

Hi !

I'm setting up a filter and jail to block a specific malicious request we're seeing lots of lately. I'm using v0.11.2.

For testing purposes, I'm using cheeseburger as the failregex. Here's my filter:

[Definition]
failregex = ^<HOST>.*cheeseburger.*
ignoreregex = 

Here's my jail as seen in jail.local:

[carding]
ignoreip = 127.0.0.1/8 ::1 
ignorecommand =
bantime = 86400
findtime = 600
maxretry = 3
backend = auto
usedns = warn
logencoding = auto
enabled = true
destemail = **_REDACTED_**
filter = carding
action = cloudflare[cftoken="**_REDACTED_**", cfuser="**_REDACTED_**"]
logpath = /var/www/vhosts/**_REDACTED_**.com/logs/access_ssl_log

I've also tried setting the action simply to cloudflare without the parameters (action = cloudflare)

I'm using the latest version of cloudflare.conf (from https://github.com/fail2ban/fail2ban/blob/master/config/action.d/cloudflare.conf) with my cftoken and cfuser filled in near the end of the file within the [Init] function.

The jail and filter successfully detect attempts to hit the /cheeseburger/ URL and I see the IPs in my ban list, but the bans apparently aren't being passed to Cloudflare. I confirmed that the API key I'm using is valid and active:

$ curl -X GET "https://api.cloudflare.com/client/v4/user/tokens/verify"      -H "Authorization: Bearer REDACTED"      -H "Content-Type:application/json"
{"result":{"id":"8cd1b42b9bec53f34c9a9133ccb95929","status":"active","not_before":"2021-11-15T00:00:00Z","expires_on":"2031-11-15T23:59:59Z"},"success":true,"errors":[],"messages":[{"code":10000,"message":"This API Token is valid and active","type":null}]}

I noted the error codes references in the comments for cloudflare.conf, but I'm not seeing any of these error codes come up in /var/log/fail2ban.log. Does anyone have any troubleshooting tips that might help me see the Cloudflare API error codes and get these bans successfully passed to Cloudflare ?

Thanks for your help and have a great afternoon !
Neil

[sebres]

You did not describe what is the issue exactly, for instance which errors do you see in fail2ban.log if the ban fails.

Due to #2318:

• CloudFlare API rate limit is 1200 calls every 5 minutes (4 calls per second).
• CloudFlare Firewall IP rules limit is 500 for free account

so both can be the reason why the ban fails.

[neil-verus]

Hi Sebres !

That's the thing; I don't get any relevant errors in the fail2ban or any indication that the ban has failed. When I visit a URL that matches the pattern (literally mysite.com/cheeseburger at the moment while I'm testing) more than 3 times the ban is triggered and I see my test machine's IP in the banned IP list. However, the test machine isn't blocked and I don't see the banned IPs at Cloudflare in the firewall as I should if it was working.

Here are log lines showing a ban (i've replaced the test machine IP with XXX.XXX.XXX.XXX here):

2021-11-16 11:40:58,937 fail2ban.filter         [28052]: INFO    [carding] Found XXX.XXX.XXX.XXX - 2021-11-16 11:40:58
2021-11-16 11:40:59,947 fail2ban.filter         [28052]: INFO    [carding] Found XXX.XXX.XXX.XXX - 2021-11-16 11:40:59
2021-11-16 11:41:00,190 fail2ban.filter         [28052]: INFO    [carding] Found XXX.XXX.XXX.XXX - 2021-11-16 11:41:00
2021-11-16 11:41:00,421 fail2ban.actions        [28052]: NOTICE  [carding] Ban XXX.XXX.XXX.XXX

I reviewed fail2ban.log just now and confirmed that the only errors I found were a single instance of a timezone hiccup. I obfuscated the IP and URL shown:

2021-11-16 15:13:00,317 fail2ban.filter         [15098]: WARNING [carding] Simulate NOW in operation since found time has too large deviation 1637104318 ~ 1637104380.32 +/- 60
2021-11-16 15:13:00,317 fail2ban.filter         [15098]: WARNING [carding] Please check jail has possibly a timezone issue. Line with odd timestamp: xxxx:xxx:xxx:xxx:xxxx:xxxx:xxxx:xxxx- - [16/Nov/2021:15:11:58 -0800] "POST /rest/default/V1/guest-carts/YKp3vCrz2bH7Eb4D0Pk429bQ8AgITcsb/totals-information HTTP/1.0" 200 2195 "https://www.mysite.com/checkout/cart/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/95.0.4638.54 Safari/537.36"

My test pattern is, by design, one that only I am visiting, so rate limiting doesn't seem likely. Looking at fail2ban.log I don't see any activity that would exceed the thresholds mentioned, and the site in question is on Cloudflare's pro plan.

Any idea why no error messages are being logged in relationship to the cloudflare API ? Do my jail's [filter] and [action] look right ?

Thanks again for your help,
Neil

[sebres]

I don't see the banned IPs at Cloudflare in the firewall as I should if it was working.

Can you run the commands of action in the shell? For some real IP or test IP like 192.0.2.1. Do you see the result then?

that the only errors I found were a single instance of a timezone hiccup

This is not an errors but warnings... and they have nothing to do with your issue...

Any idea why no error messages are being logged in relationship to the cloudflare API ?

I have no idea why the ban doesn't take place on cloudflare side (I don't use it)... But if the action works (the command doesn't fail) there must be nothing in log after Ban <ip>, otherwise you'd see an error with some additional info.

Did you restart fail2ban or jail as you configured the jail with cloudflare action? Just to ensure the action is really active...

If nothing helps, you can switch to DEBUG or HEAVYDEBUG logging to see what happens if fail2ban banning IP with the action.
You can try to do it temporary during the run-time (without to restart fail2ban) - with:

## show current log level:
?sudo? fail2ban-client get loglevel
## switch to DEBUG:
?sudo? fail2ban-client set loglevel 9; # 1 level below DEBUG
#?sudo? fail2ban-client set loglevel 5; # HEAVYDEBUG
## ... do some tests or ban some IPs ...
?sudo? fail2ban-client set carding banip 192.0.2.1
## switch back to default level:
?sudo? fail2ban-client set loglevel INFO

[neil-verus]

Hi Sebres,

Thanks!

I ran the action from the shell(X-Auth-Key and email obfuscated below, but I used the actual email and API token there when testing):

curl -s -o /dev/null -X POST -H 'X-Auth-Email: me@mysite.com' -H 'X-Auth-Key: XXXXXXXXXXXXXXXXXXXXXXXXX' -H 'Content-Type: application/json' -d '{"mode":"block","configuration":{"target":"192.0.2.1","value":"192.0.2.1"},"notes":"Fail2Ban carding"}' https://api.cloudflare.com/client/v4/user/firewall/access_rules/rules
I received no output and the test IPs were not banned within the target account's firewall rule. I've opened a ticket with Cloudflare as the syntax needed there may have changed and I'll get back to you and let you know what I find out.

I did restart fail2ban entirely after setting the jail up and after making changes to it or the filter.

When I enabled HEAVYDEBUG and tested the ban I didn't get any smoking guns back and the ban again appeared to work, but again the banned IPs weren't present in Cloudflare at firewall > tools:

2021-11-17 12:16:02,320 fail2ban.transmitter    [15098]: HEAVY   Command: ['set', 'carding', 'banip', '192.0.2.2']
2021-11-17 12:16:02,320 fail2ban.actions        [15098]: NOTICE  [carding] Ban 192.0.2.2
2021-11-17 12:16:02,320 fail2ban.action         [15098]: #09-Lev. curl -s -o /dev/null -X POST -H 'X-Auth-Email: redacted@site.com' -H 'X-Auth-Key: REDACTED' -H 'Content-Type: application/json' \
-d '{"mode":"block","configuration":{"target":"ip","value":"192.0.2.2"},"notes":"Fail2Ban carding"}' \
https://api.cloudflare.com/client/v4/user/firewall/access_rules/rules
2021-11-17 12:16:02,334 fail2ban.observer       [15098]: DEBUG   [carding] Observer: ban found 192.0.2.2, 86400
2021-11-17 12:16:02,335 fail2ban.filterpyinotify[15098]: DEBUG   Event queue size: 16
2021-11-17 12:16:02,335 fail2ban.filterpyinotify[15098]: DEBUG   <_RawEvent cookie=0 mask=0x2 name='' wd=3 >

I'm guessing maybe the syntax being sent to cloudflare needs to be changed and I'll let you know what I find out once I hear back from Cloudflare.

Have a great evening,
Neil

[sebres]

I ran the action from the shell...

You can try it without -s -o /dev/null then you could probably see some answer of cloudflare.
With that curl working completely silently, so must basically set the error code in shell only.

[sebres]

And adding -v would make the output additionally more verbose, and -vv or -vvv increase the verbosity level still more, so curl becomes more talkative, so it'd allow to investigate deeper if some protocol, connectivity or encryption issues present.

[neil-verus]

Hi Sebres, sorry for the delay.

I eventually ended up using the Firewall Access Rules API for a zone: https://api.cloudflare.com/#firewall-access-rule-for-a-zone-create-access-rule

I created a token scoped to the zone I was working with in Cloudflare and with zone:firewall services:edit privileges, then used the API token generated in actionban like this:

actionban = curl -X POST "https://api.cloudflare.com/client/v4/zones/<ZONE_ID>/firewall/access_rules/rules" -H "Authorization: Bearer <TOKEN>" -H "Content-Type: application/json" --data '{"mode":"block","configuration":{"target":"ip","value":"<IP>"},"notes":"Fail2Ban carding"}'

The action is working great now.

Thanks for your help with this and have a Merry Christmas !
Neil