Rule set to block requests to overloaded proxied hosts

[theseion]

I'd like to contribute this set of rules. Feel free to use them as you wish (e.g. in the CRS). While writing the rules I ran into a couple of issues that I'd like to highlight.

We use NGiNX as reverse proxy for multiple hosts.
I've written a rule set that does the following:

• if a host answers with 503 or times out (NGiNX will set status code 504 which I replace with 503 using proxy_intercept_errors on; and error_page 504 =503 /503.html;) I set a resource flag
• if the resource flag is set in phase 1, block all requests (with 503) for 10 seconds to allow the affected host to recover
• unset the resource flag after 10 seconds

This is the set of rules that make it work:

# -=[ Recovery blockade for overloaded hosts ]=-
#
# These rule use a RESOURCE collection per host name to track proxy errors.
# When a proxy error occurs, all requests to the host are blocked until
# the recovery timeout has expired.

# -- phase 1 --
# Initialise the resource collection for the host.
SecAction \
 "id:2000,\
  phase:1,\
  pass,\
  t:none,\
  nolog,\
  tag:'paranoia-level/1',\
  setrsc:'%{SERVER_NAME}'"

# Check whether the recovery blockade is active
SecRule RESOURCE:timeout_flag "@eq 1" \
 "id:2001,\
  phase:1,\
  pass,\
  t:none,\
  tag:'paranoia-level/1',\
  chain"
    # AND check whether the recovery blockade has expired.
    # If both conditions match, disable the blockade. 
    SecRule RESOURCE:expire_timestamp "@lt %{TIME_EPOCH}" \
    "msg:'Lifting blockade for %{SERVER_NAME} after recovery timeout expired.',\
      setvar:'RESOURCE.timeout_flag=0'"

# Check whether the recovery blockade is still active after checking the timeout.
# If so, deny access immediately.
SecRule RESOURCE:timeout_flag "@eq 1" \
 "id:2002,\
  phase:1,\
  deny,status:503,\
  t:none,\
  tag:'paranoia-level/1'"

# -- phase 2 --
# Set flag initially.
# For some reason this doesn't work in phase 1. This also can't seem to be done
# conditionally, so the flag will be reset on every unblocked request, but that's fine.
SecAction \
 "id:2003,\
  phase:2,\
  pass,\
  t:none,\
  nolog,\
  setvar:'RESOURCE.timeout_flag=0'"

# -- phase 3 --
# Check whether the response from the proxy did respond (NGiNX will set status 504 for proxy timeout).
# If so, enable recovery blockade and set the recovery timeout.
# Note: We use TIME_EPOCH because currently (2018-12-18) `expirevar` doesn't work for this case.
SecRule RESOURCE:timeout_flag "@eq 0" \
 "msg:'Proxy error from %{SERVER_NAME}. Blocking all requests to %{SERVER_NAME} for 10 seconds to allow host to recover.',\
  severity:'WARNING',\
  id:2004,\
  phase:3,\
  pass,\
  t:none,\
  tag:'paranoia-level/1',\
  chain"

    SecRule RESPONSE_STATUS "@eq 503" \
     "setvar:'RESOURCE.timeout_flag=1',\
      setvar:'RESOURCE.expire_timestamp=%{TIME_EPOCH}',\
      setvar:'RESOURCE.expire_timestamp=+10'"

I ran into three issues:

1. I can't set RESOURCE.timeout_flag=0 in phase 1 unconditionally, as that would override any previously set RESOURCE.timeout_flag=1. So I tried testing whether the variable is set (SecRule RESOURCE:timeout_flag "@rx ^$", SecRule RESOURCE:timeout_flag "!@rx ^.*$", SecRule RESOURCE:timeout_flag "@eq 0" and other variants) but I couldn't get it to work. More frustratingly, I then tried to test whether I could set the variable at all by using a SecAction in phase 1 with setvar:'RESOURCE.timeout_flag=0' and even that didn't work. Rules 2001, 2002 and 2004 would always return a 0 match (gleaned from the debug log). I then accidentally stumbled over a solution that worked: unconditionally setting the flag in phase 2. It's a lucky coincidence that phase 2 is available for the particular problem.
2. expirevar doesn't work (I've seen a couple of issues here about that, so I assume that this will be fixed sooner or later). I use an additional variable to manually expire the flag variable.
3. I struggled for a couple of hours to understand how I was supposed to use the RESOURCE collection:

• the (old) documentation doesn't clearly state that initcol is no longer supported for RESOURCE and that one should use setrsc
• the (old) documentation isn't very clear about the fact that using setrsc (or setsid or initcol or any of the other collection initialisers for that matter) will make RESOURCE (or the other collection names) the alias for that collection. This behaviour didn't seem logical to me as it implies (if I'm not mistaken) that I can only use a single resource per transaction. E.g., what if I now wanted to write a rule that targets access to a particular URI? I think that would currently throw an error around the lines of "this collection has already been initialised".

I'd love some feedback on these points if you have the time. I'd like to improve the rules if possible. Otherwise I just wanted to say thank you for the great work that you are doing with ModSecurity!

[theseion]

I think I figured out how to check whether a variable has been initialised: SecRule &RESOURCE.timeout_flag "@eq 0".
The trick is to use the counter mode (&). I stumbled across this by accident in the CRS. PLEASE document this "trick".