Improve SQL character anomaly rules (942420/942430)

[lifeforms]

As part of the CRS3 paranoia project, we will be discussing possible stricter siblings for some rules.

The first two rules, 'SQL Injection Character Anomaly Usage' 942420/942430 (old ids: 981173/981172), are some of the most controversial rules in CRS2 (in my opinion). They cause a lousy user experience due to very high false positives in many web applications. So we should think carefully what to do with them.

981173 : SQL Injection Character Anomaly Usage

981172 : SQL Injection Character Anomaly Usage

My initial impression: These characters in the regexps in themselves can be normal. But if you're a smart attacker and you tamper a lot, you might be able to do an SQL injection with mostly special characters and few detectable keywords, but you might need a lot of these characters. At the same time, in CRS3 we now have @detectSQLi in our toolbox, so we have more diversity in detecting SQLi. So, I'm for keeping these rules in the CRS, but move them a moderately high paranoia level, maybe even PL3.

However since these characters are so normal, and a rule hit can happen for benign reasons, I would prefer to keep those rules at 'anomaly' level, because they hit FP so often that they made CRS2 almost unusuable from time to time.

If a higher sibling hits, the lower sibling has also hit by definition (due to our earlier decision to allow multiple sibling rules firing). So that means that if you hit the base rule plus a higher sibling, you get an incoming score of 4 (for 942430) + 4 (for 942431) = 8 which is critical. This means that people running at high PL might actually be troubled more by this rule than a CRS2 user is right now. I think that people would probably not want that, unless the character limit on the sibling is very high. So I think we don't absolutely need stricter siblings for these ones, but if we'd add them, then maybe I’d put the siblings at a very high paranoia level like PL4-5, and give them a pretty high character limit.

As for the character limits, it would be helpful if we would have some test traffic of attacks carried out with only these kinds of characters. I remember reading some examples, so I'm pretty sure it can be done, but I’m not 100% sure if a useful injection can be done within the CRS2 character limits, so there might be a specific reason for them, and using stricter limits MIGHT not help much. It’s a tough call without that information.

Christian said: I tried to reduce the FPs for 942420 and 942430 in pl2. But to no avail. The legitimate traffic simply has a ton of those and raising the limits a bit does not change much (+2 -~> 10% reduction).

[dune73]

These rules are very hard to deal with. We have talked about them a great deal and still we
do not know what to do. I spent a few hours gathering stats about how they behave in practice.
Based on that I am proposing a set of limits and PL for the rules and their possible siblings.

942420: This rule targets only cookies and cookie names and as is, there is a limit of 8 special
characters from this set: ~!@#$%^&*()-+={}[]|:;"'\´\’\‘`<>

I have tried to run a body of about legitimate 8K requests against this rule and change the
limit. In the following graph you see the number of fps on the y axis and x represents the
limit imposed:

                  False positives (y) depending on the limit (x) of the rule 942420
                               Based on a body of ~8K legitimate reqs

      +-----------+--------------+--------------+-------------+--------------+--------------+--+
18000 ++          +              +              +             +              +   942420 ******++
      |  *                                                                                     |
16000 ++  *                                                                                   ++
      |   *                                                                                    |
      |    *                                                                                   |
14000 ++   *                                                                                  ++
      |     *                                                                                  |
12000 ++     *                                                                                ++
      |      *                                                                                 |
      |       *                                                                                |
10000 ++      *                                                                               ++
      |        *                                                                               |
 8000 ++       ******                                                                         ++
      |              ***                                                                       |
      |                 ***                                                                    |
 6000 ++                   *                                                                  ++
      |                     *                                                                  |
 4000 ++                     *                                                                ++
      |                       *****************************                                    |
      |                                                    *************************************
 2000 ++                                                                                      ++
      |           +              +              +             +              +              +  |
    0 ++----------+--------------+--------------+-------------+--------------+--------------+-++
                  5              10             15            20             25             30

As you can see, there is a large group of FPs which does not disappear even with a limit of
30 characters. The problem is obvious: With cookies, you get a high number of hits in the
legitimate traffic. Once an offending cookie is being set, the FPs skyrocket.

Here are the numbers in detail. Percents calculated again a number of 8262 reqs (as the rule
can trigger more than once per request, the percent numbers can be really high).

  1  17126 207.29%
  2  12973 157.02%
  3  8256 99.93%
  4  8126 98.35%
  5  7425 89.87%
  6  6346 76.81%
  7  6056 73.3%
  8  3257 39.42%
  9  3246 39.29%
 10  3157 38.21%
 11  3096 37.47%
 12  3074 37.21%
 13  3058 37.01%
 14  2931 35.48%
 15  2905 35.16%
 16  2895 35.04%
 17  2881 34.87%
 18  2787 33.73%
 19  2787 33.73%
 20  2787 33.73%
 21  2787 33.73%
 22  2787 33.73%
 23  2607 31.55%
 24  2607 31.55%
 25  2607 31.55%
 26  2607 31.55%
 27  2535 30.68%
 28  2535 30.68%
 29  2535 30.68%
 30  2535 30.68%

So no matter how high the limit, there are FPs, that won't go away. I have then checked out the
"bad" cookies and identidied 16 different cookies that make up the 2535 alerts with limit 30.
So the problem is abundant.

I think this warrants Walter's call to push these rules to higher paranoia levels, I suggest to

• move 942420 to PL3 with a limit of 8 (as is!)
• add a stricter sibling 942421 in PL4 with a limit of 3.

These limits are based on the numbers above. I doubt my traffic is representative, but it's the
best quantitative data I have around. The limits are thus on perceived gaps in the data.
There is a step between 2 and 3. And also between 7 and 8.

I have also looked into attack traffic based on a body of nikto/sqlmap requests. But these two
tools seem to leave the cookies alone. No hits whatsoever.

Let's turn to 942430. This rule works on the the arguments:
ARGS_NAMES|ARGS|XML:/*
And it has a default limit of 5 special characters from the same list:
~!@#$%^&*()-+={}[]|:;"'\´\’\‘`<>

Here we see a substantial reduction of FPs when we raise the limit:

                   False positives (y) depending on the limit (x) of the rule 942430
                                Based on a body of ~8K legitimate reqs

     +-----------+--------------+--------------+--------------+--------------+--------------+--+
     |           +              +              +              +              +   942430 ****** |
4000 ++ *                                                                                     ++
     |  *                                                                                      |
3500 ++  *                                                                                    ++
     |   *                                                                                     |
     |   *                                                                                     |
3000 ++  *                                                                                    ++
     |    *                                                                                    |
     |    *                                                                                    |
2500 ++   *                                                                                   ++
     |    *                                                                                    |
2000 ++    *                                                                                  ++
     |     *                                                                                   |
     |     *                                                                                   |
1500 ++     *                                                                                 ++
     |       *                                                                                 |
     |        ***                                                                              |
1000 ++          ***                                                                          ++
     |              ***                                                                        |
 500 ++                ************                                                           ++
     |                             ***                                                         |
     |           +              +     ***************************************************   +  |
   0 ++----------+--------------+--------------+--------------+--------------+-----------*******
                 5              10             15             20             25             30


   1  3925 47.51%
   2  1693 20.49%
   3  1143 13.83%
   4  961 11.63%
   5  686 8.3%
   6  606 7.33%
   7  544 6.58%
   8  499 6.04%
   9  483 5.85%
  10  304 3.68%
  11  248 3.0%
  12  223 2.7%
  13  220 2.66%
  14  210 2.54%
  15  185 2.24%
  16  134 1.62%
  17  126 1.53%
  18  125 1.51%
  19  125 1.51%
  20  121 1.46%
  21  119 1.44%
  22  116 1.4%
  23  105 1.27%
  24  102 1.23%
  25  102 1.23%
  26  102 1.23%
  27  95 1.15%
  28  91 1.1%
  29  90 1.09%
  30  90 1.09%

So we have fewer FPs and they get fewer with a growing limit. When looking at the attack traffic,
the following picture can be derived (based on a body of 26409 attack requests):

                  Valid positives (y) depending on the limit (x) of the rule 942430
                                 Based on a body of ~24K attack reqs

      +-----------+--------------+--------------+-------------+--------------+--------------+--+
10000 ++          +              +              +             +              +   942430 ******++
      |  *                                                                                     |
      |   *                                                                                    |
      |   *                                                                                    |
      |    *                                                                                   |
 8000 ++   *                                                                                  ++
      |     ***                                                                                |
      |        ***                                                                             |
      |           ***                                                                          |
 6000 ++             ***                                                                      ++
      |                 ***                                                                    |
      |                                                                                        |
      |                    ***                                                                 |
 4000 ++                      ***                                                             ++
      |                          ******                                                        |
      |                                ***                                                     |
      |                                   ******                                               |
      |                                         ***                                            |
 2000 ++                                           ********                                   ++
      |                                                    ***************                     |
      |                                                                   **********************
      |           +              +              +             +              +              +  |
    0 ++----------+--------------+--------------+-------------+--------------+--------------+-++
                  5              10             15            20             25             30


  1  9648 36.53%
  2  7343 27.8%
  3  7095 26.87%
  4  6628 25.1%
  5  6025 22.81%
  6  5386 20.39%
  7  4753 18.0%
  8  4238 16.05%
  9  3772 14.28%
 10  3471 13.14%
 11  3153 11.94%
 12  2873 10.88%
 13  2552 9.66%
 14  2272 8.6%
 15  2018 7.64%
 16  1821 6.9%
 17  1667 6.31%
 18  1560 5.91%
 19  1457 5.52%
 20  1376 5.21%
 21  1282 4.85%
 22  1185 4.49%
 23  1087 4.12%
 24  1033 3.91%
 25  963 3.65%
 26  913 3.46%
 27  854 3.23%
 28  824 3.12%
 29  768 2.91%
 30  745 2.82%

I think this rule clearly had its merits. It catches a substantial amount of attacks. The
FP rate is high. But it can be reduced by setting the limits accordingly.

So I suggest the following:

• Leave 942430 at PL2, but set the limit to 12
• Add a stricter sibling at PL3 at a limit of 6
• Add a stricter sibling at PL4 at a limit of 2

With these limits, we catch 10%, 20% and 28% of the attacks in my test set.

I am open to a debate about the individual limits. The ones I propose are reasonable, I think.
Maybe we can adjust some day in the future, when we have better stats.

Leaving 942420 out of PL2 and having 942430 present in PL2 also seems the right move right now.

[dune73]

I forgot to add that setting the anomaly score to "anomaly" / 4 for these rules and siblings seems a good idea to me.

[lifeforms]

Terrific stats! This is exactly what we need. The graphs show clearly that we can make these rules more palatable with small changes. I think you're right on the mark with the basic character limits and paranoia levels.

I'm not yet fully convinced on the stricter sibling paranoia levels. This is mainly because of the 'stacking effect' of the multiple subling rules firing. which causes the total score to go beyond anomaly to critical severity. So if a person is running at PL3, hitting the regexp of 942430 with just 6 characters will cause 2 times the anomaly score, so a critical score. This is a much more strong effect than CRS2's anomaly score, which I already found hard to deal with in production. It's a hard call, as I also agree that increasing paranoia can definitely have such effects, but I think with such FP prone rules, I have a slight preference for reserving them for PL4 and PL5 respectively. So I would prefer:

• Leave 942430 at PL2, but set the limit to 12
• Add a stricter sibling at PL4 at a limit of 6
• Add a stricter sibling at PL5 at a limit of 2

[dune73]

Thanks for the feedback Walter.

We have defined PL 0-4. Of course, this is completely arbitrary and there might be need for PL5, but maybe not just yet because of this single rule.

These rules are a nuisance, but people using higher PLs should be willing to carry that burden. We could lighten it by adding a proposal for a tuning rule / whitelisting rule / ignore rule in the documentation / comment section above the rule. This could be as simple as:

# Expect a lot of false positives with this rule. 
# This will make it necessary to disable the rule for certain parameters.
# The following directive will switch off the rule globally for the
# request parameter foo (GET and POST). Place this instruction
# in your configuration after the include directive for the Core Rules Set
#
# SecRuleUpdateTargetById 942430 "!ARGS:foo"
#

I'd do the same with 942420 as well.

[lifeforms]

Ah, you're right, PL4 is the max currently! I was confused by PL1 being the default. So, PL3 and PL4 are the strongest levels. Then a user probably won't get too confused by false positives. So this choice of levels is acceptable to me!

[dune73]

Thank you.

[dune73]

Implemented in pull request 334.

[dune73]

Pull has been merged. Issue solved.