Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP

Loading…

IP persistence storage seems to not clean up #473

Closed
rcbarnett opened this Issue · 3 comments

2 participants

@rcbarnett
Owner

MODSEC-325: It seem that mod_security is not cleaning up the persistence storage for the IP-storage I am using.

Here is a simple test ruleset I am using (nothing else):

SecRule IP:dos_block "@eq 1" \
"phase:2,block,msg:'IP address blocked due to high connection rate.', \
id:2003,status:403,severity:'CRITICAL',tag:'DoS'"
SecRule SCRIPT_FILENAME "@rx /www/content/prod/dl/dl.php" \
"phase:5,chain,t:none,nolog,pass,id:2001,severity:'INFO',tag:'DoS',setvar:IP.dos_counter=+1,expirevar:IP.dos_counter=60"
SecRule IP:dos_counter "@gt 100" \
"t:none,setvar:IP.dos_block,setvar:!IP.dos_counter,expirevar:IP.dos_block=60"

The ruleset is working fine. But what I am noticing is, that the /var/tmp/ip.pag file is constantly growing. It is currently at >1GiB
size. When I do a "strings" on that file, I see entries like this:

__expire_KEY
1344192686
794.224.68.113_64d9d88c927c58359a3649f30a00e95070bbb8c0
TIMEOUT
3600
__key
794.224.68.113_64d9d88c927c58359a3649f30a00e95070bbb8c0
__name
CREATE_TIME
1344189037
UPDATE_COUNTER
dos_counter
__expire_dos_counter
1344189097
LAST_UPDATE_TIME
1344189086
94.224.68.113_64d9d88c927c58359a3649f30a00e95070bbb8c0
__expire_KEY
1343169406
868.167.225.250_77150c41806fda817314ba9c40e040c598830d5d
TIMEOUT
3600
__key
868.167.225.250_77150c41806fda817314ba9c40e040c598830d5d
__name
CREATE_TIME
1343165769
UPDATE_COUNTER
dos_counter
__expire_dos_counter
1343165829
LAST_UPDATE_TIME
1343165806
68.167.225.250_77150c41806fda817314ba9c40e040c598830dIR
__expire_KEY
1344005533
786.132.126.93_aa739e3aaaa1fbfc8667bb26120e9930df881a82
TIMEOUT
3600
__key
786.132.126.93_aa739e3aaaa1fbfc8667bb26120e9930df881a82
__name
[...]

Now when I convert the UNIX time strings to UTC, it shows me dates, which are already
expired since weeks. I would asume, that if an entry is expired, that the garbage collection
of mod_sec would remove the entry from the ip.pag file, to keep it small. Or am I wrong?

Any assistance would be highly appreciated.

@zimmerle zimmerle was assigned
@rcbarnett
Owner

Original reporter: wneessen

@rcbarnett
Owner

bpinto: Hello,

The persistent storage files are sparse data files. Run a "du -b" vs "ls -l" and see what the difference is.

The engine will overwrite the expired ones when necessary.

Also i suggest you use SecCollectionTimeout directive. Take a look in Reference Manual.

Thanks

Breno

@rcbarnett
Owner

wneessen: Hi Breno,

I am sorry that I have to re-open this old ticket. After reading your response I didn't further investigate this issue and simply disabled mod_security on the specific host, which resolved our problems so far.
After almost a year now, I had to re-enable mod_security with the same ruleset again (hoping that the problem was gone given that there were couple of version upgrades... unfortunately it didn't.

Here is a short timeline of what happend and what I was able to capture:

Re-Enabled mod_security with the same ruleset on Tuesday. The rules worked as expected.
On wednesday I already noticed an increase in the /var/tmp/ip.pag file.
Thurday night, the apache got unresponsive for a period of at least 10 minutes (we switched to the backup server
to not lose traffic). The server-status page (which happend to load (very slowly) after a couple of minutes), showed
tons of "L" processes". While the mod-sec debug log was filled with lots of these:

[05/Sep/2013:21:47:09 +0000] [xxxxxxx.cleverbridge.com/sid#80171c530][rid#802dbe0a0][/XXXXXXXXX/xxxxxxx-xxxxxxx-xxxxxxx-xxxxxxx/xxxxxxx.pdf][1] collections_remove_stale: Failed deleting collection (name "ip", key "80.XXX.XXX.XXX_6e1c38356769095ad76267f9edb43ddae2af1491"): Internal error
[05/Sep/2013:21:47:09 +0000] [xxxxxxx.cleverbridge.com/sid#80171c530][rid#802dac0a0][/XXXXXXXXX/xxxxxxx-xxxxxxx-xxxxxxx-xxxxxxx/xxxxxxx.msi][1] collections_remove_stale: Failed deleting collection (name "ip", key "80.XXX.XXX.XXX_6e1c38356769095ad76267f9edb43ddae2af1491"): Internal error
[05/Sep/2013:21:50:47 +0000] [xxxxxxx.cleverbridge.com/sid#801739a50][rid#802d760a0][/server-status][1] collections_remove_stale: Failed deleting collection (name "ip", key "85.XXX.XXX.XXX_6e1c38356769095ad76267f9edb43ddae2af1491"): Internal error
[05/Sep/2013:21:50:47 +0000] [xxxxxxx.cleverbridge.com/sid#801739a50][rid#802d760a0][/server-status][1] collections_remove_stale: Failed deleting collection (name "ip", key "85.XXX.XXX.XXX_6e1c38356769095ad76267f9edb43ddae2af1491"): Internal error

The /var/tmp/ip.pag file grew to a size of 7GiB:

% ls -l /var/tmp/ip.pag :-/
-rw-r----- 1 www wheel 7560283136 Sep 6 12:05 /var/tmp/ip.pag

% du -A /var/tmp/ip.pag (FreeBSD userland "du" - values in kbytes)
7383089 /var/tmp/ip.pag

% gdu -b /var/tmp/ip.pag (GNU "du" on FreeBSD)
7560283136 /var/tmp/ip.pag

The SecCollectionTimeout was set to "600".

Though this is a download server, it is not hit that hard, that it should have a collection of 7 GiB of IP-Useragent combinations given
that the values should be overwritten after 10 minutes. Do you have any idea what is going on here? As said, the problem started
again after I enabled (the latest) mod_security version on this server. It was running w/o this problem for almost a year with mod_sec
disabled.

Thanks for you help.
Winfried

@rcbarnett rcbarnett closed this
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.