Skip to content

Persistent session SDBM file size increases indefinitely #314

Closed
rcbarnett opened this Issue Oct 17, 2013 · 11 comments

2 participants

@rcbarnett

MODSEC-160: Whenever we set any rule that utilizes the session collection, as clients renew their session cookie the session.pag file size increases indefinitely. Looks like "setvar:session.TIMEOUT" is ignored.

SecDataDir /var/www/modsecurity/data
SecRule REQUEST_COOKIES:frontend !^$ \
"nolog,phase:2,pass,\
initcol:session=%{REQUEST_COOKIES.frontend},\
setsid:%{REQUEST_COOKIES.frontend}"
SecRule SESSION:IS_NEW "@eq 1" \
"phase:2,nolog,pass,skip:1,\
setvar:session.TIMEOUT=1860,\
setvar:session.ip=%{REMOTE_ADDR},\
expirevar:session.ip=1800"
SecRule SESSION:ip "!@streq %{REMOTE_ADDR}" \
"phase:2,pass,log,\
msg:'Possible hijacking in session %{SESSIONID}, address %{SESSION.ip} was expected but %{REMOTE_ADDR} was used'"

We've tested this with 2.5.9 and 2.5.12 versions, no other version tested. Are we doing anything wrong?

Your help would be very appreciated. Thanks in advance!

@rcbarnett

Original reporter: amoreton

@rcbarnett

brectanus: I have not used it on Windows, so not sure that it is completly functioning there. If you try something like this to verify TIMEOUT is working:

SecRule REQUEST_COOKIES:frontend !^$ \
"nolog,phase:2,pass,\
initcol:session=%{REQUEST_COOKIES.frontend},\
setsid:%{REQUEST_COOKIES.frontend}"

Added logging to debug expirations

SecRule SESSION:IS_NEW "@eq 1" \
"phase:2,log,pass,skip:1,\
setvar:session.TIMEOUT=1860,\
setvar:session.ip=%{REMOTE_ADDR},\
expirevar:session.ip=1800,\
msg:'new session'"

SecRule SESSION:ip "!@streq %{REMOTE_ADDR}" \
"phase:2,pass,log,\
msg:'Possible hijacking in session %{SESSIONID}, address %{SESSION.ip} was expected but %{REMOTE_ADDR} was used'"

Then, make a request, verify the "new session" is logged, make further requests and verify the "new session" is NOT logged. Then wait the timeout and make another request from the same session and see if you get the "new session" log again.

What appears to be happening is that the periodic cleanup routine is not running correctly. This routine is supposed to run every 100 requests or so. In the debug log you should see periodic "Garbage collection took N microseconds." at level 4. Do you see these? At debug level 9, you should also get a dump of the data in the DB, but Christian Bockermann has a tool (CollectionViewer) to make this easier (http://www.jwall.org/tools/org.jwall.tools.jar). See some explanation here: https://secure.jwall.org/blog/2009/01/25/1232920799926.html

We do need to add a utility to do offline cleanup of the collections via cron and similar.

@rcbarnett

amoreton: Nice to meet you Brian, thanks for your soon answer!

Last two days I've been repeating my tests,

*) Apache/2.2.9 + ModSecurity for Apache/2.5.12
*) Both at Windows 2003 R2 SP2 and Centos Linux 5.5
*) SecDebugLogLevel 4
*) 40000 HTTP requests per test, 10 requests per session, sessionid is never reutilized after 10 requests

with the same results,

*) OK, verify the "new session" is logged, make further requests and verify the "new session" is NOT logged. Then wait the timeout and make another request from the same session and see if you get the "new session" log again
*) OK, TIMEOUT is working somehow properly (created collections are always marked as stale, org.jwall.tools.CollectionViewer correctly states "expired" or "expires" depending on session time annotations)

[09/Jun/2010:10:11:42 +0200] [---host---/sid#1d49fd8][rid#2091120][---uri---][4] Creating collection (name "session", key "---sessionid---").
[09/Jun/2010:10:11:43 +0200] [---host---/sid#1d49fd8][rid#2091120][---uri---][4] Persisted collection (name "session", key "---sessionid---").
[09/Jun/2010:10:12:26 +0200] [---host---/sid#1d49fd8][rid#209b140][---uri---][4] Retrieved collection (name "session", key "---sessionid---").
[...]
[09/Jun/2010:10:51:46 +0200] [---host---/sid#1d49fd8][rid#250f0f8][---uri---][4] Removed stale collection (name "session", key "---sessionid---").

*) NOK, in the debug log we don't see any "Garbage collection took N microseconds" entrance (no "grep -i gargabe" matching lines after thousands of requests in ~1Gb debug log)

As you state, seems that the periodic cleanup routine is not running correctly, please notice this also affects Linux.

Thanks for the reference to jWall tools, we have now a workaround. We periodically compact session.pag online, running the following Java class,

import java.io.File;
import org.solinger.sdbm.Sdbm;

public class Purge {
public Purge() {
}
public static void main(String[] args) throws Exception {
Sdbm db = new Sdbm(new File(args[0]),args[1],"rw");
System.out.println("initial elementCount="+db.size());
long start = System.currentTimeMillis();
db.clean();
long end = System.currentTimeMillis();
System.out.println("final elementCount="+db.size()+", execution time was "+(end-start)+" ms");
}

}

We've tested compaction time doesn't affect client HTTP requests (no delay and no error) but has some side effects (no influence in our business, we're not a nuclear power plant),

*) mod_security could add new sessions to original SDBM file which could get lost in new compacted SDMB file (Purge class doesn't lock the original one)

*) mod_security can't delete some sessions expected to be in SDMB file but already purged in new compacted SDBM file

[Fri Jun 11 10:39:42 2010] [error] [client ---clientip---] ModSecurity: Failed deleting collection (name "session", key "---sessionid---"): Internal error [hostname "---host---"] [uri "---uri---"] [unique_id "TBH2TKweATIAAE40Qq8AAADK"]

I hope JIRA advises when this issue gets solved... Thanks again!

@rcbarnett

bpinto: Hi,

Fixed some bugs in gargabe collection code. Also added SecDefaultCollectionTimeout option to set default collection timeout in seconds.

Looks like it is working fine on Linux. Need to be tested under Windows.

@zimmerle zimmerle was assigned Oct 17, 2013
@rcbarnett

marcstern: What is the exact fix for the bugs in gargabe collection code?

The only fix I see in revision 1853 is "srand(time(NULL));". Is this supposed to be sufficient or is the SecDefaultCollectionTimeout option also needed?

@rcbarnett

bpinto: SecDefaultCollectionTImeout is a good option.
But the fix was point the right filename to be opened by garbage collection code. ModSecurity was trying to open a unknown file.

@rcbarnett

bpinto: SecCollectionTImeout is 3600s by default. This number looks to high for me. I just keep it for compatible reasons.
Depending of what kind of traffic env, resources, collections and rules you are using.. it can be adjusted to a value to make the file size stable.

@rcbarnett

marcstern: What is the exact fix? What filename has to be corrected?

@rcbarnett

bpinto: we must consider namespace for some sdbm files

@rcbarnett

marcstern: I have the problem with RESOURCE, so it's not that one.
I also use 3600 on a very high load server. Could the file become too big? Is there a limitation on max. size?

@rcbarnett

bpinto: Yes Marc, 3600 is too much. I think you can try few minutes.

Try 180/300. Monitor your file size for 1 or 2 days i think the size will be stable

@rcbarnett rcbarnett closed this Oct 17, 2013
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.