rake snorby:soft_reset task is broken #102
Comments
|
Oh, and I technically have 32 sensors (really one box with 32 copies of snort) and about 8000 alerts in the database now, for what that's worth. Most of the problems I was experiencing previously may have been due to having several million alerts, but I've tuned my signatures substantially since then. Notice the cache rebuild dies on Sensor 12... |
|
@Vineyard It looks like you have an event with no source_ip - I'll need to figure out a way to store events without all the proper information. I can't seem to figure out why some peoples setup stores events like that. What was the signature for that event? |
|
@mephux - I did a bit of digging and there are quite a few impossible events in my database. I'm guessing it's something wonky in barnyard2 - either the fact that I've got IPv6 turned on (and the schema's not ready for it yet) or that my sensors don't always shut down gracefully, especially while I'm developing / debugging new code. I don't see any v6 addresses in my alert logs though, so I'm leaning toward the latter. This may take some doing to track down - you mentioned others are having this problem too? |
|
@mephux, since you asked, here are what I believe the 43 offending events (assuming my sql is correct here): mysql> select event.sid, event.cid, signature.sig_sid, signature.sig_name, inet_ntoa(iphdr.ip_src) as ip_src, inet_ntoa(iphdr.ip_dst) as ip_src, event.timestamp from iphdr natural join event join signature on (event.signature=signature.sig_id) where (event.sid, event.cid) in (select iphdr.sid, iphdr.cid from iphdr where ip_src = 0 or ip_dst = 0) order by event.timestamp asc; |
|
Ok, that first one can't be me messing around... there's no way I was up at 5:32am on Saturday :-p |
|
Another dirty hack (I still haven't figured out why I'm getting the bogus records in my database) which seems to make the cache-building workers happy... I'm sure there's a better way to do this but I got tired of working through all the JOINs you have to do to in order to accomplish this with a single query. Since I'm using InnoDB on the backend, the next best thing is to run it all as a single transaction. I've got a cronjob feeding this into mysql every ten minutes: begin;
delete from iphdr where ip_src = 0 or ip_dst = 0;
delete from caches where (sid, cid) not in (select iphdr.sid as sid, iphdr.cid as cid from iphdr);
delete from daily_caches where (sid, cid) not in (select iphdr.sid as sid, iphdr.cid as cid from iphdr);
delete from data where (sid, cid) not in (select iphdr.sid as sid, iphdr.cid as cid from iphdr);
delete from event where (sid, cid) not in (select iphdr.sid as sid, iphdr.cid as cid from iphdr);
delete from favorites where (sid, cid) not in (select iphdr.sid as sid, iphdr.cid as cid from iphdr);
delete from notes where (sid, cid) not in (select iphdr.sid as sid, iphdr.cid as cid from iphdr);
delete from opt where (sid, cid) not in (select iphdr.sid as sid, iphdr.cid as cid from iphdr);
delete from icmphdr where (sid, cid) not in (select iphdr.sid as sid, iphdr.cid as cid from iphdr);
delete from tcphdr where (sid, cid) not in (select iphdr.sid as sid, iphdr.cid as cid from iphdr);
delete from udphdr where (sid, cid) not in (select iphdr.sid as sid, iphdr.cid as cid from iphdr);
commit; |
|
@Vineyard I am also going to build proper handeling of this into the cache jobs. Just to make sure. |
|
@mephux when you say "proper handling" - do you mean deleting the bogus records like I am doing (which is probably the right way to handle them, since none of their other attributes make sense), or just skipping over them when generating the caches? I used to experience the NilClass crash issue much more often, and before I realized the connection between this error and the bogus events in my snort database one of the fixes I implemented was to ensure that required fields were always checked for Nil before populating, ensuring that an invalid event object was never created. Unfortunately, this had severe performance implications (at least the way I implemented it) and did not completely alleviate the problem. On the plus side, after running the sql commands above and modifying the code for the soft_reset task, it appears that I am now able to rebuild my caches and charts at will whenever something gets fubared. Crossing my fingers I won't have to do that very often now that most of the other issues that were affecting me have been resolved. :-) |
|
I had the same problem with my install, and after using vineyards query, it all works again, thank you vineyard for sharing this with us :) Let's hope that somebody locates the problem, and fixes it the right way. |
|
I have the same problem with Snorby 2.3.11. The SQL query above seems to help, but it looks like I'm also going to need a cron job to re-engage the sensor cache job after it fails. |
So I logged in to Snorby this morning and noticed that yet again my dashboard was "stuck" - events were flowing in but the numbers and graphs were not getting updated, this time not even after manually restarting the delayed_job and CacheJob tasks. I did a bit of digging and ran across the :soft_reset task in lib/tasks/snorby.rake, however in its present form it is broken.
On line 63, the code currently reads:
Snorby::Worker.reset_cache(:all, true)Considering that the :reset_cache method is defined in the Jobs class, this is probably more appropriate:
Snorby::Jobs.reset_cache(:all, true)Now, after making that change, things initially look as I would expect, but then it errors out with the dreaded "undefined method `ip_src' for nil:NilClass" error that I haven't otherwise seen since the upgrade to 2.3.2.
I say upgrade, but really it was a pave and rebuild - I dropped my snort database, deleted my old snorby directory, pulled a fresh copy from the git master and started clean with a "rake snorby:setup".
Here's a trace of my modified :soft_reset task:
The text was updated successfully, but these errors were encountered: