NSM: avoid filling disk if CRIT_DISK_USAGE exceeded in one day

[GoogleCodeExporter]

NSM: stop netsniff-ng if CRIT_DISK_USAGE exceeded and no more pcaps to delete

Original issue reported on code.google.com by doug.bu...@gmail.com on 21 Aug 2014 at 9:06

[GoogleCodeExporter]

Original comment by doug.bu...@gmail.com on 21 Aug 2014 at 9:14

• Changed title: NSM: avoid filling disk if CRIT_DISK_USAGE exceeded in one day
• Added labels: ****
• Removed labels: ****

[GoogleCodeExporter]

changed /etc/cron.d/sensor-clean from every 5 minutes to every minute:
* * * * * root /usr/sbin/nsm_sensor_clean -y >> /var/log/nsm/sensor-clean.log


changed /usr/lib/nsmnow/lib-nsm-sensor-utils sensor_cleandisk() as follows:

                       # find the oldest dailylogs directory and exclude today
                        OLDEST_DIR=$(ls $SENSOR/dailylogs | sort | grep -v $TODAY | head -n 1)
                        if [ -z "$OLDEST_DIR" -o "$OLDEST_DIR" == ".." -o "$OLDEST_DIR" == "." ]
                        then
                                # there are no pcaps from previous days, so look for pcaps from today that we can delete
                                CUR_USAGE=$(df -P $SENSOR_DIR | grep -v -i filesystem | awk '{print $5}' | tr -d %)
                                # keep looping until EITHER disk usage is no longer critical OR we're out of pcaps to delete
                                while [ "$CUR_USAGE" -gt "$CRIT_DISK_USAGE" ] && [ `ls $SENSOR/dailylogs/$TODAY/snort.log.* | wc -l` -gt 1 ]; do
                                        PCAP=`ls $SENSOR/dailylogs/$TODAY/snort.log.* |head -1`
                                        echo_msg 1 "removing pcap from today's directory: $PCAP"
                                        rm -f "$PCAP"
                                        CUR_USAGE=$(df -P $SENSOR_DIR | grep -v -i filesystem | awk '{print $5}' | tr -d %)
                                done
                                # if we're out of pcaps to delete, then stop writing pcaps
                                if [ `ls $SENSOR/dailylogs/$TODAY/snort.log.* | wc -l` -le 1 ]; then
                                        echo_msg 1 "${RED}no old pcaps available to clean up in $SENSOR/dailylogs/"
                                        echo_msg 1 "${RED}stopping pcap!"
                                        /usr/sbin/nsm_sensor_ps-stop --only-pcap > /dev/null 2>&1
                                fi
                        else
                                # there are pcaps from previous days, so delete the oldest day's worth of pcaps
                                echo_msg 1 "removing directory: $SENSOR/dailylogs/$OLDEST_DIR"
                                rm -rf "$SENSOR"/dailylogs/"$OLDEST_DIR"
                                REMOVED="yes"
                        fi

Original comment by doug.bu...@gmail.com on 28 Aug 2014 at 7:36

• Added labels: ****
• Removed labels: ****

[GoogleCodeExporter]

Submitted for testing:
https://groups.google.com/d/topic/security-onion-testing/2psadomJFeU/discussion

Original comment by doug.bu...@gmail.com on 10 Sep 2014 at 5:36

• Added labels: ****
• Removed labels: ****

[GoogleCodeExporter]

Published:
http://blog.securityonion.net/2014/09/new-securityonion-nsmnow-admin-scripts.htm
l

Original comment by doug.bu...@gmail.com on 10 Sep 2014 at 6:51

• Changed state: Verified
• Added labels: ****
• Removed labels: ****

[GoogleCodeExporter]

The issue with this approach is that the sensor_cleandisk function goes through 
each sensor directory under /nsm/sensor_data/ looking for pcaps to remove.  If 
you have a sensor that is not enabled, it won't find any logs to be removed and 
will issue a nsm_sensor_ps-stop command before it has finished checking all 
sensors.  

The line in question is:
if [ `ls $SENSOR/dailylogs/$TODAY/snort.log.* | wc -l` -le 1 ]; then

This doesn't correctly handle the directory being missing.  

The cleanup process should either ignore sensor directories not listed in 
/etc/nsm/sensortab or it should correctly handle the situation where the today 
daily log directory is not present for a sensor folder.  

My active sensor is eth5.  Every time the sensor_cleandisk() function is 
called, it starts trying to cleanup eth0, finds no pcaps to clear and stops the 
pcap capture process.  It then works its way through to eth5 and cleans up a 
load of files.  The trouble is that pcap capture has been killed prematurely 
and I lose up to a day of valuable pcaps (until the nightly reload of 
processes).  

Original comment by hr...@aqa.org.uk on 14 Oct 2014 at 1:33

• Added labels: ****
• Removed labels: ****

[GoogleCodeExporter]

Hi hreed,

Please see:
https://code.google.com/p/security-onion/issues/detail?id=620

If you have further questions or problems, please use our mailing list.

Thanks!

Original comment by doug.bu...@gmail.com on 14 Oct 2014 at 1:36

• Added labels: ****
• Removed labels: ****