Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

NSM: avoid filling disk if CRIT_DISK_USAGE exceeded in one day #581

Closed
GoogleCodeExporter opened this issue Mar 24, 2015 · 6 comments
Closed

Comments

@GoogleCodeExporter
Copy link

@GoogleCodeExporter GoogleCodeExporter commented Mar 24, 2015

NSM: stop netsniff-ng if CRIT_DISK_USAGE exceeded and no more pcaps to delete

Original issue reported on code.google.com by doug.bu...@gmail.com on 21 Aug 2014 at 9:06

@GoogleCodeExporter

This comment has been minimized.

Copy link
Author

@GoogleCodeExporter GoogleCodeExporter commented Mar 24, 2015

Original comment by doug.bu...@gmail.com on 21 Aug 2014 at 9:14

  • Changed title: NSM: avoid filling disk if CRIT_DISK_USAGE exceeded in one day
  • Added labels: ****
  • Removed labels: ****
@GoogleCodeExporter

This comment has been minimized.

Copy link
Author

@GoogleCodeExporter GoogleCodeExporter commented Mar 24, 2015

changed /etc/cron.d/sensor-clean from every 5 minutes to every minute:
* * * * * root /usr/sbin/nsm_sensor_clean -y >> /var/log/nsm/sensor-clean.log


changed /usr/lib/nsmnow/lib-nsm-sensor-utils sensor_cleandisk() as follows:

                       # find the oldest dailylogs directory and exclude today
                        OLDEST_DIR=$(ls $SENSOR/dailylogs | sort | grep -v $TODAY | head -n 1)
                        if [ -z "$OLDEST_DIR" -o "$OLDEST_DIR" == ".." -o "$OLDEST_DIR" == "." ]
                        then
                                # there are no pcaps from previous days, so look for pcaps from today that we can delete
                                CUR_USAGE=$(df -P $SENSOR_DIR | grep -v -i filesystem | awk '{print $5}' | tr -d %)
                                # keep looping until EITHER disk usage is no longer critical OR we're out of pcaps to delete
                                while [ "$CUR_USAGE" -gt "$CRIT_DISK_USAGE" ] && [ `ls $SENSOR/dailylogs/$TODAY/snort.log.* | wc -l` -gt 1 ]; do
                                        PCAP=`ls $SENSOR/dailylogs/$TODAY/snort.log.* |head -1`
                                        echo_msg 1 "removing pcap from today's directory: $PCAP"
                                        rm -f "$PCAP"
                                        CUR_USAGE=$(df -P $SENSOR_DIR | grep -v -i filesystem | awk '{print $5}' | tr -d %)
                                done
                                # if we're out of pcaps to delete, then stop writing pcaps
                                if [ `ls $SENSOR/dailylogs/$TODAY/snort.log.* | wc -l` -le 1 ]; then
                                        echo_msg 1 "${RED}no old pcaps available to clean up in $SENSOR/dailylogs/"
                                        echo_msg 1 "${RED}stopping pcap!"
                                        /usr/sbin/nsm_sensor_ps-stop --only-pcap > /dev/null 2>&1
                                fi
                        else
                                # there are pcaps from previous days, so delete the oldest day's worth of pcaps
                                echo_msg 1 "removing directory: $SENSOR/dailylogs/$OLDEST_DIR"
                                rm -rf "$SENSOR"/dailylogs/"$OLDEST_DIR"
                                REMOVED="yes"
                        fi

Original comment by doug.bu...@gmail.com on 28 Aug 2014 at 7:36

  • Added labels: ****
  • Removed labels: ****
@GoogleCodeExporter

This comment has been minimized.

Copy link
Author

@GoogleCodeExporter GoogleCodeExporter commented Mar 24, 2015

Submitted for testing:
https://groups.google.com/d/topic/security-onion-testing/2psadomJFeU/discussion

Original comment by doug.bu...@gmail.com on 10 Sep 2014 at 5:36

  • Added labels: ****
  • Removed labels: ****
@GoogleCodeExporter

This comment has been minimized.

Copy link
Author

@GoogleCodeExporter GoogleCodeExporter commented Mar 24, 2015

Published:
http://blog.securityonion.net/2014/09/new-securityonion-nsmnow-admin-scripts.htm
l

Original comment by doug.bu...@gmail.com on 10 Sep 2014 at 6:51

  • Changed state: Verified
  • Added labels: ****
  • Removed labels: ****
@GoogleCodeExporter

This comment has been minimized.

Copy link
Author

@GoogleCodeExporter GoogleCodeExporter commented Mar 24, 2015

The issue with this approach is that the sensor_cleandisk function goes through 
each sensor directory under /nsm/sensor_data/ looking for pcaps to remove.  If 
you have a sensor that is not enabled, it won't find any logs to be removed and 
will issue a nsm_sensor_ps-stop command before it has finished checking all 
sensors.  

The line in question is:
if [ `ls $SENSOR/dailylogs/$TODAY/snort.log.* | wc -l` -le 1 ]; then

This doesn't correctly handle the directory being missing.  

The cleanup process should either ignore sensor directories not listed in 
/etc/nsm/sensortab or it should correctly handle the situation where the today 
daily log directory is not present for a sensor folder.  

My active sensor is eth5.  Every time the sensor_cleandisk() function is 
called, it starts trying to cleanup eth0, finds no pcaps to clear and stops the 
pcap capture process.  It then works its way through to eth5 and cleans up a 
load of files.  The trouble is that pcap capture has been killed prematurely 
and I lose up to a day of valuable pcaps (until the nightly reload of 
processes).  


Original comment by hr...@aqa.org.uk on 14 Oct 2014 at 1:33

  • Added labels: ****
  • Removed labels: ****
@GoogleCodeExporter

This comment has been minimized.

Copy link
Author

@GoogleCodeExporter GoogleCodeExporter commented Mar 24, 2015

Hi hreed,

Please see:
https://code.google.com/p/security-onion/issues/detail?id=620

If you have further questions or problems, please use our mailing list.

Thanks!

Original comment by doug.bu...@gmail.com on 14 Oct 2014 at 1:36

  • Added labels: ****
  • Removed labels: ****
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
1 participant
You can’t perform that action at this time.