Upgrade httpry_agent to http_agent to support Bro logs

[GoogleCodeExporter]

https://github.com/int13h/http_agent

Original issue reported on code.google.com by doug.bu...@gmail.com on 9 May 2012 at 10:46

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

Updated /usr/local/sbin/nsm_sensor_add as follows:

THE_TIME=$(date)
cat >/etc/nsm/$SENSOR_NAME/http_agent.conf << EOF_HTTP_AGENT_CONF
# DEBUG is VERY chatty. Use it only when needed.
# 1=on 0=off
set DEBUG 1
# Run in background
# 1=yes 0=no
set DAEMON 0
# Name of sguild server
set SERVER_HOST $SENSOR_SERVER_HOST
# Port sguild listens on for sensor connects
set SERVER_PORT $SENSOR_SERVER_PORT
# Local hostname - that means this machines name
# Note: Sensors monitoring multiple interfaces need to use a unique 'hostname'
set HOSTNAME $SENSOR_NAME
# The net id is used to correlate data from different agents.
set NET_GROUP $SENSOR_NET_GROUP
# INVERT MATCHNG
# Only process the domains listed in http_agent.excludes (as opposed to 
excluding them)
# 1=yes 0=no
set INVERT_MATCH 0
# IGNORE EMPTY HOST ENTRIES
# 1=yes 0=no
set EMPTY_HOST 1
# LOG_FORMAT
# httpry or suricata or bro
set LOG_FORMAT bro
EOF_HTTP_AGENT_CONF

THE_TIME=$(date)
cat >/etc/nsm/$SENSOR_NAME/http_agent.exclude << EOF_HTTP_AGENT_EXCLUDE
# Place the domain names or IP addresses that you want 
# to exclude (or include) from processing in this file. 
# 
# Filter by TLD's
#*.ca
#*.com
#*.net
#*.org
#*.co.uk
#
# Filter by FQDN
#*.facebook.com
#*.windowsupdate.com
#*.microsoft.com
#*.dropbox.com
#*.google.ca
#*.google.com
EOF_HTTP_AGENT_EXCLUDE

Original comment by doug.bu...@gmail.com on 9 May 2012 at 12:10

• Added labels: ****
• Removed labels: ****

[GoogleCodeExporter]

Update /usr/local/sbin/nsm_sensor_ps-start as follows:

Changed all occurrences of httpry to http

        # start http_agent
        # http_agent is going to read the Bro http.log
        # If Bro is monitoring a single interface, it will be http.log
        # If Bro is monitoring multiple interfaces, the http.log will be per-interface:
        # http_eth0.log, http_eth1.log, etc.
        if grep "^type=standalone$" /usr/local/etc/node.cfg 
        then
                BRO_HTTP_LOG=/nsm/bro/logs/current/http.log
        else
                BRO_HTTP_LOG=/nsm/bro/logs/current/http_$SENSOR_INTERFACE_SHORT.log
        fi
        [ -z "$SKIP_HTTP_AGENT" ] && process_start "http_agent.tcl" "-c /etc/nsm/$SENSOR/http_agent.conf -e /etc/nsm/$SENSOR/http_agent.exclude -f $BRO_HTTP_LOG" "$PROCESS_PID_DIR/$SENSOR/http_agent.pid" "$PROCESS_LOG_DIR/$SENSOR/http_agent.log" "http_agent (sguil)"

Original comment by doug.bu...@gmail.com on 9 May 2012 at 12:26

• Added labels: ****
• Removed labels: ****

[GoogleCodeExporter]

Made similar changes to /usr/local/sbin/nsm_sensor_ps-restart

Original comment by doug.bu...@gmail.com on 9 May 2012 at 12:39

• Added labels: ****
• Removed labels: ****

[GoogleCodeExporter]

For /usr/local/sbin/nsm_sensor_ps-stop and /usr/local/sbin/nsm_sensor_ps-start, 
changed all occurrences of httpry to http

Original comment by doug.bu...@gmail.com on 9 May 2012 at 12:40

• Added labels: ****
• Removed labels: ****

[GoogleCodeExporter]

Removed httpry and httpry_agent from /usr/local/sbin/nsm_sensor_ps-daily-restart

Original comment by doug.bu...@gmail.com on 9 May 2012 at 12:41

• Added labels: ****
• Removed labels: ****

[GoogleCodeExporter]

Patched http_agent to change "tail -f" to "tail -F"

Original comment by doug.bu...@gmail.com on 9 May 2012 at 2:08

• Added labels: ****
• Removed labels: ****

[GoogleCodeExporter]

Also patched http_agent to not check for $FILENAME since Bro may not have 
started yet

Original comment by doug.bu...@gmail.com on 9 May 2012 at 4:14

• Added labels: ****
• Removed labels: ****

[GoogleCodeExporter]

Packaged http_agent:

/usr/bin/fpm -s dir -t deb -n securityonion-http-agent -v 20120511 
/usr/local/bin/http_agent.tcl

Original comment by doug.bu...@gmail.com on 10 May 2012 at 12:20

• Added labels: ****
• Removed labels: ****

[GoogleCodeExporter]

Added to security-onion-upgrade.sh:


sed -i 's| |=|g' $CONF
source $CONF
if [ "$VERSION" = "20120508" ]; then
        NEW="20120511"
        echo "**********************************************"   | $LOGGER
        echo "* Upgrading from $VERSION to $NEW."               | $LOGGER
        echo "**********************************************"   | $LOGGER
        DIR="/nsm/backup/$NEW"
        mkdir -p $DIR                                           | $LOGGER
        cd $DIR

    if pgrep httpry>/dev/null; then
        echo "* Stopping old httpry processes."     | $LOGGER
        nsm_sensor_ps-stop --only-httpry
        nsm_sensor_ps-stop --only-httpry-agent
        pkill -f httpry
        echo ""
    fi

        for FILE in securityonion-nsmnow-admin-scripts_20120511_i386.deb securityonion-http-agent_20120511_i386.deb securityonion-bro-security-onion_20120511_i386.deb; do
                echo -n "* Downloading $FILE..."                | $LOGGER
                wget -q http://sourceforge.net/projects/security-onion/files/$NEW/$FILE -O $FILE | $LOGGER
                if [ $? -eq 1 ]; then
                        echo "FAIL"     | $LOGGER
                        exit 1
                else
                        echo "OK"       | $LOGGER
                fi
        done

        echo -n "* Installing downloaded packages..." | $LOGGER
        dpkg -i *.deb                                           >> $LOG
        if [ $? -eq 1 ]; then
                echo "FAIL"     | $LOGGER
                exit 1
        else
                echo "OK"       | $LOGGER
        fi
    echo ""

    echo "* Updating Bro."         | $LOGGER
    echo "@load security-onion" >> /usr/local/share/bro/site/local.bro
    broctl install      | $LOGGER
    if pgrep -f broctl>/dev/null; then
        broctl restart  | $LOGGER
        sleep 5
        echo ""
    fi

    grep -v "^#" /etc/nsm/sensortab |awk '{print $1}' |while read SENSOR; do
        mv /etc/nsm/$SENSOR/httpry_agent.exclude /etc/nsm/$SENSOR/http_agent.exclude
        mv /etc/nsm/$SENSOR/httpry_agent.conf /etc/nsm/$SENSOR/http_agent.conf
        echo "# LOG_FORMAT" >> /etc/nsm/$SENSOR/http_agent.conf
        echo "# httpry or suricata or bro" >> /etc/nsm/$SENSOR/http_agent.conf
        echo "set LOG_FORMAT bro" >> /etc/nsm/$SENSOR/http_agent.conf
        nsm_sensor_ps-start --sensor-name=$SENSOR --only-http-agent | $LOGGER
    done

        sed -i "s|VERSION=$VERSION|VERSION=$NEW|g" $CONF        | $LOGGER
        echo "* Upgrade to $NEW complete."                      | $LOGGER
        echo 
fi

Original comment by doug.bu...@gmail.com on 10 May 2012 at 12:21

• Added labels: ****
• Removed labels: ****

[GoogleCodeExporter]

Tested by:
Scott Runnels
Tom De Vries
David Zawdie

Original comment by doug.bu...@gmail.com on 10 May 2012 at 12:22

• Added labels: ****
• Removed labels: ****

[GoogleCodeExporter]

Published:
http://securityonion.blogspot.com/2012/05/security-onion-20120511-now-available.
html

Original comment by doug.bu...@gmail.com on 10 May 2012 at 12:22

• Changed state: Verified
• Added labels: ****
• Removed labels: ****