New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Upgrade httpry_agent to http_agent to support Bro logs #265

Closed
GoogleCodeExporter opened this Issue Mar 24, 2015 · 15 comments

Comments

Projects
None yet
1 participant
@GoogleCodeExporter
https://github.com/int13h/http_agent

Original issue reported on code.google.com by doug.bu...@gmail.com on 9 May 2012 at 10:46

@GoogleCodeExporter

This comment has been minimized.

Show comment
Hide comment
@GoogleCodeExporter

GoogleCodeExporter Mar 24, 2015

[deleted comment]
[deleted comment]
@GoogleCodeExporter

This comment has been minimized.

Show comment
Hide comment
@GoogleCodeExporter

GoogleCodeExporter Mar 24, 2015

[deleted comment]
[deleted comment]
@GoogleCodeExporter

This comment has been minimized.

Show comment
Hide comment
@GoogleCodeExporter

GoogleCodeExporter Mar 24, 2015

[deleted comment]
[deleted comment]
@GoogleCodeExporter

This comment has been minimized.

Show comment
Hide comment
@GoogleCodeExporter

GoogleCodeExporter Mar 24, 2015

[deleted comment]
[deleted comment]
@GoogleCodeExporter

This comment has been minimized.

Show comment
Hide comment
@GoogleCodeExporter

GoogleCodeExporter Mar 24, 2015

Updated /usr/local/sbin/nsm_sensor_add as follows:

THE_TIME=$(date)
cat >/etc/nsm/$SENSOR_NAME/http_agent.conf << EOF_HTTP_AGENT_CONF
# DEBUG is VERY chatty. Use it only when needed.
# 1=on 0=off
set DEBUG 1
# Run in background
# 1=yes 0=no
set DAEMON 0
# Name of sguild server
set SERVER_HOST $SENSOR_SERVER_HOST
# Port sguild listens on for sensor connects
set SERVER_PORT $SENSOR_SERVER_PORT
# Local hostname - that means this machines name
# Note: Sensors monitoring multiple interfaces need to use a unique 'hostname'
set HOSTNAME $SENSOR_NAME
# The net id is used to correlate data from different agents.
set NET_GROUP $SENSOR_NET_GROUP
# INVERT MATCHNG
# Only process the domains listed in http_agent.excludes (as opposed to 
excluding them)
# 1=yes 0=no
set INVERT_MATCH 0
# IGNORE EMPTY HOST ENTRIES
# 1=yes 0=no
set EMPTY_HOST 1
# LOG_FORMAT
# httpry or suricata or bro
set LOG_FORMAT bro
EOF_HTTP_AGENT_CONF

THE_TIME=$(date)
cat >/etc/nsm/$SENSOR_NAME/http_agent.exclude << EOF_HTTP_AGENT_EXCLUDE
# Place the domain names or IP addresses that you want 
# to exclude (or include) from processing in this file. 
# 
# Filter by TLD's
#*.ca
#*.com
#*.net
#*.org
#*.co.uk
#
# Filter by FQDN
#*.facebook.com
#*.windowsupdate.com
#*.microsoft.com
#*.dropbox.com
#*.google.ca
#*.google.com
EOF_HTTP_AGENT_EXCLUDE

Original comment by doug.bu...@gmail.com on 9 May 2012 at 12:10

  • Added labels: ****
  • Removed labels: ****
Updated /usr/local/sbin/nsm_sensor_add as follows:

THE_TIME=$(date)
cat >/etc/nsm/$SENSOR_NAME/http_agent.conf << EOF_HTTP_AGENT_CONF
# DEBUG is VERY chatty. Use it only when needed.
# 1=on 0=off
set DEBUG 1
# Run in background
# 1=yes 0=no
set DAEMON 0
# Name of sguild server
set SERVER_HOST $SENSOR_SERVER_HOST
# Port sguild listens on for sensor connects
set SERVER_PORT $SENSOR_SERVER_PORT
# Local hostname - that means this machines name
# Note: Sensors monitoring multiple interfaces need to use a unique 'hostname'
set HOSTNAME $SENSOR_NAME
# The net id is used to correlate data from different agents.
set NET_GROUP $SENSOR_NET_GROUP
# INVERT MATCHNG
# Only process the domains listed in http_agent.excludes (as opposed to 
excluding them)
# 1=yes 0=no
set INVERT_MATCH 0
# IGNORE EMPTY HOST ENTRIES
# 1=yes 0=no
set EMPTY_HOST 1
# LOG_FORMAT
# httpry or suricata or bro
set LOG_FORMAT bro
EOF_HTTP_AGENT_CONF

THE_TIME=$(date)
cat >/etc/nsm/$SENSOR_NAME/http_agent.exclude << EOF_HTTP_AGENT_EXCLUDE
# Place the domain names or IP addresses that you want 
# to exclude (or include) from processing in this file. 
# 
# Filter by TLD's
#*.ca
#*.com
#*.net
#*.org
#*.co.uk
#
# Filter by FQDN
#*.facebook.com
#*.windowsupdate.com
#*.microsoft.com
#*.dropbox.com
#*.google.ca
#*.google.com
EOF_HTTP_AGENT_EXCLUDE

Original comment by doug.bu...@gmail.com on 9 May 2012 at 12:10

  • Added labels: ****
  • Removed labels: ****
@GoogleCodeExporter

This comment has been minimized.

Show comment
Hide comment
@GoogleCodeExporter

GoogleCodeExporter Mar 24, 2015

Update /usr/local/sbin/nsm_sensor_ps-start as follows:

Changed all occurrences of httpry to http

        # start http_agent
        # http_agent is going to read the Bro http.log
        # If Bro is monitoring a single interface, it will be http.log
        # If Bro is monitoring multiple interfaces, the http.log will be per-interface:
        # http_eth0.log, http_eth1.log, etc.
        if grep "^type=standalone$" /usr/local/etc/node.cfg 
        then
                BRO_HTTP_LOG=/nsm/bro/logs/current/http.log
        else
                BRO_HTTP_LOG=/nsm/bro/logs/current/http_$SENSOR_INTERFACE_SHORT.log
        fi
        [ -z "$SKIP_HTTP_AGENT" ] && process_start "http_agent.tcl" "-c /etc/nsm/$SENSOR/http_agent.conf -e /etc/nsm/$SENSOR/http_agent.exclude -f $BRO_HTTP_LOG" "$PROCESS_PID_DIR/$SENSOR/http_agent.pid" "$PROCESS_LOG_DIR/$SENSOR/http_agent.log" "http_agent (sguil)"

Original comment by doug.bu...@gmail.com on 9 May 2012 at 12:26

  • Added labels: ****
  • Removed labels: ****
Update /usr/local/sbin/nsm_sensor_ps-start as follows:

Changed all occurrences of httpry to http

        # start http_agent
        # http_agent is going to read the Bro http.log
        # If Bro is monitoring a single interface, it will be http.log
        # If Bro is monitoring multiple interfaces, the http.log will be per-interface:
        # http_eth0.log, http_eth1.log, etc.
        if grep "^type=standalone$" /usr/local/etc/node.cfg 
        then
                BRO_HTTP_LOG=/nsm/bro/logs/current/http.log
        else
                BRO_HTTP_LOG=/nsm/bro/logs/current/http_$SENSOR_INTERFACE_SHORT.log
        fi
        [ -z "$SKIP_HTTP_AGENT" ] && process_start "http_agent.tcl" "-c /etc/nsm/$SENSOR/http_agent.conf -e /etc/nsm/$SENSOR/http_agent.exclude -f $BRO_HTTP_LOG" "$PROCESS_PID_DIR/$SENSOR/http_agent.pid" "$PROCESS_LOG_DIR/$SENSOR/http_agent.log" "http_agent (sguil)"

Original comment by doug.bu...@gmail.com on 9 May 2012 at 12:26

  • Added labels: ****
  • Removed labels: ****
@GoogleCodeExporter

This comment has been minimized.

Show comment
Hide comment
@GoogleCodeExporter

GoogleCodeExporter Mar 24, 2015

Made similar changes to /usr/local/sbin/nsm_sensor_ps-restart

Original comment by doug.bu...@gmail.com on 9 May 2012 at 12:39

  • Added labels: ****
  • Removed labels: ****
Made similar changes to /usr/local/sbin/nsm_sensor_ps-restart

Original comment by doug.bu...@gmail.com on 9 May 2012 at 12:39

  • Added labels: ****
  • Removed labels: ****
@GoogleCodeExporter

This comment has been minimized.

Show comment
Hide comment
@GoogleCodeExporter

GoogleCodeExporter Mar 24, 2015

For /usr/local/sbin/nsm_sensor_ps-stop and /usr/local/sbin/nsm_sensor_ps-start, 
changed all occurrences of httpry to http

Original comment by doug.bu...@gmail.com on 9 May 2012 at 12:40

  • Added labels: ****
  • Removed labels: ****
For /usr/local/sbin/nsm_sensor_ps-stop and /usr/local/sbin/nsm_sensor_ps-start, 
changed all occurrences of httpry to http

Original comment by doug.bu...@gmail.com on 9 May 2012 at 12:40

  • Added labels: ****
  • Removed labels: ****
@GoogleCodeExporter

This comment has been minimized.

Show comment
Hide comment
@GoogleCodeExporter

GoogleCodeExporter Mar 24, 2015

Removed httpry and httpry_agent from /usr/local/sbin/nsm_sensor_ps-daily-restart

Original comment by doug.bu...@gmail.com on 9 May 2012 at 12:41

  • Added labels: ****
  • Removed labels: ****
Removed httpry and httpry_agent from /usr/local/sbin/nsm_sensor_ps-daily-restart

Original comment by doug.bu...@gmail.com on 9 May 2012 at 12:41

  • Added labels: ****
  • Removed labels: ****
@GoogleCodeExporter

This comment has been minimized.

Show comment
Hide comment
@GoogleCodeExporter

GoogleCodeExporter Mar 24, 2015

Patched http_agent to change "tail -f" to "tail -F"

Original comment by doug.bu...@gmail.com on 9 May 2012 at 2:08

  • Added labels: ****
  • Removed labels: ****
Patched http_agent to change "tail -f" to "tail -F"

Original comment by doug.bu...@gmail.com on 9 May 2012 at 2:08

  • Added labels: ****
  • Removed labels: ****
@GoogleCodeExporter

This comment has been minimized.

Show comment
Hide comment
@GoogleCodeExporter

GoogleCodeExporter Mar 24, 2015

Also patched http_agent to not check for $FILENAME since Bro may not have 
started yet

Original comment by doug.bu...@gmail.com on 9 May 2012 at 4:14

  • Added labels: ****
  • Removed labels: ****
Also patched http_agent to not check for $FILENAME since Bro may not have 
started yet

Original comment by doug.bu...@gmail.com on 9 May 2012 at 4:14

  • Added labels: ****
  • Removed labels: ****
@GoogleCodeExporter

This comment has been minimized.

Show comment
Hide comment
@GoogleCodeExporter

GoogleCodeExporter Mar 24, 2015

Packaged http_agent:

/usr/bin/fpm -s dir -t deb -n securityonion-http-agent -v 20120511 
/usr/local/bin/http_agent.tcl

Original comment by doug.bu...@gmail.com on 10 May 2012 at 12:20

  • Added labels: ****
  • Removed labels: ****
Packaged http_agent:

/usr/bin/fpm -s dir -t deb -n securityonion-http-agent -v 20120511 
/usr/local/bin/http_agent.tcl

Original comment by doug.bu...@gmail.com on 10 May 2012 at 12:20

  • Added labels: ****
  • Removed labels: ****
@GoogleCodeExporter

This comment has been minimized.

Show comment
Hide comment
@GoogleCodeExporter

GoogleCodeExporter Mar 24, 2015

Added to security-onion-upgrade.sh:


sed -i 's| |=|g' $CONF
source $CONF
if [ "$VERSION" = "20120508" ]; then
        NEW="20120511"
        echo "**********************************************"   | $LOGGER
        echo "* Upgrading from $VERSION to $NEW."               | $LOGGER
        echo "**********************************************"   | $LOGGER
        DIR="/nsm/backup/$NEW"
        mkdir -p $DIR                                           | $LOGGER
        cd $DIR

    if pgrep httpry>/dev/null; then
        echo "* Stopping old httpry processes."     | $LOGGER
        nsm_sensor_ps-stop --only-httpry
        nsm_sensor_ps-stop --only-httpry-agent
        pkill -f httpry
        echo ""
    fi

        for FILE in securityonion-nsmnow-admin-scripts_20120511_i386.deb securityonion-http-agent_20120511_i386.deb securityonion-bro-security-onion_20120511_i386.deb; do
                echo -n "* Downloading $FILE..."                | $LOGGER
                wget -q http://sourceforge.net/projects/security-onion/files/$NEW/$FILE -O $FILE | $LOGGER
                if [ $? -eq 1 ]; then
                        echo "FAIL"     | $LOGGER
                        exit 1
                else
                        echo "OK"       | $LOGGER
                fi
        done

        echo -n "* Installing downloaded packages..." | $LOGGER
        dpkg -i *.deb                                           >> $LOG
        if [ $? -eq 1 ]; then
                echo "FAIL"     | $LOGGER
                exit 1
        else
                echo "OK"       | $LOGGER
        fi
    echo ""

    echo "* Updating Bro."         | $LOGGER
    echo "@load security-onion" >> /usr/local/share/bro/site/local.bro
    broctl install      | $LOGGER
    if pgrep -f broctl>/dev/null; then
        broctl restart  | $LOGGER
        sleep 5
        echo ""
    fi

    grep -v "^#" /etc/nsm/sensortab |awk '{print $1}' |while read SENSOR; do
        mv /etc/nsm/$SENSOR/httpry_agent.exclude /etc/nsm/$SENSOR/http_agent.exclude
        mv /etc/nsm/$SENSOR/httpry_agent.conf /etc/nsm/$SENSOR/http_agent.conf
        echo "# LOG_FORMAT" >> /etc/nsm/$SENSOR/http_agent.conf
        echo "# httpry or suricata or bro" >> /etc/nsm/$SENSOR/http_agent.conf
        echo "set LOG_FORMAT bro" >> /etc/nsm/$SENSOR/http_agent.conf
        nsm_sensor_ps-start --sensor-name=$SENSOR --only-http-agent | $LOGGER
    done

        sed -i "s|VERSION=$VERSION|VERSION=$NEW|g" $CONF        | $LOGGER
        echo "* Upgrade to $NEW complete."                      | $LOGGER
        echo 
fi

Original comment by doug.bu...@gmail.com on 10 May 2012 at 12:21

  • Added labels: ****
  • Removed labels: ****
Added to security-onion-upgrade.sh:


sed -i 's| |=|g' $CONF
source $CONF
if [ "$VERSION" = "20120508" ]; then
        NEW="20120511"
        echo "**********************************************"   | $LOGGER
        echo "* Upgrading from $VERSION to $NEW."               | $LOGGER
        echo "**********************************************"   | $LOGGER
        DIR="/nsm/backup/$NEW"
        mkdir -p $DIR                                           | $LOGGER
        cd $DIR

    if pgrep httpry>/dev/null; then
        echo "* Stopping old httpry processes."     | $LOGGER
        nsm_sensor_ps-stop --only-httpry
        nsm_sensor_ps-stop --only-httpry-agent
        pkill -f httpry
        echo ""
    fi

        for FILE in securityonion-nsmnow-admin-scripts_20120511_i386.deb securityonion-http-agent_20120511_i386.deb securityonion-bro-security-onion_20120511_i386.deb; do
                echo -n "* Downloading $FILE..."                | $LOGGER
                wget -q http://sourceforge.net/projects/security-onion/files/$NEW/$FILE -O $FILE | $LOGGER
                if [ $? -eq 1 ]; then
                        echo "FAIL"     | $LOGGER
                        exit 1
                else
                        echo "OK"       | $LOGGER
                fi
        done

        echo -n "* Installing downloaded packages..." | $LOGGER
        dpkg -i *.deb                                           >> $LOG
        if [ $? -eq 1 ]; then
                echo "FAIL"     | $LOGGER
                exit 1
        else
                echo "OK"       | $LOGGER
        fi
    echo ""

    echo "* Updating Bro."         | $LOGGER
    echo "@load security-onion" >> /usr/local/share/bro/site/local.bro
    broctl install      | $LOGGER
    if pgrep -f broctl>/dev/null; then
        broctl restart  | $LOGGER
        sleep 5
        echo ""
    fi

    grep -v "^#" /etc/nsm/sensortab |awk '{print $1}' |while read SENSOR; do
        mv /etc/nsm/$SENSOR/httpry_agent.exclude /etc/nsm/$SENSOR/http_agent.exclude
        mv /etc/nsm/$SENSOR/httpry_agent.conf /etc/nsm/$SENSOR/http_agent.conf
        echo "# LOG_FORMAT" >> /etc/nsm/$SENSOR/http_agent.conf
        echo "# httpry or suricata or bro" >> /etc/nsm/$SENSOR/http_agent.conf
        echo "set LOG_FORMAT bro" >> /etc/nsm/$SENSOR/http_agent.conf
        nsm_sensor_ps-start --sensor-name=$SENSOR --only-http-agent | $LOGGER
    done

        sed -i "s|VERSION=$VERSION|VERSION=$NEW|g" $CONF        | $LOGGER
        echo "* Upgrade to $NEW complete."                      | $LOGGER
        echo 
fi

Original comment by doug.bu...@gmail.com on 10 May 2012 at 12:21

  • Added labels: ****
  • Removed labels: ****
@GoogleCodeExporter

This comment has been minimized.

Show comment
Hide comment
@GoogleCodeExporter

GoogleCodeExporter Mar 24, 2015

Tested by:
Scott Runnels
Tom De Vries
David Zawdie

Original comment by doug.bu...@gmail.com on 10 May 2012 at 12:22

  • Added labels: ****
  • Removed labels: ****
Tested by:
Scott Runnels
Tom De Vries
David Zawdie

Original comment by doug.bu...@gmail.com on 10 May 2012 at 12:22

  • Added labels: ****
  • Removed labels: ****
@GoogleCodeExporter

This comment has been minimized.

Show comment
Hide comment
@GoogleCodeExporter

GoogleCodeExporter Mar 24, 2015

Published:
http://securityonion.blogspot.com/2012/05/security-onion-20120511-now-available.
html

Original comment by doug.bu...@gmail.com on 10 May 2012 at 12:22

  • Changed state: Verified
  • Added labels: ****
  • Removed labels: ****
Published:
http://securityonion.blogspot.com/2012/05/security-onion-20120511-now-available.
html

Original comment by doug.bu...@gmail.com on 10 May 2012 at 12:22

  • Changed state: Verified
  • Added labels: ****
  • Removed labels: ****
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment