diff --git a/.gitignore b/.gitignore index 9cae15f..378eac2 100644 --- a/.gitignore +++ b/.gitignore @@ -1,2 +1 @@ -README.html -trace-summary-*.tgz +build diff --git a/.update-changes.cfg b/.update-changes.cfg index afa5634..04951c7 100644 --- a/.update-changes.cfg +++ b/.update-changes.cfg @@ -4,7 +4,6 @@ function new_version_hook { version=$1 - echo NEW $1 replace_version_in_script trace-summary $version replace_version_in_rst README $version } diff --git a/CHANGES b/CHANGES index 755d9b4..5dfaf27 100644 --- a/CHANGES +++ b/CHANGES @@ -1,8 +1,14 @@ +0.72 | 2011-10-18 10:18:05 -0700 + + * Cleaning up the distribution. (Robin Sommer) + + * Updating README (Jon Siwek) + 0.71-19 | 2011-09-08 12:52:20 -0700 * Now ignoring all lines starting with a pound Closes #602. (Robin - Sommer) + Sommer) * Install binaries with an RPATH (Jon Siwek) diff --git a/COPYING b/COPYING index 03a48c4..5ae3c62 100644 --- a/COPYING +++ b/COPYING @@ -1,35 +1,34 @@ +Copyright (c) 1995-2011, The Regents of the University of California +through the Lawrence Berkeley National Laboratory and the +International Computer Science Institute. All rights reserved. -Copyright (c) 2007-2009, Robin Sommer +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: -All rights reserved. +(1) Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. -Redistribution and use in source and binary forms, with or without -modification, are permitted provided that the following conditions -are met: +(2) Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. - * Redistributions of source code must retain the above copyright - notice, this list of conditions and the following disclaimer. - - * Redistributions in binary form must reproduce the above - copyright notice, this list of conditions and the following - disclaimer in the documentation and/or other materials provided - with the distribution. - - * Neither the name of the International Computer Science - Institute nor the names of its contributors may be used to - endorse or promote products derived from this software without - specific prior written permission. +(3) Neither the name of the University of California, Lawrence Berkeley + National Laboratory, U.S. Dept. of Energy, International Computer + Science Institute, nor the names of contributors may be used to endorse + or promote products derived from this software without specific prior + written permission. -THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS -"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT -LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS -FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE -COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, -INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, -BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; -LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER -CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT -LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN -ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE +LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR +CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF +SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN +CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) +ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. +Note that some files in the distribution may carry their own copyright +notices. diff --git a/Makefile b/Makefile index 73f1fbc..caadd89 100644 --- a/Makefile +++ b/Makefile @@ -1,16 +1,17 @@ -DISTFILES = README README.html COPYING CHANGES Makefile trace-summary +DISTFILES = README COPYING CHANGES Makefile trace-summary -DISTDIR=trace-summary-`cat VERSION` +BUILD=build +DISTDIR=trace-summary-`test -e VERSION && cat VERSION || cat ../VERSION` -docs: README - rst2html.py README >README.html +dist: + @install -d $(BUILD) + rm -rf $(BUILD)/$(DISTDIR) + mkdir $(BUILD)/$(DISTDIR) + cp $(DISTFILES) $(BUILD)/$(DISTDIR) + ( cd $(BUILD) && tar czvf $(DISTDIR).tar.gz $(DISTDIR) ) + rm -rf $(BUILD)/$(DISTDIR) + @echo "Package: $(BUILD)/$(DISTDIR).tar.gz" -dist: docs - rm -rf $(DISTDIR) - mkdir $(DISTDIR) - cp $(DISTFILES) $(DISTDIR) - tar czvf $(DISTDIR).tgz $(DISTDIR) - rm -rf $(DISTDIR) diff --git a/README b/README index d0dc2f4..f8ccb68 100644 --- a/README +++ b/README @@ -1,78 +1,85 @@ .. -*- mode: rst-mode -*- - +.. .. Version number is filled in automatically. -.. |version| replace:: 0.71-19 +.. |version| replace:: 0.72 -trace-summary - Generating network traffic summaries. -===================================================== +==================================================== +trace-summary - Generating network traffic summaries +==================================================== -:Version: |version| +.. class:: opening -.. contents:: + ``trace-summary`` is a Python script that generates break-downs of + network traffic, including lists of the top hosts, protocols, + ports, etc. Optionally, it can generate output separately for + incoming vs. outgoing traffic, per subnet, and per time-interval. -Overview +Download -------- -``trace-summary`` is a Python script which generates break-downs of -network traffic, including lists of the top hosts, protocols, ports, -etc. Optionally, it can generate output separately for incoming vs. -outgoing traffic, per subnet, and per time-interval. +You can find the latest trace-summary release for download at +http://www.bro-ids.org/download. + +trace-summary's git repository is located at `git://git.bro-ids.org/trace-summary.git +`__. You can browse the repository +`here `__. + +This document describes trace-summary |version|. See the `CHANGES +<{{git('trace-summary:CHANGES')}}>`__ file for version history. -The script reads both packet traces in `libpcap -`_ format and connection logs produced by -the `Bro `_ network intrusion detection -system. + +Overview +-------- + +The ``trace-summary`` script reads both packet traces in `libpcap +`_ format and connection logs produced by the +`Bro `_ network intrusion detection system +(for the latter, it support both 1.x and 2.x output formats). Here are two example outputs in the most basic form (note that IP -addresses are 'anonymized'). The first is from a packet trace and -the second from a Bro connection log:: +addresses are 'anonymized'). The first is from a packet trace and the +second from a Bro connection log:: >== Total === 2005-01-06-14-23-33 - 2005-01-06-15-23-43 - - Bytes 918.3m - Payload 846.3m - Pkts 1.8m - Frags 0.9% - MBit/s 1.9 - + - Bytes 918.3m - Payload 846.3m - Pkts 1.8m - Frags 0.9% - MBit/s 1.9 - Ports | Sources | Destinations | Protocols | - 80 33.8% | 131.243.89.214 8.5% | 131.243.89.214 7.7% | 6 76.0% | - 22 16.7% | 128.3.2.102 6.2% | 128.3.2.102 5.4% | 17 23.3% | - 11001 12.4% | 204.116.120.26 4.8% | 131.243.89.4 4.8% | 1 0.5% | - 2049 10.7% | 128.3.161.32 3.6% | 131.243.88.227 3.6% | | - 1023 10.6% | 131.243.89.4 3.5% | 204.116.120.26 3.4% | | - 993 8.2% | 128.3.164.194 2.7% | 131.243.89.64 3.1% | | - 1049 8.1% | 128.3.164.15 2.4% | 128.3.164.229 2.9% | | - 524 6.6% | 128.55.82.146 2.4% | 131.243.89.155 2.5% | | - 33305 4.5% | 131.243.88.227 2.3% | 128.3.161.32 2.3% | | - 1085 3.7% | 131.243.89.155 2.3% | 128.55.82.146 2.1% | | + 80 33.8% | 131.243.89.214 8.5% | 131.243.89.214 7.7% | 6 76.0% | + 22 16.7% | 128.3.2.102 6.2% | 128.3.2.102 5.4% | 17 23.3% | + 11001 12.4% | 204.116.120.26 4.8% | 131.243.89.4 4.8% | 1 0.5% | + 2049 10.7% | 128.3.161.32 3.6% | 131.243.88.227 3.6% | | + 1023 10.6% | 131.243.89.4 3.5% | 204.116.120.26 3.4% | | + 993 8.2% | 128.3.164.194 2.7% | 131.243.89.64 3.1% | | + 1049 8.1% | 128.3.164.15 2.4% | 128.3.164.229 2.9% | | + 524 6.6% | 128.55.82.146 2.4% | 131.243.89.155 2.5% | | + 33305 4.5% | 131.243.88.227 2.3% | 128.3.161.32 2.3% | | + 1085 3.7% | 131.243.89.155 2.3% | 128.55.82.146 2.1% | | >== Total === 2005-01-06-14-23-33 - 2005-01-06-15-23-42 - - Connections 43.4k - Payload 398.4m - + - Connections 43.4k - Payload 398.4m - Ports | Sources | Destinations | Services | Protocols | States | - 80 21.7% | 207.240.215.71 3.0% | 239.255.255.253 8.0% | other 51.0% | 17 55.8% | S0 46.2% | - 427 13.0% | 131.243.91.71 2.2% | 131.243.91.255 4.0% | http 21.7% | 6 36.4% | SF 30.1% | - 443 3.8% | 128.3.161.76 1.7% | 131.243.89.138 2.1% | i-echo 7.3% | 1 7.7% | OTH 7.8% | - 138 3.7% | 131.243.90.138 1.6% | 255.255.255.255 1.7% | https 3.8% | | RSTO 5.8% | - 515 2.4% | 131.243.88.159 1.6% | 128.3.97.204 1.5% | nb-dgm 3.7% | | SHR 4.4% | - 11001 2.3% | 131.243.88.202 1.4% | 131.243.88.107 1.1% | printer 2.4% | | REJ 3.0% | - 53 1.9% | 131.243.89.250 1.4% | 117.72.94.10 1.1% | dns 1.9% | | S1 1.0% | - 161 1.6% | 131.243.89.80 1.3% | 131.243.88.64 1.1% | snmp 1.6% | | RSTR 0.9% | - 137 1.4% | 131.243.90.52 1.3% | 131.243.88.159 1.1% | nb-ns 1.4% | | SH 0.3% | - 2222 1.1% | 128.3.161.252 1.2% | 131.243.91.92 1.1% | ntp 1.0% | | RSTRH 0.2% | - + 80 21.7% | 207.240.215.71 3.0% | 239.255.255.253 8.0% | other 51.0% | 17 55.8% | S0 46.2% | + 427 13.0% | 131.243.91.71 2.2% | 131.243.91.255 4.0% | http 21.7% | 6 36.4% | SF 30.1% | + 443 3.8% | 128.3.161.76 1.7% | 131.243.89.138 2.1% | i-echo 7.3% | 1 7.7% | OTH 7.8% | + 138 3.7% | 131.243.90.138 1.6% | 255.255.255.255 1.7% | https 3.8% | | RSTO 5.8% | + 515 2.4% | 131.243.88.159 1.6% | 128.3.97.204 1.5% | nb-dgm 3.7% | | SHR 4.4% | + 11001 2.3% | 131.243.88.202 1.4% | 131.243.88.107 1.1% | printer 2.4% | | REJ 3.0% | + 53 1.9% | 131.243.89.250 1.4% | 117.72.94.10 1.1% | dns 1.9% | | S1 1.0% | + 161 1.6% | 131.243.89.80 1.3% | 131.243.88.64 1.1% | snmp 1.6% | | RSTR 0.9% | + 137 1.4% | 131.243.90.52 1.3% | 131.243.88.159 1.1% | nb-ns 1.4% | | SH 0.3% | + 2222 1.1% | 128.3.161.252 1.2% | 131.243.91.92 1.1% | ntp 1.0% | | RSTRH 0.2% | -Download --------- - -Download the ``trace-summary`` git repository like:: - - > git clone git://git.bro-ids.org/trace-summary - -Or if you don't have git, the `latest snapshot `_. Prerequisites ------------- * This script requires Python 2.4 or newer. -* The `pysubnettree `_ - Python module + +* The `pysubnettree + `_ Python + module. + * Eddie Kohler's `ipsumdump `_ if using ``trace-summary`` with packet traces (versus Bro connection logs) @@ -84,21 +91,21 @@ Simply copy the script into some directory which is in your ``PATH``. Usage ----- -The general usage is +The general usage is:: trace-summary [options] [input-file] Per default, it assumes the ``input-file`` to be a ``libpcap`` trace file. If it is a Bro connection log, use ``-c``. If ``input-file`` is not given, the script reads from stdin. It writes its output to -stdout. +stdout. Options ~~~~~~~ There are a bunch of options. The most important ones summmarized below. Run ``trace-summary \--help`` to see the full list including -some more estoric ones. +some more estoric ones. :-c: Input is a Bro connection log instead of a ``libpcap`` trace @@ -107,12 +114,12 @@ some more estoric ones. :-b: Counts all percentages in bytes rather than number of packets/connections. - + :-E : Gives a file which contains a list of networks to ignore for the analysis. The file must contain one network per line, where each network is of the CIDR form ``a.b.c.d/mask``. Empty lines and - lines starting with a "#" are ignored. + lines starting with a "#" are ignored. :-i : Creates totals for each time interval of the given length @@ -124,12 +131,12 @@ some more estoric ones. Generates separate summaries for incoming and outgoing traffic. ```` is a file which contains a list of networks to be considered local. Format as for ``-E``. - + :-n : Show top n entries in each break-down. Default is 10. - + :-r: - Resolves hostnames in the output. + Resolves hostnames in the output. :-s : Gives the sample factor if the input has been sampled. diff --git a/README.html b/README.html deleted file mode 100644 index d80bf60..0000000 --- a/README.html +++ /dev/null @@ -1,449 +0,0 @@ - - - - - - -trace-summary - Generating network traffic summaries. - - - -
-

trace-summary - Generating network traffic summaries.

- - -

Date: 2011-08-03

- -
-

Overview

-

trace-summary is a Python script which generates break-downs of -network traffic, including lists of the top hosts, protocols, ports, -etc. Optionally, it can generate output separately for incoming vs. -outgoing traffic, per subnet, and per time-interval.

-

The script reads both packet traces in libpcap format and connection logs produced by -the Bro network intrusion detection -system.

-

Here are two example outputs in the most basic form (note that IP -addresses are 'anonymized'). The first is from a packet trace and -the second from a Bro connection log:

-
->== Total === 2005-01-06-14-23-33 - 2005-01-06-15-23-43
-  - Bytes 918.3m - Payload 846.3m - Pkts 1.8m - Frags   0.9% - MBit/s      1.9 -
-    Ports        | Sources                   | Destinations              | Protocols |
-    80     33.8% | 131.243.89.214       8.5% | 131.243.89.214       7.7% | 6   76.0% |
-    22     16.7% | 128.3.2.102          6.2% | 128.3.2.102          5.4% | 17  23.3% |
-    11001  12.4% | 204.116.120.26       4.8% | 131.243.89.4         4.8% | 1    0.5% |
-    2049   10.7% | 128.3.161.32         3.6% | 131.243.88.227       3.6% |           |
-    1023   10.6% | 131.243.89.4         3.5% | 204.116.120.26       3.4% |           |
-    993     8.2% | 128.3.164.194        2.7% | 131.243.89.64        3.1% |           |
-    1049    8.1% | 128.3.164.15         2.4% | 128.3.164.229        2.9% |           |
-    524     6.6% | 128.55.82.146        2.4% | 131.243.89.155       2.5% |           |
-    33305   4.5% | 131.243.88.227       2.3% | 128.3.161.32         2.3% |           |
-    1085    3.7% | 131.243.89.155       2.3% | 128.55.82.146        2.1% |           |
-
-
->== Total === 2005-01-06-14-23-33 - 2005-01-06-15-23-42
-  - Connections 43.4k - Payload 398.4m -
-    Ports        | Sources                   | Destinations              | Services           | Protocols | States        |
-    80     21.7% | 207.240.215.71       3.0% | 239.255.255.253      8.0% | other        51.0% | 17  55.8% | S0      46.2% |
-    427    13.0% | 131.243.91.71        2.2% | 131.243.91.255       4.0% | http         21.7% | 6   36.4% | SF      30.1% |
-    443     3.8% | 128.3.161.76         1.7% | 131.243.89.138       2.1% | i-echo        7.3% | 1    7.7% | OTH      7.8% |
-    138     3.7% | 131.243.90.138       1.6% | 255.255.255.255      1.7% | https         3.8% |           | RSTO     5.8% |
-    515     2.4% | 131.243.88.159       1.6% | 128.3.97.204         1.5% | nb-dgm        3.7% |           | SHR      4.4% |
-    11001   2.3% | 131.243.88.202       1.4% | 131.243.88.107       1.1% | printer       2.4% |           | REJ      3.0% |
-    53      1.9% | 131.243.89.250       1.4% | 117.72.94.10         1.1% | dns           1.9% |           | S1       1.0% |
-    161     1.6% | 131.243.89.80        1.3% | 131.243.88.64        1.1% | snmp          1.6% |           | RSTR     0.9% |
-    137     1.4% | 131.243.90.52        1.3% | 131.243.88.159       1.1% | nb-ns         1.4% |           | SH       0.3% |
-    2222    1.1% | 128.3.161.252        1.2% | 131.243.91.92        1.1% | ntp           1.0% |           | RSTRH    0.2% |
-
-
- -
-

Prerequisites

-
    -
  • This script requires Python 2.4 or newer.
    • -
  • It also requires the installation of -- the pysubnettree Python module, and -- Eddie Kohler's ipsumdump
    • -
-
-
-

Installation

-

Simply copy the script into some directory which is in your PATH.

-
-
-

Usage

-

The general usage is

-
-trace-summary [options] [input-file]
-

Per default, it assumes the input-file to be a libpcap trace -file. If it is a Bro connection log, use -c. If input-file is -not given, the script reads from stdin. It writes its output to -stdout.

-
-

Options

-

There are a bunch of options. The most important ones summmarized -below. Run trace-summary \--help to see the full list including -some more estoric ones.

- --- - - - - - - - - - - - - - - - - - - - - - - - -
-c:Input is a Bro connection log instead of a libpcap trace -file.
-b:Counts all percentages in bytes rather than number of -packets/connections.
-E <file>:Gives a file which contains a list of networks to ignore for the -analysis. The file must contain one network per line, where each -network is of the CIDR form a.b.c.d/mask. Empty lines and -lines starting with a "#" are ignored.
-i <duration>:Creates totals for each time interval of the given length -(default is seconds; add "m" for minutes and "h" for -hours). Use -v if you also want to see the breakdowns for -each interval.
-l <file>:Generates separate summaries for incoming and outgoing traffic. -<file> is a file which contains a list of networks to be -considered local. Format as for -E.
-n <n>:Show top n entries in each break-down. Default is 10.
-r:Resolves hostnames in the output.
-s <n>:Gives the sample factor if the input has been sampled.
-S <n>:Sample input with the given factor; less accurate but faster and -saves memory.
-m:Does skip memory-expensive statistics.
-v:Generates full break-downs for each time interval. Requires --i.
-
-
-
- - diff --git a/VERSION b/VERSION index 81f0279..b214dd9 100644 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -0.71-19 +0.72 diff --git a/trace-summary b/trace-summary index 3063b1b..59bda12 100644 --- a/trace-summary +++ b/trace-summary @@ -12,7 +12,7 @@ import optparse import random import subprocess -VERSION = "0.71-19" # Automatically filled in. +VERSION = "0.72" # Automatically filled in. random.seed()