Merge branch 'master' into topic/jsiwek/file-signatures

Conflicts:
	scripts/policy/frameworks/files/detect-MHR.bro

CHANGES

@@ -1,4 +1,116 @@
 
+2.2-250 | 2014-03-19 17:20:55 -0400
+
+  * Improve performance of MHR script by reducing cloned Vals in
+    a "when" scope. (Jon Siwek)
+
+2.2-248 | 2014-03-19 14:47:40 -0400
+
+  * Make SumStats work incrementally and non-blocking in non-cluster
+    mode, but force it to operate by blocking if Bro is shutting
+    down. (Seth Hall)
+
+2.2-244 | 2014-03-17 08:24:17 -0700
+
+  * Fix compile errror on FreeBSD caused by wrong include file order.
+    (Bernhard Amann)
+
+2.2-240 | 2014-03-14 10:23:54 -0700
+
+  * Derive results of DNS lookups from from input when in BRO_DNS_FAKE
+    mode. Addresses BIT-1134. (Jon Siwek)
+
+  * Fixing a few cases of undefined behaviour introduced by recent
+    formatter work.
+
+  * Fixing compiler error. (Robin Sommer)
+
+  * Fixing (very unlikely) double delete in HTTP analyzer when
+    decapsulating CONNECTs. (Robin Sommer)
+
+2.2-235 | 2014-03-13 16:21:19 -0700
+
+  * The Ascii writer has a new option LogAscii::use_json for writing
+	out logs as JSON. (Seth Hall)
+
+  * Ascii input reader now supports all config options as per-input
+    stream "config" values. (Seth Hall)
+
+  * Refactored formatters and updated the the writers a bit. (Seth
+	Hall)
+
+2.2-229 | 2014-03-13 14:58:30 -0700
+
+  * Refactoring analyzer manager code to reuse
+    ApplyScheduledAnalyzers(). (Robin Sommer)
+
+2.2-228 | 2014-03-13 14:25:53 -0700
+
+  * Teach async DNS lookup builtin-functions about BRO_DNS_FAKE.
+    Addresses BIT-1134. (Jon Siwek)
+
+  * Enable fake DNS mode for test suites.
+
+  * Improve analysis of TCP SYN/SYN-ACK reversal situations. (Jon
+    Siwek)
+
+    - Since it's just the handshake packets out of order, they're no
+      longer treated as partial connections, which some protocol analyzers
+      immediately refuse to look at.
+
+    - The TCP_Reassembler "is_orig" state failed to change, which led to
+      protocol analyzers sometimes using the wrong value for that.
+
+    - Add a unit test which exercises the Connection::FlipRoles() code
+      path (i.e. the SYN/SYN-ACK reversal situation).
+
+    Addresses BIT-1148.
+
+  * Fix bug in Connection::FlipRoles. It didn't swap address values
+    right and also didn't consider that analyzers might be scheduled
+    for the new connection tuple. Reported by Kevin McMahon. Addresses
+    BIT-1148. (Jon Siwek)
+
+2.2-221 | 2014-03-12 17:23:18 -0700
+
+  * Teach configure script --enable-jemalloc, --with-jemalloc.
+    Addresses BIT-1128. (Jon Siwek)
+
+2.2-218 | 2014-03-12 17:19:45 -0700
+
+  * Improve DBG_LOG macro (perf. improvement for --enable-debug mode).
+    (Jon Siwek)
+
+  * Silences some documentation warnings from Sphinx. (Jon Siwek)
+
+2.2-215 | 2014-03-10 11:10:15 -0700
+
+  * Fix non-deterministic logging of unmatched DNS msgs. Addresses
+    BIT-1153 (Jon Siwek)
+
+2.2-213 | 2014-03-09 08:57:37 -0700
+
+  * No longer accidentally attempting to parse NBSTAT RRs as SRV RRs
+    in DNS analyzer. (Seth Hall)
+
+  * Fix DNS SRV responses and a small issue with NBNS queries and
+    label length. (Seth Hall)
+
+     - DNS SRV responses never had the code written to actually
+       generate the dns_SRV_reply event.  Adding this required
+       extending the event a bit to add extra information.  SRV responses
+       now appear in the dns.log file correctly.
+
+     - Fixed an issue where some Microsoft NetBIOS Name Service lookups
+       would exceed the max label length for DNS and cause an incorrect
+       "DNS_label_too_long" weird.
+
+2.2-210 | 2014-03-06 22:52:36 -0500
+
+  * Improve SSL logging so that connections are logged even when the
+    ssl_established event is not generated as well as other small SSL
+    fixes. (Bernhard Amann)
+
 2.2-206 | 2014-03-03 16:52:28 -0800
 
   * HTTP CONNECT proxy support. The HTTP analyzer now supports

CMakeLists.txt

@@ -60,6 +60,10 @@ if (NOT BinPAC_ROOT_DIR AND
 endif ()
 FindRequiredPackage(BinPAC)
 
+if (ENABLE_JEMALLOC)
+    find_package(JeMalloc)
+endif ()
+
 if (MISSING_PREREQS)
     foreach (prereq ${MISSING_PREREQ_DESCS})
         message(SEND_ERROR ${prereq})
@@ -73,6 +77,7 @@ include_directories(BEFORE
                     ${BIND_INCLUDE_DIR}
                     ${BinPAC_INCLUDE_DIR}
                     ${ZLIB_INCLUDE_DIR}
+                    ${JEMALLOC_INCLUDE_DIR}
 )
 
 # Optional Dependencies
@@ -151,6 +156,7 @@ set(brodeps
     ${OpenSSL_LIBRARIES}
     ${BIND_LIBRARY}
     ${ZLIB_LIBRARY}
+    ${JEMALLOC_LIBRARIES}
     ${OPTLIBS}
 )
 
@@ -231,6 +237,7 @@ message(
     "\ngperftools found:  ${HAVE_PERFTOOLS}"
     "\n        tcmalloc:  ${USE_PERFTOOLS_TCMALLOC}"
     "\n       debugging:  ${USE_PERFTOOLS_DEBUG}"
+    "\njemalloc:          ${ENABLE_JEMALLOC}"
     "\ncURL:              ${USE_CURL}"
     "\n"
     "\nDataSeries:        ${USE_DATASERIES}"

NEWS

@@ -25,6 +25,9 @@ New Functionality
   parsing past the GRE header in between the delivery and payload IP
   packets.
 
+- The DNS analyzer now actually generates the dns_SRV_reply() event.
+  It had been documented before, yet was never raised.
+
 Changed Functionality
 ---------------------
 

VERSION

@@ -1 +1 @@
-2.2-206
+2.2-250

configure

@@ -32,6 +32,7 @@ Usage: $0 [OPTION]... [VAR=VALUE]...
     --enable-perftools     force use of Google perftools on non-Linux systems
                            (automatically on when perftools is present on Linux)
     --enable-perftools-debug use Google's perftools for debugging
+    --enable-jemalloc      link against jemalloc
     --enable-ruby          build ruby bindings for broccoli (deprecated)
     --disable-broccoli     don't build or install the Broccoli library
     --disable-broctl       don't install Broctl
@@ -54,6 +55,7 @@ Usage: $0 [OPTION]... [VAR=VALUE]...
   Optional Packages in Non-Standard Locations:
     --with-geoip=PATH      path to the libGeoIP install root
     --with-perftools=PATH  path to Google Perftools install root
+    --with-jemalloc=PATH   path to jemalloc install root
     --with-python=PATH     path to Python interpreter
     --with-python-lib=PATH path to libpython
     --with-python-inc=PATH path to Python headers
@@ -105,6 +107,7 @@ append_cache_entry BRO_ETC_INSTALL_DIR  PATH   $prefix/etc
 append_cache_entry ENABLE_DEBUG         BOOL   false
 append_cache_entry ENABLE_PERFTOOLS     BOOL   false
 append_cache_entry ENABLE_PERFTOOLS_DEBUG BOOL false
+append_cache_entry ENABLE_JEMALLOC      BOOL false
 append_cache_entry BinPAC_SKIP_INSTALL  BOOL   true
 append_cache_entry BUILD_SHARED_LIBS    BOOL   true
 append_cache_entry INSTALL_AUX_TOOLS    BOOL   true
@@ -160,6 +163,9 @@ while [ $# -ne 0 ]; do
             append_cache_entry ENABLE_PERFTOOLS     BOOL   true
             append_cache_entry ENABLE_PERFTOOLS_DEBUG BOOL   true
             ;;
+        --enable-jemalloc)
+            append_cache_entry ENABLE_JEMALLOC     BOOL   true
+            ;;
         --disable-broccoli)
             append_cache_entry INSTALL_BROCCOLI     BOOL   false
             ;;
@@ -214,6 +220,10 @@ while [ $# -ne 0 ]; do
         --with-perftools=*)
             append_cache_entry GooglePerftools_ROOT_DIR PATH $optarg
             ;;
+        --with-jemalloc=*)
+            append_cache_entry JEMALLOC_ROOT_DIR    PATH    $optarg
+            append_cache_entry ENABLE_JEMALLOC      BOOL    true
+            ;;
         --with-python=*)
             append_cache_entry PYTHON_EXECUTABLE    PATH    $optarg
             ;;

doc/broids/index.rst

@@ -15,9 +15,9 @@ conditions specific to your particular case.
 In the following sections, we present a few examples of common uses of
 Bro as an IDS.
 
-------------------------------------------------
+-------------------------------------------------
 Detecting an FTP Brute-force Attack and Notifying
-------------------------------------------------
+-------------------------------------------------
 
 For the purpose of this exercise, we define FTP brute-forcing as too many
 rejected usernames and passwords occurring from a single address.  We

doc/index.rst

@@ -17,6 +17,8 @@ Introduction Section
 
 .. 
 
+.. _using-bro:
+
 Using Bro Section
 =================
 

doc/quickstart/index.rst

@@ -292,9 +292,10 @@ tweak the most basic options.  Here's some suggestions on what to explore next:
 
 * We only looked at how to change options declared in the notice framework,
   there's many more options to look at in other script packages.
-* Continue reading with :ref:`using-bro` chapter which goes into more
-  depth on working with Bro; then look at :ref:`writing-scripts` for
-  learning how to start writing your own scripts.
+* Continue reading with :ref:`Using Bro <using-bro>` chapter which goes
+  into more depth on working with Bro; then look at
+  :ref:`writing-scripts` for learning how to start writing your own
+  scripts.
 * Look at the scripts in ``$PREFIX/share/bro/policy`` for further ones
   you may want to load; you can browse their documentation at the
   :ref:`overview of script packages <script-packages>`.

doc/scripting/index.rst

@@ -345,13 +345,13 @@ keyword.  Unlike globals, constants can only be set or altered at
 parse time if the ``&redef`` attribute has been used.  Afterwards (in
 runtime) the constants are unalterable.  In most cases, re-definable
 constants are used in Bro scripts as containers for configuration
-options.  For example, the configuration option to log password
+options.  For example, the configuration option to log passwords
 decrypted from HTTP streams is stored in
-``HTTP::default_capture_password`` as shown in the stripped down
+:bro:see:`HTTP::default_capture_password` as shown in the stripped down
 excerpt from :doc:`/scripts/base/protocols/http/main.bro` below.
 
 .. btest-include:: ${BRO_SRC_ROOT}/scripts/base/protocols/http/main.bro
-   :lines: 8-10,19-21,120
+   :lines: 9-11,20-22,121
 
 Because the constant was declared with the ``&redef`` attribute, if we
 needed to turn this option on globally, we could do so by adding the

scripts/base/frameworks/logging/writers/ascii.bro

@@ -5,39 +5,63 @@
 ##! ``config``: setting ``tsv`` to the string ``T`` turns the output into
 ##! "tab-separated-value" mode where only a single header row with the column
 ##! names is printed out as meta information, with no "# fields" prepended; no
-##! other meta data gets included in that mode.   
-##! 
+##! other meta data gets included in that mode.
+##!
 ##! Example filter using this::
-##! 
-##!    local my_filter: Log::Filter = [$name = "my-filter", $writer = Log::WRITER_ASCII, $config = table(["tsv"] = "T")]; 
+##!
+##!    local my_filter: Log::Filter = [$name = "my-filter", $writer = Log::WRITER_ASCII, $config = table(["tsv"] = "T")];
 ##!
 
 module LogAscii;
 
 export {
 	## If true, output everything to stdout rather than
 	## into files. This is primarily for debugging purposes.
+	##
+	## This option is also available as a per-filter ``$config`` option.
 	const output_to_stdout = F &redef;
 
+	## If true, the default will be to write logs in a JSON format.
+	##
+	## This option is also available as a per-filter ``$config`` option.
+	const use_json = F &redef;
+
+	## Format of timestamps when writing out JSON. By default, the JSON formatter will
+	## use double values for timestamps which represent the number of seconds from the
+	## UNIX epoch.
+	const json_timestamps: JSON::TimestampFormat = JSON::TS_EPOCH &redef;
+
 	## If true, include lines with log meta information such as column names
 	## with types, the values of ASCII logging options that are in use, and
 	## the time when the file was opened and closed (the latter at the end).
+        ##
+	## If writing in JSON format, this is implicitly disabled.
 	const include_meta = T &redef;
 
 	## Prefix for lines with meta information.
+        ##
+	## This option is also available as a per-filter ``$config`` option.
 	const meta_prefix = "#" &redef;
 
 	## Separator between fields.
+	##
+	## This option is also available as a per-filter ``$config`` option.
 	const separator = Log::separator &redef;
 
 	## Separator between set elements.
+	##
+	## This option is also available as a per-filter ``$config`` option.
 	const set_separator = Log::set_separator &redef;
 
 	## String to use for empty fields. This should be different from
-        ## *unset_field* to make the output unambiguous. 
+	## *unset_field* to make the output unambiguous.
+	##
+	## This option is also available as a per-filter ``$config`` option.
 	const empty_field = Log::empty_field &redef;
 
 	## String to use for an unset &optional field.
+	##
+	## This option is also available as a per-filter ``$config`` option.
 	const unset_field = Log::unset_field &redef;
 }