An efficient C implementation of the network flow query language (NFQL)
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
benchmarks
examples
scripts
src
tests
.gitignore
CMakeLists.txt
Doxyfile
Makefile
README.md
RELEASES.md
nfql.1

README.md

NFQL


Network Flow Query Language (NFQL) is our in-house stream-based flow-query language. The query domain specific langauge (DSL) language allows to describe patterns in flow records in a declarative and orthogonal fashion.

Features


  • Filter flows.
  • Combine flows into groups.
  • Aggregate flows on flow-keys as one grouped flow aggregate.
  • Merge grouped flows, supporting temporal relations between groups.
  • Apply absolute or relative filters when grouping or merging.
  • Unfold grouped flows back into individual flows.

nfql


nfql is a reference implementation of NFQL. nfql is composed of an execution engine and a query parser. The execution engine is the brain of nfql where the flows are processed. nfql reads the flow-query in an intermediate JSON format and reads in the trace files in memory for efficient processing. The query parser can be used to read a custom query DSL and generate the JSON intermediate representation required by the execution engine.

Architecture


  • Supports reading and writing NetFlow v5 flows in flow-tools format.
  • The default query DSL can be mathematically expressed as a DNF expression.
  • The query is read in a JSON format allowing M2M communication.
  • The query parser and execution engine are decoupled from one another.
  • The flow processing is modular adhering to a five-stage processing pipeline.
  • Each stage of the processing pipeline can be enabled/disabled at runtime.
  • Compression levels for flow-traces can be configured at runtime.

The JSON representation need not be manually generated by the user. A NFQL parser must read, syntactically check and parse the query DSL to generate an equivalent JSON representation. Currently we are missing such a parser module. For now, we provide custom Python scripts that can generate JSON representations. A new query and its corresponding unit test can be prepared by adapting to one of the respective example/ and tests/ scripts provided in the package.

Installing on Debian-based Linux


Tried on Debian Wheezy x86_64

Install Dependencies

$ sudo apt-get install cmake flow-tools-dev zlib1g-dev libjson0-dev

Install IPFIX protocol library

$ sudo apt-get install libglib2.0-dev  
$ wget http://tools.netsa.cert.org/releases/libfixbuf-1.4.0.tar.gz
$ tar -zxvf libfixbuf-1.4.0.tar.gz
[libfixbuf-1.4.0] $ ./configure
[libfixbuf-1.4.0] $ make
[libfixbuf-1.4.0] $ sudo make install 

Build and install nfql

[nfql] $ make
[nfql] $ sudo make install
$ sudo ldconfig

Read the man page

$ sudo apt-get install most
$ man nfql | most

Generate Documentation (optional)

$ sudo apt-get install doxygen
$ sudo apt-get install graphviz
[nfql] $ make doc

Cleanup

[nfql] $ make clean

Installing on Mac OS X using Homebrew


Tried on Mac OS X 10.9.1

Install Homebrew →

$ ruby -e "$(curl -fsSL https://raw.github.com/Homebrew/homebrew/go/install)"

Install Dependencies

$ brew install cmake
$ brew install flow-tools

NOTE: json-c starting from v0.11 has renamed the library from libjson to libjson-c. Debian-based systems are currently providing v0.10 in the repository. In order to avoid complexity in the makefile, we have decided to freeze the library dependency to v0.10. Install json-c v0.10:

$  brew install https://raw.github.com/mxcl/homebrew/9f78662acff4c3cb1bff3587c8ead7f5935925a9/Library/Formula/json-c.rb

Install IPFIX protocol library

$ brew install libfixbuf
$ brew install gettext
$ brew link --force gettext

Build and install nfql

[nfql] $ make 
[nfql] $ make install

Read the man page

$ brew install most 
$ man nfql | most

Generate Documentation (optional)

$ brew install doxygen
$ brew install graphviz
[nfql] $ make doc

Cleanup

[nfql] $ make clean

Installing on Mac OS X using MacPorts


WARNING: This documentation is obsolete.

Tried on Mac OS X 10.7.

Install MacPorts →

Install Dependencies:

$ sudo port install cmake flow-tools json-c

Build nfql:

[nfql] $ make CMAKE_PREFIX_PATH=/opt/local
[nfql] $ make install

Read the man page:

$ man nfql

Generate Documentation (optional)

$ sudo port install doxygen graphviz
[nfql] $ make doc

Cleanup

[nfql] $ make clean

Installing on Fedora Linux


Tried on Fedora 18.

Install Dependencies

$ sudo yum install cmake flow-tools-devel json-c-devel

Install IPFIX protocol library

$ sudo yum install glib2-devel
$ wget http://tools.netsa.cert.org/releases/libfixbuf-1.4.0.tar.gz
$ tar -zxvf libfixbuf-1.4.0.tar.gz
[libfixbuf-1.4.0] $ ./configure
[libfixbuf-1.4.0] $ make
[libfixbuf-1.4.0] $ sudo make install 

Build and install nfql

[nfql] $ make 
[nfql] $ sudo make install
# echo "/usr/local/lib" >> /etc/ld.so.conf
# ldconfig

Read the man page

$ sudo yum install most
$ man nfql | most

Generate Documentation (optional)

$ sudo yum install doxygen graphviz
[nfql] $ make doc

Cleanup

[nfql] $ make clean

Installing on FreeBSD


WARNING: This documentation is obsolete.

Tried on FreeBSD 9.2

Install Dependencies

$ cd /usr/ports/devel/cmake
[cmake] $ sudo make install 

$ cd /usr/ports/net-mgmt/flow-tools
[flow-tools] $ sudo make install 

$ cd /usr/ports/devel/json-c
[json-c] $ sudo make install 

$ cd /usr/ports/devel/glib20
[glib20] $ sudo make install

$ cd /usr/ports/net/libfixbuf
[libfixbuf] $ sudo make install

$ cd /usr/ports/devel/libexecinfo
[libexecinfo] $ sudo make install

Build and install nfql

[nfql] $ make CMAKE_PREFIX_PATH=/usr/local
[nfql] $ make install

Read the man page

$ cd /usr/ports/sysutils/most
[most] $ sudo make install
$ man nfql | most

Generate Documentation (optional)

$ cd /usr/ports/devel/doxygen
[doxygen] $ sudo make install 

$ cd /usr/ports/graphics/graphviz
[graphviz] $ sudo make install 

[nfql] $ make doc

Cleanup

[nfql] $ make clean

Running nfql


Some example queries are provided in examples/ along with a sample trace.

[nfql] $ nfql --ipfix examples/query-http-tcp-session.json examples/trace-2009.ipfix  
[nfql] $ nfql examples/query-http-tcp-session.json examples/trace-2009.ft

The sample queries can also be run on your own NetFlow v5 records

[nfql] $ flow-cat tracefile[s] | nfql examples/query-http-tcp-session.json

Running the Test Suite


Requirements: Python 2.7+

To run the complete regression test-suite:

[nfql] $ tests/regression.py [-v]

Regression tests can also be run individually on a specific example query type. For instance:

[nfql] $ tests/test-query-http-tcp-session.py [-v]

Running the Benchmarks


Requirements: Python 2.7+

To run the nfql benchmarks:

[nfql] $ make
[nfql] $ sudo benchmarks/nfql.py nfql tracefile[s]/ querie[s]/

Example nfql traces and queries are provided in examples/

To run the SiLK benchmarks:

[nfql] $ sudo benchmarks/silk.py tracefile[s]/ querie[s]/

Example SiLK traces and queries are provided in examples/silk/

Authors