BLIP

Bid-Request Capture Utility (or “Bid Logging In the Pipe” for backronym lovers)

Description

blip is a Python utility for capturing network traffic from a network device or pcap file, extracting the Protobuf or JSON payloads, and storing them to disk. blip uses pcapy, a fast libpcap interface and dpkt, a fast and simple packet manipulation library.

blip also includes a pretty-printer named blip_showdb, which can read and print blip’s binary output.

Setup Requirements

Basic Setup

cd "$REPOPATH" # Where $REPOPATH is the path to the blip repository.
sudo python3 setup.py install

pcapy (libpcap) requires root privileges (or the appropriate permissions); we recommend requesting sudo permissions from your local system administrator.

Configuration File

blip requires a configuration file to filter through received packets and determine whether they’re valid requests or not. This file is passed via command line arguments.

Example file: config.ini

[EXCHANGES]
/requests/bob = 1
/requests/alice = 2
/requests/eve = 3

Note: The name of the file is not important.

This file defines three valid exchanges identified by path and the numeric ID correspondingly assigned in the binary output format. These are evaluated at runtime and should not be modified while they are being used.

There can be up to 255 entries as the binary output format stores exchange IDs as uint8 values.

External Dependencies

blip requires the libpcap development files to filter packets from a network device or PCAP file. It also requires gcc to compile the interface between Python and libpcap.

Recommended Setup

It is recommended to install the application inside a virtualenv.

virtualenv -p python3 "$DIRNAME" # Where $DIRNAME is the desired virtenv path.
cd "$DIRNAME"
source bin/activate
cp -r "$REPOPATH" "$DIRNAME" # Where $REPOPATH is the path of your repository.
# Cloning the git repository locally instead of copying it is also an option.
cd "$REPONAME" # Where $REPONAME is the name of the directory you copied.
sudo python3 setup.py install

This prepares a clean environment for running blip without interference from any other libraries installed on the system.

To return shell variables to their original state, run deactivate.

pip3 Installation

It is also possible to install the project only via pip3, though this method is deprecated due to the availability of better tooling via setuptools.

pip3 install -r blip/requirements.txt

Examples

Warning: Each example assumes your shell is within the virtualenv’s root directory.

Note: sudo is used for convenience’s sake, as an account with administrative or root permissions would work just as well.

Example 1

sudo bin/blip -c config.ini -d wlp4s0 -o /tmp/output.bin

This command captures packets from wlp4s0 (a wireless device) and writes the extracted data to /tmp/output.bin.

Example 2

sudo bin/blip -c config.ini -d wlp4s0 | pv > /tmp/output.bin

This command captures packets from wlp4s0 (a wireless device), pipes the output through pv (to display a progress bar) and writes the extracted data to /tmp/output.bin.

Example 3

sudo bin/blip -c config.ini -d wlp4s0 -o /tmp/output.bin -f "dst port 80"

This command captures packets with a destination equal to port 80, and writes the extracted data to /tmp/output.bin. Any valid BPF syntax expression should be compatible with the program.

Example 4

sudo bin/blip -c config.ini -d wlp4s0 -o /tmp/output.bin -l 3

This command captures 3 packets from wlp4s0 (a wireless device) and writes the extracted data to /tmp/output.bin. (A limit of zero means unlimited capture, which is the default.)

Example 5

sudo bin/blip -c config1.ini config2.ini -p /tmp/input.pcap -o /tmp/output.bin

This command reads packets from the file /tmp/input.pcap and writes the extracted data to /tmp/output.bin. It also demonstrates it is possible to use more than a single configuration file.

Example 6

sudo bin/blip -c config.ini -p /tmp/input.pcap | blip_showdb -t | head

This command reads packets from the file /tmp/input.pcap and writes them to stdout piping them into blip_showdb stdin, which then prints out a truncated version of the binary output’s human-readable form. In this case the head command ensures only the first few lines of output will be visible, after which program execution terminates normally.

Debugging

To assist with debugging, blip generates log files highlighting its actions. By default, only WARNING-leveled messages or higher are logged. The resulting logs are written to a file named along the lines of /blip.blip_.+\.log/ in your temporary folder.

It is possible to change the debugging level and output file via command line flags. It is not recommended to use stdout as a logging output. A debug level of DEBUG is recommended for best results.

The valid debug levels are listed in blip’s help menu on using an invalid option. Not all of these have a visible effect on blip’s execution.

bin/blip -L stuff