Skip to content
A python library for writing a "super grep" for your log files. Acts as a wrapper around regexes and does aggregations on any named groups found in the regex.
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.


A generic library for gathering stats from log files by running regexes on them. Things you can do:

  • Create and run any number of regexes on any number of files in parallel.
  • Aggregate stats by creating named regex groups in your regexes
  • Grab archived logs (so long as you tell it where your archives live)
  • Grab files from remote boxes
  • Print stats to console
  • Print regex matches to console
  • Search on gzipped files


The easiest manner of installation is to grab the package from the PyPI repository.

pip install log_scraper


Base Usage

For off the cuff usage, you can just create a LogScraper object and tell it what regexes to run and where to look for files. Eg.

from log_scraper.base import LogScraper
import log_scraper.consts as LSC

filepath = '/path/to/file'
filename = 'filename.ext'
scraper = LogScraper(default_filepath={LSC.DEFAULT_PATH : filepath, LSC.DEFAULT_FILENAME : filename})
scraper.add_regex(name='regex1', pattern=r'your_regex_here')

# To get aggregated stats
data = scraper.get_log_data()

# To print all the stats

# To print each file's individual stats

# To view log lines matching the regex

The real power, though, is in creating your own class deriving from LogScraper that presets the paths and the regexes to run so that anyone can then use that anywhere to mine data from a process' logs.




To test successfully, you must set up a virtual environment On Unix, in the root folder for the package, do the following:

python -m virtualenv .
source ./bin/activate
./bin/python develop

Now you can make any changes you want and then run the unit-tests by doing:

./bin/python test
You can’t perform that action at this time.