Skip to content
Logolas is a light-weight multi-file log scraper with an accompanying real-time web interface.
Python JavaScript
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
images
logolas
sample
.gitignore
.pylintrc
LICENSE.txt
MANIFEST.in
README.rst
setup.py

README.rst

Logolas

Logolas is a light-weight multi-file log scraper with an accompanying real-time web interface.

Screenshot

images/screenshot.png

Design

  • Logolas utilizes inotify to watch files for updates. This means it will not work on Windows or over NFS.
  • Lines are scraped using regular expressions. At a minimum, each line must include a date/time.
  • The scraped lines are persisted to a SQL backend. This addresses loss of historical data due to log rotation. However, currently purging is not automatic.
  • Web interface performs standard polling at a configurable rate.

Features

  • Multi-file logging
  • Multi-regular expression scraping per-file
  • Real-time streaming of logged events
  • Filtering on any matched field
  • Access to historical data (admittedly a bit awkward)

Limitations

  • Cannot parse timezones currently.

Why?

An administrator of my Minecraft server needed a web interface that provided real-time access to information stored in multiple log files on my server.

This started as a toy project, but I've decided to publish it as I can imagine there is someone out there who would benefit from this work.

Installation

The following instructions assumes a debian based distribution.

Using virtual environments:

git clone https://github.com/bheiskell/logolas.git
cd logolas
sudo aptitude install python-virtualenv
virtualenv --no-site-packages venv
. venv/bin/activate
easy_install .
cp sample/config.yml .

To start the log scraper:

cd logolas
. venv/bin/activate
logolas config.yml

Alternatively, deploy to the root python installation:

git clone https://github.com/bheiskell/logolas.git
cd logolas
sudo easy_install .
cp sample/config.yml .
logolas config.yml

Web

To launch the web interface, run:

cd logolas
. venv/bin/activate
export LOGOLAS_CONFIG=config.py
logolas_web

If you plan on using SQLite, make sure you provide a fully qualified path to the database. The Flask framework changes the current working directory which will cause relative paths to break.

TODO: wsgi configuration

MySQL

If you choose to use MySQL with virtual environments, you'll need to install libmysqlclient-dev and python-dev with aptitude. Then you can easy_install mysql-python.

Alternatively, you should be able to simply install to python-mysqldb if you're not using virtual environments.

You can’t perform that action at this time.