Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP
A Cassandra demo application, log management
Python
Tree: bcce50dc5f

The graph now displays all data

The graph is updated as more and more data is returned from a query
which contains alot of data.
latest commit bcce50dc5f
@jbohman authored
Failed to load latest commit information.
docs Created a pylon project and merged the code into it.
logsandra.egg-info Updated README, still more work to be done
logsandra
scripts
.gitignore
MANIFEST.in Created a pylon project and merged the code into it.
README Forgot to use aspell...
development.ini Created a pylon project and merged the code into it.
ez_setup.py Created a pylon project and merged the code into it.
logsandra-httpd.py Refactored the Cassandra Python model code
logsandra-monitord.py Refactored the Cassandra Python model code
logsandra.yaml Added automatic ident discovery
setup.cfg Created a pylon project and merged the code into it.
setup.py
test.ini Created a pylon project and merged the code into it.

README

Logsandra
=========
Logsandra is a log management application written in Python and using Cassandra as back-end.


Requirements
------------
Cassandra 0.6.x, Python 2.6/2.7 (not tested with Python 3.x) and the following Python modules:

    * Pylons >= 1.0
    * Jinja2
    * PyYAML
    * Pycassa
    * python-dateutil
    * CherryPy >= 3.1
    * Thrift
    * ordereddict
    * Cassandra (not automatically installed during the installation phase)


Installation
------------
To install Logsandra you run this command:

    python setup.py install

All Python modules will be automatically installed by that command except for the Cassandra bindings.


Configuration
-------------
To use Logsandra, you need to configure your Cassandra cluster. Below is the current Keyspace configuration needed by Logsandra:

    <Keyspaces>
        <Keyspace Name="logsandra">
            <ColumnFamily Name="entries" CompareWith="BytesType" />
            <ColumnFamily Name="by_date" CompareWith="LongType" />
            <ColumnFamily Name="by_date_data" CompareWith="LongType" />
            <ColumnFamily Name="categories" CompareWith="UTF8Type" />
          <ReplicaPlacementStrategy>org.apache.cassandra.locator.RackUnawareStrategy</ReplicaPlacementStrategy>
          <ReplicationFactor>3</ReplicationFactor>
          <EndPointSnitch>org.apache.cassandra.locator.EndPointSnitch</EndPointSnitch>
        </Keyspace>
    </Keyspaces>

(In Cassandra 0.7 and higher you will not need to change the Keyspace configuration manually, but as for now Cassandra 0.7 is in development and not supported by Logsandra)


The main configuration file for Logsandra is logsandra.yaml, the format is yaml. Below is a list of configuration variables (including type and optional/required):

    ident               identity of the logsandra server [String, Optional/Discovery]

    httpd_host          host on which to serve the logsandra web service [String, Required]
    httpd_port          port on which to serve the logsandra web service [String, Required]
    httpd_config        name of the pylons configuration file [String]

    cassandra_host      cassandra host to connect too [String, Required]
    cassandra_port      cassandra port to connect too [String, Required]
    cassandra_timeout   in seconds until connection is aborted if no response is delivered [String, Optional]

    paths               list of paths to monitor [List, Optional]
      path              directory or file [String, Required]
      recursive         recursively enter sub directories [True/False, Optional]
      parser            dictionary containing parser name and arguments [Dictionary, Required]
        name            name of the parser [String, Required]
        ...             arguments, see specific parser [Depends, Optional]

Logsandra is built on Pylons and therefore a Pylons configuration file should be present for Logsandras web service to work.


Usage
-----
Logsandra consists of two main scripts, which both acts as a daemon, below is the usage pattern:

    python logsandra-httpd.py [options] start|stop|restart
    python logsandra-monitord.py [options] start|stop|restart

If you want a list of options invoke either one these commands:

    python logsandra-httpd.py --help
    python logsandra-monitord.py --help

An example setup is to run logsandra-httpd on one server and logsandra-monitord on all servers producing log files. Logsandra-monitord will send the data to the specified Cassandra Cluster. 


Create your own parser
----------------------
To create your own Logsandra parser you need to create a python file in this directory:

    logsandra/monitor/parsers/

The name of the file should be in lower case and end with '.py', an example of a files content (the filename is foo.py):

    from logsandra.monitor.parsers import BaseParser

    class FooParser(BaseParser):
        def parse(self, line, source, data):
            date = None
            keywords = []
            return self.log_entries.add(date=date, entry=line, source=source, keywords=keywords)

A better example is to study how existing parsers work in logsandra/monitor/parsers/ directory. 
Something went wrong with that request. Please try again.