Django application to store log messages on Cassandra
Python JavaScript Shell
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
dev-scripts
src-client
src/daedalus
.gitignore
.project
.pydevproject
LICENSE.txt
MANIFEST
README.md
TODO.md
daedalus_version.py
pydev_manage.py Fixed references in imports, scripts, etc. Tests passed ok. Aug 3, 2012
requirements-dev.txt
requirements.txt
setup.py

README.md

Daedalus

Daedalus is client/server application. The server (what receives the messages) is implemented using Django. The log messages are stored in Cassandra. The clients send the messages using a POST messages, allowing to send messages from virtually any language.

Please, report any issue here.

Server

  • Backend: what receives the log messages

  • Frontend: the webapp to list the messages (see screenshots). You can:

    • Filter by application
    • Filter by host
    • Filter by severity
    • Show all messages (default for home page)
    • Simplest form of pagination
    • Show line chart counting messages received

Implemented clients

Log messages

A log messages includes:

  • message: any string with a single or multi-line text
  • application: the application that generated the meessage
  • host: the host that generated the message
  • severity: the serverity of the message: one of 'DEBUG', 'INFO', 'WARN' or 'ERROR'
  • timestamp: seconds from EPOCH (in UTC) associated with the message.

How to manually send a message from Python:

    import httplib
    import urllib
    params = urllib.urlencode({
        'application': u'intranet',
        'host': u'appserver3.example.com',
        'severity': u"WARN",
        'message': u"The account 'johndoe' was locked after three login attempts.",
        'timestamp': u"1343607762.837293",
    })
    headers = {"Content-type": "application/x-www-form-urlencoded"}
    conn = httplib.HTTPConnection("localhost", "64364")
    conn.request("POST", "/backend/save/", params, headers)
    response = conn.getresponse()
    print response.read()
    conn.close()

Sending the same message using the Python client:

    msg = u"The account 'johndoe' was locked after three login attempts."
    daedalus_client = DaedalusClient("localhost", 64364)
    daedalus_client.send_message(msg, "ERROR", "appserver3.example.com", "intranet")

How to install from PYPI using virtualenv

To install and run Daedalus server using pip, (assuming a virtualenv directory exists and Cassandra running on localhost) use:

$ ./virtualenv/bin/pip install daedalus
$ export DJANGO_SETTINGS_MODULE=daedalus.settings
$ ./virtualenv/bin/django-admin.py runserver

To install only the Python client (and logging handler), run:

$ pip install daedalus-python-client

For the absolutely newby: how to install Cassandra + Daedalus in Ubuntu

See dev-scripts/install-on-ubuntu.sh.

I recommend you to download and run the script in a newly created virtual machine (mainly because it install many packages, and must be run as root). The virtual machine should have at least 1GB of RAM (Cassandra may not work with less memory). The scripts installs JDK, Cassandra, clones the Daedalus repository and launch the Django development server.

You can download this script and run it as root, or use it as a guide, copying-and-pasting each of the commans of the script in a console or ssh session.

For developers: how to download and hack

JDK: download and install JDK 6 (needed for Cassandra).

Cassandra: download, install and start Cassandra.

Download from GitHub

$ git clone http://github.com/hgdeoro/daedalus
$ cd daedalus

Create virtualenv and install requeriments

$ virtualenv virtualenv
$ ./virtualenv/bin/pip install -r requirements.txt
$ ./virtualenv/bin/pip install -r requirements-dev.txt

Run syncdb and syncdb_cassandra

$ ./dev-scripts/manage.sh syncdb
$ ./dev-scripts/manage.sh syncdb_cassandra

Start memchaed

$ sudo service memcached start

Start development server

$ ./dev-scripts/runserver.sh

By now you'll have the Django development server running. Both the Daedalus backend (the Django app that receives the logs via HTTP) and the frontend (the application used to see the logs) are started. To use it, go to http://127.0.0.1:64364/.

To create some random log messages, you could run:

$ ./dev-scripts/bulk_save_random_messages.sh

(and press Ctrl+C twice to stop it).

The project could be imported from within Eclipse PyDev.

Not implemented right now / Ideas / TODOs

See TODOs.

Cassandra

  • A single keyspace holds all the column families.

  • Messages are stored using 4 column families:

    • CF: Logs - Cols[]: { uuid1/timestamp: JSON encodded message }
    • CF: Logs_by_app - Cols[]: { uuid1/timestamp: JSON encodded message }
    • CF: Logs_by_host - Cols[]: { uuid1/timestamp: JSON encodded message }
    • CF: Logs_by_severity - Cols[]: { uuid1/timestamp: JSON encodded message }
  • Alternative format (implemented by StorageServiceUniqueMessagePlusReferences):

    • CF: Logs - Cols[]: { uuid1/timestamp: JSON encodded message }
    • CF: Logs_by_app - Cols[]: { uuid1/timestamp: '' }
    • CF: Logs_by_host - Cols[]: { uuid1/timestamp: '' }
    • CF: Logs_by_severity - Cols[]: { uuid1/timestamp: '' }
  • No SuperColumn nor secondary indexes by now.

Changelog

v0.0.10

  • Upgrade to pycassa 1.7
  • Backend: new storage: StorageServiceRowPerMinute
  • Created multi-messages log entryies to group related messages

v0.0.9

  • Frontend: added 'host' column to list of messages
  • Client: implemented command line arguments for setting the severity
  • Client: implemented tests for CLI
  • General: automatized steps needed for creating a 'release'

v0.0.8

  • General: refactored Python package (from hgdeoro.daedalus.web.frontend/backend to daedalus.frontend/backend)
  • General: changed default port from 8084 to 64364
  • Frontend: implemented reset of memcache

v0.0.7

  • Client: created a command line to send log events
  • Client: created a handler for the Python logging framework
  • General: fixed various issues around setup.py and created setup.py for the Python client.
  • General: now the Daedalus server and Python client are uploaded to PYPI

License

#-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
#    daedalus - Centralized log server
#    Copyright (C) 2012 - Horacio Guillermo de Oro <hgdeoro@gmail.com>
#
#    This file is part of daedalus.
#
#    daedalus is free software; you can redistribute it and/or modify
#    it under the terms of the GNU General Public License as published by
#    the Free Software Foundation version 2.
#
#    daedalus is distributed in the hope that it will be useful,
#    but WITHOUT ANY WARRANTY; without even the implied warranty of
#    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
#    GNU General Public License version 2 for more details.
#
#    You should have received a copy of the GNU General Public License
#    along with daedalus; see the file LICENSE.txt.
#-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-