Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with HTTPS or Subversion.

Download ZIP
Real-time (Nginx, PHP, syslog, "mysqladmin extended") log file analysis and plotting using Python, mongoDB, Orbited, and flot
branch: master

This branch is even with saltycrane:master

Fetching latest commit…

Cannot retrieve the latest commit at this time

Failed to load latest commit information.
liveloganalyzer
staticmedia
.gitignore
README.rst
orbited.cfg
pip-requirements.txt
supervisord.conf

README.rst

What it does

  • Stores log file data from multiple remote hosts to MongoDB
    • Creates processes that ssh to a remote host and run "tail --follow=name" on the log file
    • Parses the log file using Regular Expressions
    • Stores the data in MongoDB
  • Queries MongoDB for interesting statistics e.g.:
    • Requests/mintute
    • Cache hit rate
    • Count of HTTP 500 statuses
    • Average upstream response time
    • MySQL questions/second
  • Creates real-time plots in a web browser

Dependencies

  • MongoDB
  • PyMongo
  • Orbited
  • Twisted
  • stompservice
  • flot
  • a few more in pip-requirements.txt

To use it

  • Copy settings_template.py to settings.py and edit it
  • Start mongod
  • Start orbited
  • In first terminal, run "sourceexecutive.py"
  • In second terminal, run "analyzerexecutive.py"

Problems / Limitations

  • Leaves "mysqladmin extended" processes running on remote host
  • The bulk of this is set up for my custom Nginx log file (but it is easy to write your own custom sources, parsers, and analyzers.
Something went wrong with that request. Please try again.