Real-time (Nginx, PHP, syslog, "mysqladmin extended") log file analysis and plotting using Python, mongoDB, Orbited, and flot
Pull request Compare This branch is even with saltycrane:master.
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Failed to load latest commit information.
liveloganalyzer
staticmedia
.gitignore
README.rst
orbited.cfg
pip-requirements.txt
supervisord.conf

README.rst

What it does

  • Stores log file data from multiple remote hosts to MongoDB
    • Creates processes that ssh to a remote host and run "tail --follow=name" on the log file
    • Parses the log file using Regular Expressions
    • Stores the data in MongoDB
  • Queries MongoDB for interesting statistics e.g.:
    • Requests/mintute
    • Cache hit rate
    • Count of HTTP 500 statuses
    • Average upstream response time
    • MySQL questions/second
  • Creates real-time plots in a web browser

Dependencies

  • MongoDB
  • PyMongo
  • Orbited
  • Twisted
  • stompservice
  • flot
  • a few more in pip-requirements.txt

To use it

  • Copy settings_template.py to settings.py and edit it
  • Start mongod
  • Start orbited
  • In first terminal, run "sourceexecutive.py"
  • In second terminal, run "analyzerexecutive.py"

Problems / Limitations

  • Leaves "mysqladmin extended" processes running on remote host
  • The bulk of this is set up for my custom Nginx log file (but it is easy to write your own custom sources, parsers, and analyzers.