A toolkit for automated log parsing [ICSE'19, TDSC'18, DSN'16]
Branch: master
Clone or download
thomasryck and jimzhu Drain compatibility with Python 3.7 (#18)
* Add notebooks files to .gitignore + 
* Fix Drain compatibility with Python 3.7
Latest commit 85b60bd Feb 8, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
benchmark update May 31, 2018
demo Drain compatibility with Python 3.7 (#18) Feb 9, 2019
docs Update Drain.md Dec 25, 2018
logparser Drain compatibility with Python 3.7 (#18) Feb 9, 2019
logs Add files via upload Dec 25, 2018
test update May 31, 2018
.gitignore Drain compatibility with Python 3.7 (#18) Feb 9, 2019
LICENSE.md Add license May 31, 2018
README.md Update README.md Feb 3, 2019

README.md

Logparser

Documentation Status license

Logparser provides a toolkit and benchmarks for automated log parsing, which is a crucial step towards structured log analytics. By applying logparser, users can automatically learn event templates from unstructured logs and convert raw log messages into a sequence of structured events. In the literature, the process of log parsing is sometimes refered to as message template extraction, log key extraction, or log message clustering.


An illustrative example of log parsing

👉 Read the docs: https://logparser.readthedocs.io

🔭 If you use any of our tools or benchmarks in your research for publication, please kindly cite the following papers.

Log parsers currently available:

Tools References
SLCT [IPOM'03] A Data Clustering Algorithm for Mining Patterns from Event Logs, by Risto Vaarandi.
AEL [QSIC'08] Abstracting Execution Logs to Execution Events for Enterprise Applications, by Zhen Ming Jiang, Ahmed E. Hassan, Parminder Flora, Gilbert Hamann.
[JSME'08] An Automated Approach for Abstracting Execution Logs to Execution Events, by Zhen Ming Jiang, Ahmed E. Hassan, Gilbert Hamann, Parminder Flora.
IPLoM [KDD'09] Clustering Event Logs Using Iterative Partitioning, by Adetokunbo Makanju, A. Nur Zincir-Heywood, Evangelos E. Milios.
[TKDE'12] A Lightweight Algorithm for Message Type Extraction in System Application Logs, by Adetokunbo Makanju, A. Nur Zincir-Heywood, Evangelos E. Milios.
LKE [ICDM'09] Execution Anomaly Detection in Distributed Systems through Unstructured Log Analysis, by Qiang Fu, Jian-Guang Lou, Yi Wang, Jiang Li.
LFA [MSR'10] Abstracting Log Lines to Log Event Types for Mining Software System Logs, by Meiyappan Nagappan, Mladen A. Vouk.
LogSig [CIKM'11] LogSig: Generating System Events from Raw Textual Logs, by Liang Tang, Tao Li, Chang-Shing Perng.
SHISO [SCC'13] Incremental Mining of System Log Format, by Masayoshi Mizutani.
LogCluster [CNSM'15] LogCluster - A Data Clustering and Pattern Mining Algorithm for Event Logs, by Risto Vaarandi, Mauno Pihelgas.
LenMa [CNSM'15] Length Matters: Clustering System Log Messages using Length of Words, by Keiichi Shima.
LogMine [CIKM'16] LogMine: Fast Pattern Recognition for Log Analytics, by Hossein Hamooni, Biplob Debnath, Jianwu Xu, Hui Zhang, Geoff Jiang, Adbullah Mueen.
Spell [ICDM'16] Spell: Streaming Parsing of System Event Logs, by Min Du, Feifei Li.
Drain [ICWS'17] Drain: An Online Log Parsing Approach with Fixed Depth Tree, by Pinjia He, Jieming Zhu, Zibin Zheng, and Michael R. Lyu.
MoLFI [ICPC'18] A Search-based Approach for Accurate Identification of Log Message Formats, by Salma Messaoudi, Annibale Panichella, Domenico Bianculli, Lionel Briand, Raimondas Sasnauskas.

Demo

Please follow the installation steps and demo in the docs to get started.

Benchmarking results

All the log parsers have been evaluated across 16 different logs available in loghub. We report parsing accuracy as the percentage of accurately parsed log messages.

👇 Check the detailed bechmarking result table (click to expand)

Note that accuracy values above 0.9 are marked in bold, and the best accuracy results achieved are marked with *.

Publications about logparser

Acknowledgement

Logparser is implemented based on a number of existing open-source projects:

Feedback

For any questions or feedback, please post to the issue page.