A collection of system log datasets for intelligent log analysis
Branch: master
Clone or download
Latest commit f93c961 Jan 29, 2019
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
Andriod Update readme Nov 21, 2018
Apache Update readme Nov 21, 2018
BGL Update readme Nov 27, 2018
HDFS Update readme Nov 25, 2018
HPC Update readme Nov 25, 2018
Hadoop Update readme Nov 27, 2018
HealthApp Update readme Nov 25, 2018
Linux Update readme Nov 25, 2018
Mac Update readme Nov 25, 2018
OpenSSH Update readme Nov 25, 2018
OpenStack Update readme Nov 25, 2018
Proxifier Update readme Nov 27, 2018
Spark Update readme Nov 25, 2018
Thunderbird Update readme Nov 25, 2018
Windows Update readme Nov 25, 2018
Zookeeper Update readme Nov 25, 2018
.gitignore Add logs Jan 10, 2018
README.md Update README.md Jan 29, 2019

README.md

Loghub

Loghub maintains a collection of system logs, which are freely accessible for research purposes. Some of the logs are production data released from previous studies, while some others are collected from real systems in our lab environment. Wherever possible, the logs are NOT sanitized, anonymized or modified in any way. All these logs amount to over 77GB in total. We thus host only a small sample (2k lines) on Github for each dataset.

🔭 If you use the loghub datasets in your research for publication, please kindly cite the following paper.

How to get the data?

If you are interested in these datasets, please request the raw logs at Zenodo. Kindly note that the affiliation information is minimally required for your data request.

Logs currently available:

Software System Description Time Span #Messages Data Size
Distributed systems
HDFS Hadoop distributed file system log (labeled) 38.7 hours 11,175,629 1.47GB
Hadoop distributed file system log N.A. 71,118,073 16.06GB
Hadoop Hadoop mapreduce job log (labeled) N.A. 394,308 48.61MB
Spark Spark job log N.A. 33,236,604 2.75GB
Zookeeper ZooKeeper service log 26.7 days 74,380 9.95MB
OpenStack OpenStack infrastructure log (labeled) N.A. 207,820 58.61MB
Supercomputers
BGL Blue Gene/L supercomputer log (labeled) 214.7 days 4,747,963 708.76MB
HPC High performance cluster log N.A. 433,489 32.00MB
Thunderbird Thunderbird supercomputer log (labeled) 244 days 211,212,192 29.60GB
Operating systems
Windows Windows event log 226.7 days 114,608,388 26.09GB
Linux Linux system log 263.9 days 25,567 2.25MB
Mac Mac OS log 7.0 days 117,283 16.09MB
Mobile systems
Andriod Andriod framework log N.A. 1,555,005 183.37MB
HealthApp Health app log 10.5 days 253,395 22.44MB
Server applications
Apache Apache web server error log 263.9 days 56,481 4.90MB
OpenSSH OpenSSH server log 28.4 days 655,146 70.02MB
Standalone software
Proxifier Proxifier software log N.A. 21,329 2.42MB

Publications using these datasets

Organizations that download these datasets

🔭 We proudly announce that the loghub datasets have been downloaded 1300+ times by more than 170 organizations from both industry and academia.

Additional Logs

We have some links to additional log datasets that are related to security research.

Feedback

For any questions or feedback, please post to our issue page.

License

The log datasets are freely available for research purposes.