Skip to content
forked from logpai/loghub

A large collection of system log datasets for log analysis research

Notifications You must be signed in to change notification settings

YeYuan1107/loghub

 
 

Repository files navigation

Loghub

Loghub maintains a collection of system logs, which are freely accessible for research purposes. Some of the logs are production data released from previous studies, while some others are collected from real systems in our lab environment. Wherever possible, the logs are NOT sanitized, anonymized or modified in any way. All these logs amount to over 77GB in total.

🔭 If you use the loghub datasets in your research for publication, please kindly cite the following paper.

Logs currently available:

Software System Description Labeled Time Span #Messages Data Size
Distributed systems
HDFS_1 Hadoop distributed file system log ✔️ 38.7 hours 11,175,629 1.47GB
HDFS_2 Hadoop distributed file system log N.A. 71,118,073 16.06GB
Hadoop Hadoop mapreduce job log ✔️ N.A. 394,308 48.61MB
Spark Spark job log N.A. 33,236,604 2.75GB
Zookeeper ZooKeeper service log 26.7 days 74,380 9.95MB
OpenStack OpenStack infrastructure log ✔️ N.A. 207,820 58.61MB
Supercomputers
BGL Blue Gene/L supercomputer log ✔️ 214.7 days 4,747,963 708.76MB
HPC High performance cluster log N.A. 433,489 32.00MB
Thunderbird Thunderbird supercomputer log ✔️ 244 days 211,212,192 29.60GB
Operating systems
Windows Windows event log 226.7 days 114,608,388 26.09GB
Linux Linux system log 263.9 days 25,567 2.25MB
Mac Mac OS log 7.0 days 117,283 16.09MB
Mobile systems
Android Android framework log N.A. 1,555,005 183.37MB
HealthApp Health app log 10.5 days 253,395 22.44MB
Server applications
Apache Apache web server error log 263.9 days 56,481 4.90MB
OpenSSH OpenSSH server log 28.4 days 655,146 70.02MB
Standalone software
Proxifier Proxifier software log N.A. 21,329 2.42MB

Datasets download

We host only a small sample (2k lines) of each dataset on Github. If you are interested in these datasets, please download the raw logs at Zenodo.

🔭 We proudly announce that the loghub datasets have been downloaded 48000+ times by more than 380+ organizations (incomplete list) from both industry and academia.

Publications using these datasets

Additional Logs

We have some links to additional log datasets that are related to security research.

Feedback

For any questions or feedback, please raise an issue here.

License

The log datasets are freely available for research purposes.

About

A large collection of system log datasets for log analysis research

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published