Display original log with results?

[jmlane8]

Do you have anything that displays the original log records, their ground truth status as normal and abnormal, and the result from logdeep predictions?

[cherishwsx]

I can help with this one. :)
I think you can find the original log data in loghub. You can find the HDFS (specifically HDFS_1) data and BGL data along with their labels in the corresponding folders. For the logdeep predictions, are you saying the evaluation result? The result is shown in the Benchmark results section in README.md.

[cherishwsx]

Forgot to put the link. loghub

[jmlane8]

Thank you. I wanted to run the abnormal and normal predictions, and be able to point back to the original unstructured log records, and say: the neural network picked up something abnormal here.

[cherishwsx]

Thank you. I wanted to run the abnormal and normal predictions, and be able to point back to the original unstructured log records, and say: the neural network picked up something abnormal here.

I think you can actually print out the block_id (which is the event sequence identifer in HDFS dataset) or row number when there is a abnormal record detected. Looking at the inference part script predict.py might help.

[cherishwsx]

When I was thinking about the "tracking back to raw log records" problem, it seems to me like there is no way to actually track record by record (more of a streaming analysis) since we are training and predicting on event sequence, instead of every log records/single event. So I guess we can only know which event sequence is abnormal, right? And it's more suitable for batch log analysis?

Correct me if I'm wrong and any ideas are welcome! @donglee-afar

[d0ng1ee]

You are right, @cherishwsx
I think if you understand the pipeline of log anomaly detection, this is a very simple job ...