Author: Loan Lassalle
To protect data from cyber-attacks from multiple sources such as businesses or individuals, anomaly detection is an effective and efficient way to deal with them. The machine learning is a structured and optimal method to quickly examine a large volume of data. Accurate data analysis improves responsiveness and quality of response to project needs and objectives.
Data mining makes it possible to understand what the data hides in order to give them all their meaning and to understand their values. In IT, this analytical power can be applied to detect intrusion into networks or fraud in the banking field.
The purpose of this Bachelor's work is to perform data processing to an anomaly detection system with Spark, through the implementation of an application. Different data sets captured and pre-labelled will be used. Data processing will also use Spark libraries for the machine learning. He has a proven track record in the IT field. Its assets have won it over in various fields and are adopted by the greatest number.
For demonstration purposes, an anomaly detection application has been implemented. It aims to detect abnormal HTTP requests inducing attacks. The application is similar to a Web Application Firewall. The k-means clustering algorithm is at the heart of the detection system. The detection of anomalies is a vast subject rooted in the field of computing. But it has to be said that its usefulness is undeniable in several and varied fields.
We use datasets from the HTTP DATASET CSIC 2010. You can find the complete description of the task here.
To be able to carry out the realization of this application, we implemented beforehand an algorithm of detection of anomalies. These essays were based on the chapter Anomaly Detection in Network Traffic with 𝑘-means clustering in Advanced Analytics with Spark Patterns for Learning from Data at Scale. They allowed the concepts and means of designing an anomaly detection algorithm to be assimilated. This training helped to understand how to optimally parameterize the clustering algorithm 𝑘-means. This test phase was interesting to initiate the application.
We use datasets from the KDD Cup 1999. You can find the complete description of the task here.
Some code tips were inspired by the following references
- Parsing raw HTTP Request by Igor Zelaya
- Machine Learning Library (MLlib) Guide
The reference book for these and other Spark related topics is:
- Advanced Analytics with Spark by Sandy Ryza, Uri Laserson, Sean Owen & Josh Wills
- Learning Spark by Holden Karau, Andy Konwinski, Patrick Wendell & Matei Zaharia
- Spark: The Definitive Guide by Bill Chambers and Matei Zaharia