url

URL Analysis

URL Filteration Machine learing MODEL

Detection of Malicious URLs by RF Model Classifier.

Agenda !

• Problem Statment.
• Data Set.
• Importing libraries.
• Feature Engineering..
• Split Data.
• Classifier Model.
  • Hyperparameters tunning.
  • Model training & prediction.
  • Ploting Results (features importance, Accuracy , confusion matrix , ROC curve).
• References.

Problem Statement

In this case study, we address the detection of malicious URLs as a multi-class classification problem. Classify the raw URLs into different class types such as benign or safe URL, defacement URL, phishing URL or malware URL.

As we know machine learning algorithms only support numeric inputs so we will create lexical numeric features from input URLs. So the input to machine learning algorithms will be the numeric lexical features rather than actual raw URLs. If you don't know about lexical features you can refer to this discussion about a lexical feature in StackOverflow.

So, in this case study, we will be using well-known boosting machine learning classifiers namely Random Forest/Logistic Regression. That decision came up after testing many other known boosting machine learning classifiers like Light GBM, Gradient Boosting Machines. Random Forest/Logistic Regression was the most sufficient and accurate models with best performance parameters.

Data Set

As known one of the most crucial tasks is to curate the dataset for a machine learning project. We have searched and looked for a clear and defined dataset curated different sources for more details check the references.

For training and testing machine learning algorithms, we have used a huge dataset of 651,191 URLs.

.

Out of which 428103 benign or safe URLs, 96457 defacement URLs, 94111 phishing URLs, and 32520 malware URLs.

Depicts their distribution in terms of percentage.

Importing libraries

• pandas
• itertools
• sklearn
  • metrics: classification_report, mean_squared_error, confusion_matrix, precision_score, recall_score, auc,roc_curve
  • model_selection: train_test_split
• numpy
• random
• math
• matplotlib
• Random Forest/Logistic Regression
• os
• socket
• tld
• urllib
• GridSearchCV
• seaborn

Feature Engineering

lexical features: whole word, prefix/suffix (various lengths possible), stemmed word, lemmatized word

• Export Dataset with Features (optional)

Split Data

RF Classifier Model

Model parameters

Model training.

Results ploting

features importance, Accuracy , confusion matrix

References

• For collecting benign, phishing, malware and defacement URLs we have used URL dataset (ISCX-URL-2016) and here is the full research paper outlining the details of the dataset and its underlying principles.

• For increasing phishing and malware URLs, we have used Malware domain black list dataset. We have increased benign URLs using faizan git repo At last, we have increased more number of phishing URLs using Phishtank dataset and PhishStorm dataset As we have told you that dataset is collected from different sources.

• So firstly, we have collected the URLs from different sources into separate dataframe and finally merge them to retain only URLs and their class type.

• The final pre-processed dataset is available at kaggle