Click-Through Rate Prediction: Optimizing Ad Relevance with Machine Learning

by Vivek Praharsha

Overview

This project tackles the challenge of predicting click-through rates (CTR) for online advertisements, a crucial task for maximizing ad relevance and campaign ROI. Leveraging the large-scale Avazu CTR Prediction dataset (Kaggle), I developed and evaluated multiple machine learning models, exploring advanced feature engineering techniques to enhance predictive accuracy. The project demonstrates my proficiency in data preprocessing, feature engineering, model selection, hyperparameter tuning, and performance evaluation, key skills for a data scientist or machine learning engineer.

Table of Contents

• Project Motivation
• Dataset and Features
• Methodology
  • Data Exploration and Visualization
  • Data Preprocessing and Feature Engineering
  • Model Selection and Training
  • Model Evaluation and Selection
• Results and Analysis
• Deployment (Optional)
• Conclusions and Future Work
• How to Run
• References

Project Motivation

Target Audience: Online Advertising platforms and related stakeholders

Driven by a strong interest in applying machine learning to real-world business challenges, I chose to focus on CTR prediction due to its significance in online advertising. This project allowed me to:

• Develop Expertise: Gain hands-on experience with key machine learning techniques for classification, a common task in various domains.
• Solve a Practical Problem: Address a challenge faced by businesses across industries, demonstrating my ability to deliver practical solutions.
• Showcase Analytical Skills: Demonstrate my ability to analyze large datasets, extract meaningful insights, and communicate findings effectively.

Dataset and Features

• Dataset Source: Link to the Avazu CTR Prediction dataset on Kaggle
• Description: The Avazu dataset contains over 40 million records of ad impressions, each with 24 features encompassing user demographics, ad attributes, context information, and the target variable (click or no click).
• Key Features:
  • User Features: User agent, device type, operating system, geographic location.
  • Ad Features: Ad ID, advertiser ID, campaign ID, ad category.
  • Context Features: Time of day, day of week, site ID, app ID.

Methodology

Data Exploration and Visualization

• Initial Analysis: Explored data distributions, checked for missing values, identified unique values, and visualized key features to understand patterns and relationships.
• Key insights gained:
  • Ad impressions were high during the business hours contrary to the expectation that the impressions would be higher during the non-working hours
  • Ad impressions were higher on weekdays but CTR was actually high on weekends

Data Preprocessing and Feature Engineering

• Categorical Feature Encoding: Applied in-built hash encoder of polars framework to transform categorical features into numerical representations suitable for machine learning models.
• Feature Scaling: Standardized numerical features using StandardScaler to ensure consistency in feature magnitudes while combining hashed features with other continuous numerical features and aiding algorithms sensitive to feature scales.
• Feature Engineering:
  • Engineered new features, such as hr_time & weekday from the bucketized timestamp, for simplification & better understanding while doing EDA and to aid the models to understand the impact of the day and hour.
• Data Sampling: Made use of fixed as well as sophisticated targeted criteria based under-sampling techniques and leveraged GPU power to speed up the process through the help of RAPIDS cuML library, the techniques used were:
  • TomekLinks to mitigate the problem of mis-classification by removing overlapping majority class samples at the decision boundary
  • Condensed Nearest Neighbors to remove redundant samples

Model Selection and Training

• Approach : Made use of Sophisticated Undersampling techniques (both fixed and targeted under-sampling) to tackle high class imbalance.
• Baseline Model: Established a baseline performance using Logistic Regression, a simple yet interpretable model.
• Advanced Models: Explored more sophisticated models, including:
  • Tree-based Models: Gradient Boosting (XGBoost) to model complex relationships and higher-order interactions through boosting, making it powerful for capturing intricate patterns in data. While it is also excellent at capturing pairwise interactions.
• Training: Split the data into training (70%) and validation (30%) sets. Trained each model using the training set and tuned hyperparameters using the validation set to optimize performance.

Model Evaluation and Selection

• Evaluation Metrics: Selected appropriate metrics for CTR prediction:
  • AUC (Area Under the ROC Curve): Provides an overall measure of model performance, especially important for imbalanced datasets.
  • Log Loss: Penalizes incorrect predictions more heavily, aligning with the goal of minimizing misclassifications.
  • Precision, Recall, F1-Score: Precision and Recall would indicate the amount of true positives and the amount of true negatives the model could predict.
• Model Comparison: Compared model performance on the held-out test set based on the chosen metrics, selecting the best model based on a balance of accuracy and interpretability.

Results and Analysis

• Model Performance:
  • The XGBoost model tuned with Optuna achieved the highest AUC of 83.04% on the test set, outperforming the baseline Logistic Regression model by 10.12%.
  • The log-loss achieved was 0.506612
• Feature Importance:

![[Permutation_Feature_Importance.png]]

• Feature Interactions: We can see that 27 % of C16's impact on CTR Prediction comes from interacting with other features whereas most of the features have less than 13% of effect through interaction.

![[Interaction_H-Test.png]]

Model	Model Type	Accuracy (Baseline)	Accuracy (Tuned)	AUC (Baseline)	AUC (Tuned)	95% CI	Lower	Upper
Logistic Regression	Linear	65.79 %	65.808 %	72.95 %	72.92 %	0.0082	0.7211	0.7375
SVM	Non-Linear	72.285%	76.568 %	79.877 %	82.417 %	0.0070	0.8172	0.8312
XGBoost	Ensemble (Boosting)	75.48 %	75.12 %	83.63 %	83.04 %	0.0069	0.8235	0.8373

Conclusions and Future Work

This project successfully developed a robust CTR prediction model, achieving 83.04 % AUC score which achieved a 10.12 % boost over the baseline and thus providing valuable insights into factors influencing ad click-through rates.

While building this model the focus was on balancing out both precision and recall values to curb costs incurred in CPC (Cost-Per-Click) or PPC (Pay-Per-Click) business models by prioritizing to reduce false positives as well as reducing false negatives in the case where failing to predict a click (a false negative) leads to missed revenue opportunities.

Research work and studies show that user interest has an important influence on CTR prediction. User behavior over multiple sessions makes up a sequence, and it is known for the user interest to change in every session and capturing the user’s dynamics and evolving interests from the user’s sequential behavior has been the robust way to predict CTR. The deviance residual plot obtained after running logistic regression reveals that this dataset has non-linear or other patterns that can be best handled by only non-linear models, additionally Friedman's H-Statistic also shows the presence of multiple feature interactions. However because this project is limited to machine learning techniques and thus focusses only on modelling higher order feature interactions at most and neglecting user interests.

Future work could involve:

• Exploring More Advanced Techniques: Experimenting with deep learning architectures like MLP, Time-Series model, ensemble methods, or incorporating user behavior sequences.
• Real-Time CTR Prediction: Developing a system for real-time CTR prediction to enable dynamic ad serving and optimization.
• A/B Testing: Deploying the model in a live setting to A/B test its effectiveness against existing CTR prediction methods.

References

• Faster Resampling with Imbalanced-learn and cuML
• CLICK-THROUGH RATE PREDICTION: A COMPARATIVE STUDY OF ENSEMBLE TECHNIQUES IN REAL-TIME BIDDING
• Session interest model for CTR prediction based on self-attention mechanism