Detecting Credit Card Fraud with Interpretable Machine Learning - A Cloud Computing Project for Spring 2025 Semester
Trinity Chamblin huz2ph@virginia.edu , Morgan Simmons rxk8gn@virginia.edu, and Amanda Appiah-Yeboah vhz8xf@virginia.edu
Credit card fraud is a persistent problem for financial institutions, and increasing transaction volume has made rule-based detection less effective. Machine learning offers a more adaptive alternative by uncovering patterns in transaction data that may indicate fraud.
This project evaluates whether machine learning models can identify fraudulent transactions in a highly imbalanced dataset, where fraud represents less than 0.2% of all observations. Because traditional accuracy is uninformative in this setting, the focus is on metrics that capture minority-class performance.
The first of our three objectives is to identify patterns that distinguish fraud from legitimate activity. Secondly, we aim to compare several machine learning models under class imbalance. Lastly, we will assess model interpretability, which is essential for financial institutions that must justify automated decisions. Through this comparison, we examine the tradeoffs between predictive performance and transparency in fraud detection systems.
The dataset used in this project is the Credit Card Fraud Detection dataset https://www.kaggle.com/datasets/mlg-ulb/creditcardfraud/data , publicly available on Kaggle and originally released by researchers at the Université Libre de Bruxelles (ULB). It is widely recognized as one of the standard benchmark datasets for evaluating fraud detection algorithms, due to both its real-world origin and its highly imbalanced class distribution. The data represent credit card transactions made by European cardholders over a two-day period in September 2013. The dataset contains a total of 284,807 credit card transactions, each stored as a row in a single CSV file. To protect sensitive financial information, the dataset has undergone extensive preprocessing and anonymization. The original transaction features were transformed using Principal Component Analysis (PCA), resulting in 28 anonymized numerical variables labeled V1 through V28. Only two original features, Time (seconds elapsed between the transaction and the first transaction in the dataset) and Amount (transaction amount in Euros) remain untransformed. Because of the PCA transformation, the identities and meanings of the V-features are not directly interpretable, which motivates the use of model-agnostic interpretability tools in our analysis.
A defining characteristic of this dataset is its extreme class imbalance. Only 492 transactions are labeled as fraudulent, representing approximately 0.17% of all observations. This imbalance presents a meaningful challenge for classification algorithms, as models can achieve high accuracy by predicting every transaction as legitimate while still failing to detect fraud. The response variable Class indicates whether a transaction is 0, a legitimate transaction, or 1, a fraudulent transaction.
This makes the task a binary classification problem, where the goal is to distinguish the rare fraudulent cases from the overwhelming majority of legitimate transactions. The imbalance in this response variable makes traditional accuracy metrics insufficient on their own, requiring additional evaluation tools such as precision, recall, F1-score, and AUC-based measures.
We prepared the dataset through a series of preprocessing steps designed to address its extreme class imbalance and ensure fair comparison across models. Because fewer than 0.2% of transactions are fraudulent, we tested several resampling methods, including undersampling, oversampling, and SMOTE. SMOTE provided the most stable minority-class representation and was used for model training. We then created a stratified 70/30 train-test split to preserve class proportions in both sets. Feature scaling was applied to Time and Amount using StandardScaler, since models such as Logistic Regression, KNN, and SVM are sensitive to feature magnitude. The PCA-transformed variables (V1-V28) were already standardized and required no additional processing.
Exploratory data analysis focused on understanding overall class imbalance, differences in transaction amounts, and basic temporal patterns. Because most features are PCA components, EDA emphasized distributional comparisons and correlation structure rather than domain-specific interpretation.
We evaluated a range of machine learning models that represent both interpretable linear methods and more flexible nonlinear techniques. Logistic Regression served as a baseline and offered coefficient-based interpretability. Decision Trees provided a simple nonlinear model but required tuning of depth and split thresholds to reduce overfitting. Random Forest and Gradient Boosting models were included for their ability to capture complex interactions and typically strong performance on tabular data. KNN and SVM (with linear and RBF kernels) were used to compare distance-based and margin-based methods. To ensure consistent evaluation, all models were trained using the same resampled training data and were assessed with stratified cross-validation. Hyperparameter tuning was performed with GridSearchCV for simpler models and RandomizedSearchCV for higher-dimensional ones. Finally, we applied model-agnostic interpretability tools, including feature importance, SHAP values, partial dependence plots, and LIME, to examine which features most strongly influenced predictions and to compare decision patterns across models.
Evaluating fraud detection models requires metrics that meaningfully reflect the severe class imbalance in the dataset. Traditional accuracy is not informative in this setting, since a model can achieve over 99% accuracy simply by predicting every transaction as legitimate. Instead, we focus on measures that highlight performance on the minority class. Precision and recall form the core of this evaluation. Precision captures how often a transaction flagged as fraudulent is actually fraud, which matters because false positives can introduce customer friction and operational costs. Recall measures the proportion of true fraud cases the model successfully identifies, and is especially critical because missed fraud directly translates to financial loss. The F1 score provides a harmonic balance between these two metrics, offering a single measure that penalizes models performing well on only one dimension. To assess overall discriminative ability across classification thresholds, we report both ROC-AUC and PR-AUC. ROC-AUC evaluates how well the model separates fraudulent from non-fraudulent transactions in a threshold-agnostic manner, while PR-AUC is more sensitive to class imbalance and better reflects the precision-recall tradeoff that defines practical fraud detection. Confusion matrices complement these metrics by providing a clear view of false positives and false negatives, allowing us to interpret model behavior in a direct, operationally meaningful way.
The models that relied on nonlinear decision boundaries and ensemble learning tended to perform best under the constraints imposed by the minority class. Logistic Regression served as a strong baseline, offering stable precision but limited recall. Decision Trees captured nonlinear patterns but were prone to overfitting without careful tuning. Random Forests and Gradient Boosting consistently yielded the strongest results. Their ability to aggregate many weak learners allowed them to capture subtle patterns in the PCA-transformed features and handle interactions that simpler models missed. Gradient Boosting, in particular, achieved the highest recall without excessively inflating false positives, making it well suited for imbalanced fraud detection tasks. PR and ROC curves further confirmed these findings, with ensemble methods producing curves that dominated the linear and instance-based models. KNN and SVM performed moderately but were more sensitive to scaling and imbalance, resulting in less stable performance across folds. Feature importance plots for tree-based models revealed that a small subset of PCA components, particularly those representing higher-variance directions, contributed disproportionately to classification outcomes. This was consistent across both Random Forest and Gradient Boosting models.
Interpretability tools provided insight into how models distinguished fraudulent from legitimate transactions. Feature importance measures indicated that a handful of PCA-derived variables played dominant roles, while Time and Amount contributed more modestly. Although the PCA transformation limits direct domain interpretation, SHAP values helped highlight general patterns, such as certain feature combinations that consistently increased predicted fraud probability. Partial Dependence Plots further illustrated how specific features influenced predicted outcomes, while LIME explanations clarified local decision boundaries for individual transactions. Collectively, these tools helped bridge the gap between high-performance models and practical explainability needs within financial institutions, where model transparency is essential for regulatory and operational justification.
The results demonstrate that machine learning models can effectively detect fraudulent transactions, even in highly imbalanced settings, when appropriate preprocessing, resampling, and evaluation strategies are used. Ensemble methods proved especially valuable due to their ability to learn complex, nonlinear patterns that simpler models could not capture. Logistic Regression, while interpretable, struggled to achieve high recall because its linear decision boundary could not adequately separate the minority class. Decision Trees provided interpretability but exhibited instability across folds. In contrast, Random Forests delivered consistent performance and reduced variance, while Gradient Boosting achieved the highest detection capability with manageable false positives.
However, we also note several limitations. The PCA-transformed features reduce interpretability, complicating efforts to explain exactly why certain transactions are flagged. Additionally, the dataset spans only two days of activity, limiting temporal generalizability. Some models also risk overfitting due to the low volume of fraud cases, despite cross-validation and regularization. Overall, the comparison highlights the tradeoffs between interpretability and predictability. Models like Gradient Boosting offer strong fraud detection capabilities but require interpretability tools to remain practical for real-world deployment. Conversely, simpler models offer transparency but lack predictive strength. Identifying an appropriate balance depends on the operational and regulatory environment in which the model is deployed.
Several directions could strengthen this analysis. For example, more advanced imbalance-handling methods such as SMOTE-Tomek Links or ADASYN could reduce noise introduced by oversampling and improve minority-class structure. Beyond supervised learning, anomaly detection approaches, Isolation Forests or Autoencoder-based reconstruction error, may be better suited for extremely rare fraud patterns. Another extension is cost-sensitive learning, where the model explicitly penalizes missed fraud more heavily than false positives, aligning training objectives with real financial loss. Finally, integrating the model into a simulated or real-time pipeline would allow evaluation under operational constraints, including latency, drift, and evolving fraud behavior.
This project demonstrates that machine learning offers substantial improvements over rule-based methods for detecting fraudulent credit card transactions. Although the dataset is highly imbalanced and the PCA-transformed features limit interpretability, ensemble models, particularly Gradient Boosting, achieve strong recall and overall classification performance with the support of appropriate resampling and evaluation techniques. Interpretability tools help mitigate challenges introduced by anonymized features, offering transparency into model behavior and supporting responsible deployment. While no model completely eliminates false positives or false negatives, the results indicate that thoughtfully designed machine learning systems can significantly enhance fraud detection and support financial institutions in reducing losses and protecting consumers.
Sources: