Abstract – Once a Machine Learning (ML) model is deployed, the same model is typically retrained from scratch, either on a scheduled interval or as soon as model drift is detected, to make sure the model reflects current data distributions and performance experiments. As such, once a new model is available, the old model typically is discarded. This paper challenges the notion of older models being useless by showing that old models still have substantial value compared to newly trained models, and by proposing novel post-deployment model recycling techniques that help make informed decisions on which old models to reuse and when to reuse. In an empirical study on eight long-lived Apache projects comprising a total of 84,343 commits, we analyze the performance of five model recycling strategies on three different types of Just-In-Time defect prediction models (Random Forest (RF), Logistic Regression (LR) and Neural Network (NN)). Comparison against traditional model retraining from scratch (RFS) shows that our approach significantly outperforms RFS in terms of recall, g-mean, AUC and F1 by up to a median of
- RQ1 – The results for RQ1 are available under
/rq1.ipynbnotebook here. - RQ2 – To look at the analysis of all variant results, please refer to the following notebooks
/analysis.ipynbnotebook for random forest variants/analysis-lr.ipynbnotebook for logistic regression variants/analysis-nn.ipynbnotebook for deep neural network variants
- RQ3 - For latency evaluation of best variants, please refer to
/latency.ipynbnotebook under their branch listed in variants table.
The source code to replicate the results for each project under each recycling strategy variant is available in its independent branch as listed below.
The variants with different window sizes and shift sizes are listed below.
The additional experiment of DeepJIT using commit messages is available under the branch deepjit-commit-messages with results available in results.zip attached under the releases.
- Check out the variant branch:
git checkout <<branch_name_here>>
-
Download the commit metrics from this link and put it in
/ecoselekt/data/directory. -
Download the "ApacheJIT Commits" zip from the releases and extract it in
/ecoselekt/data/directory. -
Install dependencies:
poetry install
- Create time windows and train RFS model versions:
# train random forest models
make mlms
# train logistic regression models
make lr
# train neural network models
make nn
- Evaluate RFS models and create a baseline:
# evaluate random forest models
poetry run eval_models
# evaluate logistic regression models
poetry run eval_lr
# evaluate neural network models
poetry run eval_nn
- [Required only for Model Stacking and Model Voting variants] Install ScottKnottESD R package
poetry run setup_r
- Prepare for recycling:
# prepare for random forest models
make prep_reuse
# prepare for logistic regression models
make prep_reuse_lr
# prepare for neural network models
make prep_reuse_nn
[Optional] – To evaluate inference latency results (script only present under the best experiments branches):
# for rf
poetry run inf_latency
# for lr
poetry run inf_latency_lr
# for nn
poetry run inf_latency_nn
- Inference with recycling:
# for rf
make inf_reuse
# for lr
make inf_reuse_lr
# for nn
make inf_reuse_nn
- Evaluate recycling:
# for rf
make eval_reuse
# for lr
make eval_reuse_lr
# for nn
make eval_reuse_nn
- Evaluate historical models over time windows:
poetry run process_selekt_vs_best
- Finally, run statistical tests on results:
poetry run stat_test
By default, each step stores output results in the /ecoselekt/data/ directory, while the models are stored in ecoselekt/models/ directory.
- Harsh Patel @harsh8398
- Dr. Bram Adams
- Dr. Ahmed E. Hassan