Upon evolving their software, organizations and individual developers have to spend conspicuous effort to pay back technical debt, i.e., the fact that software is released in a shape not as good as it should be, e.g., in terms of functionality, reliability, or maintainability. This paper empirically investigates the extent to which technical debt can be automatically paid back by neural-based generative models, and in particular models exploiting different strategies for pre-training and fine-tuning. We start by extracting a dateset of 5,039 Self-Admitted Technical Debt (SATD) removals from 809 open-source projects. SATD refers to technical debt instances documented (\eg via code comments) by developers. We use this dataset to experiment with seven different generative deep learning (DL) model configurations. Specifically, we compare transformers pre-trained and fine tuned with different combinations of training objectives, including the fixing of generic code changes, SATD removals, and SATD-comment prompt tuning. Also, we investigate the applicability in this context of a recently-available Large Language Model-based chat bot. Results of our study indicate that the automated removal of SATD is a challenging task, with the best model we experimented with able to automatically fix $\sim$2% to 8% of test instances, depending on the number of attempts it is allowed to make. Given the limited size of the fine-tuning dataset ($\sim$5k instances), the model's pre-training plays a fundamental role in boosting performance. Also, the ability to remove SATD steadily drops if the comment documenting the SATD is not provided as input to the model.
To replicate the experiments you can rely on this two files codet5.py and t5-no-pt.py. While the former can be used to train M1, M2_cc as well as the ablation model (i.e., M2-cc-ablation), the latter serves the no pre-training experiment. Before starting replicating any of the experiments we performed, make sure to install the requirements (see requirements.txt)
python3.9 codet5.py
--model_name=model.bin
--output_dir=./models/M1/
--tokenizer_name=Salesforce/codet5-base
--model_name_or_path=Salesforce/codet5-base
--do_train
--epochs 50
--encoder_block_size 1024
--decoder_block_size 1024
--train_batch_size 2
--eval_batch_size 2
--learning_rate 2e-5
--max_grad_norm 1.0
--evaluate_during_training
--seed 123456 2>&1 | tee ./models/M1/train.log
python3.9 codet5.py
--model_name=model.bin
--output_dir=./models/M1/
--tokenizer_name=Salesforce/codet5-base
--model_name_or_path=Salesforce/codet5-base
--do_test
--encoder_block_size 1024
--decoder_block_size 1024
--eval_batch_size 8
--num_beams=1 - The datasets for fine-tuning the models are stored on GDrive here
- The dataset for supervised pre-training which includes code changes (Tufano et al.) is available here here
- T5-base No-Pretraining (M0)
- CodeT5-SATD (M1)
- CodeT5-Code-Changes
- CodeT5-Code-Changes + Fine-tuning on SATD removal dataset (M2-cc)
- CodeT5-Code-Changes + Fine-tuning on SATD removal ablated dataset (M3-cc-ablation)