XCodeval Tag Prediction using Modern BERT and GPT

This project focuses on predicting tags for the XCodeval dataset using Modern BERT and GPT.

---

🧩 Overview

• Dataset: XCodeval
• Task: Multi-label tag classification
• Model Used: Modern BERT (Base) - GPT (open AI)
• Best Macro F1: 0.81 Modern BERT

---

Launching the project

chmod +x install.sh

./install.sh 

source venv/bin/activate

uvicorn server:app --host 0.0.0.0 --port 8000

curl http://localhost:8000/health

python predict.py --from-dataprep --n  10 --server http://localhost:8000

Now The link of mlflow in the predict.py file will change , it is tight to the dns of AWS EC2 instance , and also you should be an AWS user that have access to my S3 bucket. We can check this at anytime drop a message.

⚙️ Workflow Summary

🧹 Data Preparation (data_prep.ipynb)

• Loads and cleans the XCodeval dataset.
• Saves the cleaned version as my_data.csv.

💻 Model Fine-Tuning

1. Colab Stage:
   Initial experiments and hyperparameter sweeps on Google Colab (T4 GPU).

2. RunPod Stage:
   When Colab hit usage limits, we migrated to RunPod.io — renting NVIDIA A5000 GPUs for full-scale fine-tuning.

---

☁️ System Architecture

A robust cloud-based architecture was implemented to manage and track all experiments using MLflow and AWS services.

Architecture Highlights

• Compute Layer:

  • Training initiated from Google Colab or RunPod for scalable GPU usage.
  • RunPod instances handle large-scale fine-tuning with A5000 GPUs.

• Tracking & Storage:

  • MLflow Server hosted on an AWS EC2 instance for centralized experiment tracking.
  • PostgreSQL (RDS) acts as the MLflow backend to store metrics, hyperparameters, and model metadata.
  • S3 Bucket used as the artifact store for model checkpoints and logs.

• Connectivity & Access:

  • MLflow server accessible at:
    http://ec2-13-60-228-74.eu-north-1.compute.amazonaws.com:5000/

• Pipeline Flow:

  1. Model training executed on RunPod with MLflow tracking enabled.
  2. MLflow server logs all metrics to PostgreSQL and uploads artifacts to S3.

---

🧠 Architecture Diagram

The following diagram illustrates the system flow between RunPod, and AWS services:

---

🎬 Visualization

The code for the Simulation is in the manim directory. You can do it on your own !