TalentMatch.ai — Fine-Tuning Research Study

TalentMatch.ai is an ML research project investigating the feasibility of automated resume-to-job-description fit scoring using fine-tuned transformer models on publicly available datasets. The project covers the full ML lifecycle across 6 notebooks: data exploration, full fine-tuning, LoRA fine-tuning, dataset comparison, and classification experiments.

Key finding: All three modeling approaches (regression with GPT-4o labels, regression with cosine-similarity labels, and 3-class classification) achieved reasonable validation metrics but failed to generalize to real-world inputs. The root cause is not model architecture — it is label inconsistency across public datasets. This project demonstrates that resume-JD matching at production quality requires either proprietary human-labeled data or a fundamentally different problem formulation.

---

Project Evolution

This project went through three major pivots, each driven by a concrete finding:

Phase 1 → 2: Started with regression (predict a 0-100 score). Built a working pipeline with MAE 11.95 on held-out test data. Demo revealed the model failed on real-world inputs despite good validation metrics — a generalization gap caused by limited training data (680 samples).

Phase 2 → 3: Attempted to fix generalization by retraining on a larger dataset (5,099 samples, 0xnbk/resume-ats-score-v1-en). Result was worse (MAE 14.18 vs 11.95) because the new dataset used cosine-similarity-derived labels rather than human judgment — confirming that label quality beats label quantity.

Phase 3 → Final: Pivoted from regression to 3-class classification (No Fit / Potential Fit / Good Fit) on 7,225 combined samples. Achieved 70.3% validation accuracy and 0.69 macro F1. The model correctly handles cross-domain mismatches (e.g. nurse → backend engineer) but still struggles on out-of-distribution real-world inputs due to the underlying label noise problem.

---

Notebooks

Notebook	Description	Key Output
01_data_exploration.ipynb	Load netsol/resume-score-details, fix schema mismatch, normalize scores	851 clean samples, train/val/test CSVs
02_full_finetune.ipynb	Full fine-tuning of DistilBERT for regression	MAE 11.95, RMSE 14.90 on test set
03_lora_finetune.ipynb	LoRA fine-tuning with PEFT (1.09% trainable params)	MAE 17.51 — LoRA underperforms on small datasets
04_retrain_ats_dataset.ipynb	Retrain on larger ATS dataset (5,099 samples)	MAE 14.18 — worse despite 10x more data
05_domain_classifier.ipynb	Abandoned — superseded by notebook 06	N/A
06_three_class_classifier.ipynb	3-class classifier on 7,225 combined samples	70.3% accuracy, 0.69 macro F1

---

Full Benchmark Results

Regression Experiments

Approach	MAE	RMSE	Train Samples	Epochs	Notes
Naive baseline	17.69	21.23	N/A	N/A	Always predict mean score
Full fine-tuning (Ph2)	11.95	14.90	680	4	Best regression result
LoRA fine-tuning (Ph3)	17.51	21.12	680	4	Matches baseline — LoRA needs larger models
Retrain on ATS data (Ph4)	14.18	19.17	5,099	4	More data, worse result — label quality issue

Classification Experiment

Metric	Value
Overall Accuracy	70.3%
Macro F1	0.69
No Fit F1	0.75
Potential Fit F1	0.60
Good Fit F1	0.72
Training samples	7,225 (combined)
Epochs	7 (4 initial + 3 continued)

HuggingFace Hub Models

Model	Task	Link
TalentMatch-AI-full	Regression (0-100)	LucasLisboadev/TalentMatch-AI-full
TalentMatch-AI-lora	LoRA adapter	LucasLisboadev/TalentMatch-AI-lora
TalentMatch-AI-classifier	3-class classifier	LucasLisboadev/TalentMatch-AI-classifier

---

Datasets Used

Dataset	Samples	Label Type	Label Source
netsol/resume-score-details	851 valid	Continuous 0-100	GPT-4o structured rubric
0xnbk/resume-ats-score-v1-en	6,374	Continuous 0-100 + 3-class	Cosine similarity of embeddings

---

Key Engineering Decisions

Why DistilBERT? Small enough to train on Colab free tier (T4 GPU, 15.6GB), fast enough for multiple experiments, and strong enough for sentence-pair classification tasks. DistilBERT is 40% smaller than BERT with 97% of its performance on GLUE benchmarks.

Why full fine-tuning over LoRA for small datasets? LoRA freezes 99% of model weights and only trains small adapter matrices. On large models (7B+ params) this works because the base model has enormous representational capacity. On DistilBERT (67M params) with 680 samples, freezing 99% of weights is too restrictive — the model cannot adapt enough. Full fine-tuning outperformed LoRA by 5.5 MAE points.

Why regression failed to generalize despite good validation metrics? The netsol dataset has only 680 training samples covering a narrow slice of resume-JD pairs. The model memorized patterns specific to this distribution. When presented with real-world inputs (different length, different formatting, different vocabulary distribution), it had no reference point. This is the classic train-test distribution shift problem.

Why did more data make things worse? The 0xnbk ATS dataset uses cosine similarity between sentence embeddings as the ground truth score. This measures semantic overlap between word embeddings — not actual human judgment of fit. A nurse resume has low cosine similarity with a backend engineer JD, which is correct. But a senior Python engineer resume may have different vocabulary than the JD even when the fit is strong, leading to misleading labels. Training on these labels teaches the model to measure word overlap, not actual job fit.

Why pivot to classification? Regression requires the model to predict a precise continuous value (73.4 vs 71.2), which demands very high label consistency. Classification only requires distinguishing broad categories (Good / Potential / No Fit), which is a simpler signal that the model can learn with noisier labels. Classification also maps better to real recruiter workflows — they need fit/no-fit decisions, not arbitrary scores.

Why 70.3% accuracy is honest and Potential Fit F1 of 0.60 is expected? The middle class (Potential Fit) is inherently the hardest to classify in any ordinal 3-class problem. Its boundaries with both Good Fit and No Fit are fuzzy by definition. Additionally, the combined dataset has two different labeling systems (GPT-4o rubric vs cosine similarity) which creates the most label noise exactly at these boundaries. The No Fit class (0.75 F1) and Good Fit class (0.72 F1) are well-separated and the model handles them reliably.

---

Why Real-World Generalization Failed

The core issue is distribution shift between training data and real-world inputs:

1. Training resumes in public datasets are long, dense, keyword-heavy (often from resume-sharing sites)
2. Real resumes vary enormously in format, length, and vocabulary
3. Training JDs come from specific job postings with structured formats
4. Real JDs from LinkedIn use different language, vary in detail level

The model learned the statistical patterns of the training distribution. When real inputs don't match those patterns, performance degrades. This is not fixable with more epochs or different architectures — it requires data that actually represents the target distribution.

What would actually solve this:

• 50,000+ human-labeled resume-JD pairs with consistent rubric
• Active learning: deploy, collect real user feedback, retrain
• Proprietary data from ATS platforms (LinkedIn, Greenhouse, Lever)

---

Stack

• Model: distilbert-base-uncased (HuggingFace Transformers)
• Training: HuggingFace Trainer API + PEFT for LoRA
• Evaluation: scikit-learn metrics (accuracy, F1, classification report)
• Compute: Google Colab T4 GPU (free tier)
• Data storage: Google Drive (intermediate CSVs), HuggingFace Hub (model weights)
• Demo: Gradio on HuggingFace Spaces
• Version control: GitHub

---

Author

Lucas Lisboa — GitHub | Portfolio | HuggingFace