Flight Ops Delay-Note Classifier

10-class text classification of airline ops notes into delay categories using a fine-tuned DeBERTa-v3-small transformer.

Quickstart

# 1. Install dependencies
pip install -r requirements.txt

# 2. Train the baseline (TF-IDF + Logistic Regression)
python baseline.py

# 3. Fine-tune the transformer (GPU recommended — see Colab section below)
python train.py

# 4. Evaluate transformer on test set (with latency benchmark)
python evaluate.py --model transformer --latency --compare_baseline artifacts/baseline_metrics.json

# 5. Analyze the hard set
python analyze_hard.py

Results

Model	test.csv Acc	test.csv Macro-F1	hard.csv Acc	hard.csv Macro-F1	CPU Latency
TF-IDF + LR (baseline)	99.5%	0.992	59.2%	0.347	<1ms
DeBERTa-v3-small (fine-tuned)	96.5%	0.919	28.3%	0.116	240ms

Note: test.csv and train.csv share 123 verbatim notes (946 rows after deduplication) due to the synthetic generator's finite template vocabulary. Both models are trained on the deduplicated set. hard.csv is the honest benchmark — it contains genuinely ambiguous multi-cause notes not seen during training.

Label Set

Code	Description
WX	Weather
LI	Late inbound (aircraft connection)
CR	Crew
AC	Aircraft / mechanical
ATC	ATC / airspace / slot
PX	Passenger / boarding
AP	Airport / ground
FL	Fueling / loading / catering
OD	Operational decision
SE	Security

Repo Structure

flight-ops-classifier/
├── train.py              # Fine-tune DeBERTa-v3-small (CLI)
├── evaluate.py           # Evaluate either model on any CSV
├── baseline.py           # Train + evaluate TF-IDF + LR baseline
├── analyze_hard.py       # Hard-set robustness probe with top-2 confidence
├── colab_run.ipynb       # Step-by-step Colab training notebook
├── requirements.txt
├── data/
│   ├── train.csv         # 2500 rows, ~5% noisy labels
│   ├── test.csv          # 400 rows, clean labels
│   ├── hard.csv          # 120 genuinely ambiguous notes
│   └── label_descriptions.json
├── src/
│   ├── config.py         # LABELS, SEED, MODEL_NAME, paths
│   ├── data.py           # load, clean (label token stripping), split, dedup
│   ├── baseline_model.py # TF-IDF + LR pipeline
│   ├── transformer_model.py
│   ├── trainer.py        # WeightedTrainer (class-weighted CE loss)
│   ├── metrics.py        # macro-F1, per-class F1, confusion matrix
│   ├── latency.py        # CPU inference benchmark
│   └── plots.py          # training curves, confusion matrix PNGs
├── artifacts/            # Generated: metrics JSON, model checkpoint, PNGs
└── memo/
    └── decision_memo.md

Reproducing the Model (Google Colab — free T4 GPU)

The model checkpoint is not committed (540MB). To reproduce:

1. Zip the project: cd ~/Desktop && zip -r dddd.zip dddd/
2. Open colab_run.ipynb in Google Colab
3. Set runtime to T4 GPU: Runtime → Change runtime type → T4 GPU
4. Run all cells in order (~10 minutes)
5. Download artifacts.zip from the final cell

CLI Reference

# Baseline
python baseline.py [--train_data data/train.csv] [--test_data data/test.csv]

# Training
python train.py [--epochs 5] [--lr 2e-5] [--batch_size 16]

# Evaluation
python evaluate.py --model {baseline,transformer} [--data data/test.csv] [--latency]

# Hard set analysis
python analyze_hard.py [--model_path artifacts/model] [--hard_data data/hard.csv]

Decision Memo

See memo/decision_memo.md for model choice rationale, data observations, evaluation choices, deployment sketch, and what we would do with another week.

Data Preprocessing Notes

Two preprocessing steps applied before training:

1. Label token stripping — label acronyms (WX, ATC, AC, OD etc.) are removed from note text to prevent keyword shortcutting. Without this, both models score 100% on test.csv by matching label tokens rather than learning delay semantics.

2. Train/test deduplication — 946 training rows whose notes appear verbatim in test.csv are removed before training, leaving 1,554 training rows. This ensures test.csv accuracy is a valid evaluation metric.