Evaluation and Inference

Evaluation and Inference

Implemented evaluation

compute_metrics(eval_pred) takes model logits and calculates accuracy and weighted F1 via scikit-learn. The script also prints a classification report and confusion matrix for the validation dataset.

For new examples, classify_query(query) tokenizes one message, moves tensors to the trained model device, performs inference with torch.inference_mode(), applies softmax and returns four label scores.

Returned object (schema illustration; not a recorded model prediction)

{
  "query": "Example customer request",
  "predicted_label": "billing",
  "confidence": 0.0,
  "all_scores": {
    "billing": 0.0,
    "technical": 0.0,
    "account": 0.0,
    "feature_request": 0.0
  }
}

These zero values are placeholders to show field names, not actual classifier output. For an actual inference output, run training and inspect printed predictions. The fixed sample set contains five queries.

Limits

Six validation examples do not establish model robustness, fairness, accuracy under distribution shift, or reliable class calibration. Add held-out data, repeated experiments, and misclassification analysis before any real support-ticket routing.