Model Internals

DistilBERT Model Internals

The pretrained DistilBERT encoder provides contextual representations; an added sequence-classification head produces four logits. This project fine-tunes pretrained weights and does not train a Transformer from scratch.

flowchart LR
    A[Raw support text] --> B[WordPiece tokenizer]
    B --> C[Input IDs + Attention Mask]
    C --> D[Pretrained DistilBERT Encoder]
    D --> E[CLS token hidden representation]
    E --> F[Pre-classifier Dense + ReLU]
    F --> G[Dropout + Classifier]
    G --> H[Four logits]
    H --> I[Softmax for inference]
    I --> J[billing / technical / account / feature_request]

Loading

Inputs and output contract

Tensor / value	Purpose
input_ids	Tokenized sequence of up to 128 tokens
attention_mask	Excludes padding tokens from self-attention
labels	Integer class index for training
logits	One logit per output class
softmax(logits)	Per-class probability distribution at inference

classify_query selects the maximum-probability class and returns query, predicted_label, confidence, and all_scores. These confidence values are not calibrated uncertainty estimates. The class labels are defined in Dataset-and-Labels.