Tokenization and Padding

Tokenization and padding

The project uses the tokenizer matching distilbert-base-uncased:

MODEL_NAME = "distilbert-base-uncased"
tokenizer = AutoTokenizer.from_pretrained(MODEL_NAME)

def tokenize_function(examples):
    return tokenizer(examples["text"], padding=False, truncation=True, max_length=128)

The dataset map applies tokenization in batches, removes the source text field, and converts features to Torch tensors. DataCollatorWithPadding(tokenizer=tokenizer) dynamically pads each batch for the Trainer.

Reasoning: avoiding global fixed-length padding can reduce wasted work for short samples. Long messages are truncated at 128 tokens; this may remove information needed for classification.

See data contract, training sequence.