Skip to content

Architecture LLD

Sudhanshu edited this page Nov 11, 2026 · 1 revision

Low-Level Design (LLD)

Function-level call and data dependencies

flowchart TD
    A["RAW_DATA, LABEL_NAMES, SEED"] --> B["train_test_split(stratify=labels)"]
    B --> C["Dataset.from_dict + DatasetDict"]
    C --> D["tokenize_function(examples)"]
    D --> E["data_collator"]
    F["create_training_arguments(**kwargs)"] --> G["TrainingArguments"]
    G --> H["create_trainer(**kwargs)"]
    E --> H
    H --> I["Trainer.train()"]
    I --> J["Trainer.evaluate()"]
    J --> K["compute_metrics(eval_pred)"]
    I --> L["Trainer.predict(validation)"]
    I --> M["Trainer.save_model(best_model)"]
    M --> N["classify_query(query)"]
    N --> O["torch.softmax(logits)"]
Loading

Application functions

Defined in Python source Signature Responsibility
Tokenization tokenize_function(examples) Tokenize batched text; truncate at 128 tokens, no upfront padding
Metrics compute_metrics(eval_pred) Convert logits to labels, return accuracy and weighted F1
Argument compatibility create_training_arguments(**kwargs) Map warmup and evaluation arguments by installed Transformers signature
Trainer creation create_trainer(**kwargs) Choose processing_class or legacy tokenizer parameter
Local inference classify_query(query: str) Run trained full model and map softmax probabilities to labels

RAW_DATA, dataset construction, training experiments, and inference demonstrations execute as top-level script statements rather than inside a main() function.

Framework responsibilities

Dataset and DatasetDict manage examples; AutoTokenizer encodes text; DataCollatorWithPadding pads; TrainingArguments configures optimizer/training schedule; Trainer runs training/evaluation/checkpointing; AutoModelForSequenceClassification wraps the DistilBERT encoder and four-class head.

See the dependency diagram and training sequence.

Clone this wiki locally