Repository navigation
Architecture LLD
Sudhanshu edited this page Nov 11, 2026
·
1 revision
flowchart TD
A["RAW_DATA, LABEL_NAMES, SEED"] --> B["train_test_split(stratify=labels)"]
B --> C["Dataset.from_dict + DatasetDict"]
C --> D["tokenize_function(examples)"]
D --> E["data_collator"]
F["create_training_arguments(**kwargs)"] --> G["TrainingArguments"]
G --> H["create_trainer(**kwargs)"]
E --> H
H --> I["Trainer.train()"]
I --> J["Trainer.evaluate()"]
J --> K["compute_metrics(eval_pred)"]
I --> L["Trainer.predict(validation)"]
I --> M["Trainer.save_model(best_model)"]
M --> N["classify_query(query)"]
N --> O["torch.softmax(logits)"]
| Defined in Python source | Signature | Responsibility |
|---|---|---|
| Tokenization | tokenize_function(examples) |
Tokenize batched text; truncate at 128 tokens, no upfront padding |
| Metrics | compute_metrics(eval_pred) |
Convert logits to labels, return accuracy and weighted F1 |
| Argument compatibility | create_training_arguments(**kwargs) |
Map warmup and evaluation arguments by installed Transformers signature |
| Trainer creation | create_trainer(**kwargs) |
Choose processing_class or legacy tokenizer parameter |
| Local inference | classify_query(query: str) |
Run trained full model and map softmax probabilities to labels |
RAW_DATA, dataset construction, training experiments, and inference demonstrations execute as top-level script statements rather than inside a main() function.
Dataset and DatasetDict manage examples; AutoTokenizer encodes text; DataCollatorWithPadding pads; TrainingArguments configures optimizer/training schedule; Trainer runs training/evaluation/checkpointing; AutoModelForSequenceClassification wraps the DistilBERT encoder and four-class head.
See the dependency diagram and training sequence.
Getting started
Architecture
- System Context
- High-Level Design
- Low-Level Design
- Model Components
- Training Sequence
- Data & Labels
- Tokenization
Training & results
Engineering reference