Repository navigation
Dataset and Embeddings
Sudhanshu edited this page Nov 11, 2026
·
1 revision
The make_smoke_data function creates 80 balanced Yelp-style review sentences with positive/negative adjectives. SmokeTokenizer uses a small in-memory word vocabulary and torch.randn generates a 48-dimensional frozen token embedding matrix. It is a correctness and execution-path test, not a research-grade dataset.
- Dataset identifier:
fancyzhx/yelp_polarity. - Tokenizer and frozen embedding source:
sentence-transformers/all-MiniLM-L6-v2. - The code calls
AutoTokenizer.from_pretrainedandAutoModel.from_pretrained, then copies onlyget_input_embeddings().weightas a CPU tensor. It does not use MiniLM's pretrained Transformer layers as its classifier backbone. - The custom model therefore has input width 384, with a freshly initialized encoder trained through the project's own training loop.
- Shuffle Yelp examples with a fixed seed.
- Restrict candidate pool and filter empty or overlong text.
- Split selected original training data into training/validation.
- Separately select from the official test split.
- Convert text into token IDs, attention masks, and frozen embeddings within
collate_fn.
No review data or Hugging Face model weights are bundled in the ZIP. Check the upstream dataset/model license and terms before adding any weights or datasets to your repository. Avoid claiming a benchmark from synthetic smoke-test results.
Home · HLD · LLD · CLI · Validation This project is licensed under the Apache License, Version 2.0.© 2026 Sudhanshu Shukla · Licensed under the Apache License, Version 2.0.