Skip to content

Dataset and Embeddings

Sudhanshu edited this page Nov 11, 2026 · 1 revision

Dataset and Embedding Design

Synthetic smoke-test path

The make_smoke_data function creates 80 balanced Yelp-style review sentences with positive/negative adjectives. SmokeTokenizer uses a small in-memory word vocabulary and torch.randn generates a 48-dimensional frozen token embedding matrix. It is a correctness and execution-path test, not a research-grade dataset.

Real-data path

  • Dataset identifier: fancyzhx/yelp_polarity.
  • Tokenizer and frozen embedding source: sentence-transformers/all-MiniLM-L6-v2.
  • The code calls AutoTokenizer.from_pretrained and AutoModel.from_pretrained, then copies only get_input_embeddings().weight as a CPU tensor. It does not use MiniLM's pretrained Transformer layers as its classifier backbone.
  • The custom model therefore has input width 384, with a freshly initialized encoder trained through the project's own training loop.

Data preparation order

  1. Shuffle Yelp examples with a fixed seed.
  2. Restrict candidate pool and filter empty or overlong text.
  3. Split selected original training data into training/validation.
  4. Separately select from the official test split.
  5. Convert text into token IDs, attention masks, and frozen embeddings within collate_fn.

Data governance

No review data or Hugging Face model weights are bundled in the ZIP. Check the upstream dataset/model license and terms before adding any weights or datasets to your repository. Avoid claiming a benchmark from synthetic smoke-test results.

Clone this wiki locally