Skip to content

v0.10.0 — Pretrained Embeddings (GloVe)

Choose a tag to compare

@cattolatte cattolatte released this 05 Jul 08:29

Pretrained Embeddings. A from-scratch GloVe loader and vocabulary-aligned
embedding-matrix builder, wired into both models. The benchmark honestly shows
GloVe does not break the ceiling on IMDB (+0.001 for the pooling model; the
transformer overfits harder and slips to 0.849) — with 25,000 labeled reviews the
model learns good embeddings from scratch anyway. Pretrained word vectors help
when labeled data is scarce; the lever that actually moves this task is
self-supervised (contextual) pretraining, which is the next milestone.

Added

  • load_glove and build_embedding_matrix (polaris.embeddings): load pretrained GloVe vectors and align them to a Vocabulary (GloVe vector per known word, seeded-random for out-of-vocabulary, zeros for padding).
  • pretrained_embeddings / freeze_embeddings parameters on MeanPoolingClassifier and TransformerEncoderClassifier, initializing the embedding layer from a matrix.

Changed

  • The IMDB example has a GLOVE_PATH setting: point it at a GloVe file (with the whitespace tokenizer) to initialize the model's embeddings from pretrained vectors.
  • README benchmark table gains a GloVe column and the third honest finding (pretrained word embeddings do not break the ~86% ceiling here).