v1.4.0 — Tokenizer save/load & encode_texts

Quality-of-life: tokenizer persistence and a corpus-embedding helper. Additive and
backward-compatible.

Added

• BPETokenizer.save / BPETokenizer.load (and to_dict / from_dict):
  serialize a trained BPE tokenizer (vocabulary + merges + end-of-word marker) to a
  versioned JSON file and reconstruct it exactly, encoding/decoding identically.
• encode_texts (polaris.inference): batch-embed a sequence of texts with a
  TextEmbedder, returning a (len(texts), embedding_dim) NumPy array — the
  corpus-embedding step of a dense retriever.

v1.3.0 — Sentence-Pair Cross-Encoder

Sentence-Pair Cross-Encoder. A joint-encoding pair head (SentencePairClassifier,
num_classes for rerank/gate/NLI), pair collation, encoder segment support, and
special-token reservation. Additive and backward-compatible.

Added

• SentencePairClassifier (polaris.models): a cross-encoder that joint-encodes
  [CLS] a [SEP] b [SEP], pools ("cls" or "mean"), and scores with a linear
  head — num_classes=1 (ranking score), 2 (gate), 3 (NLI). Shares the trunk,
  so it initializes from a pretrained MLM via transfer_encoder_to.
• PairBatch / collate_pairs (polaris.collation): pack (text_a, text_b, label)
  triples into [CLS] a [SEP] b [SEP] with segment ids and longest-first truncation.
• cls_token / sep_token on Vocabulary (with cls_id / sep_id),
  build_vocabulary, and train_bpe (which also gains mask_token) — reserved
  first and contiguously so adding a special never renumbers corpus tokens.
• The build_model factory now recognizes the "pair_classifier" model type.

Changed

• TransformerEncoder.forward gains an optional token_type_ids argument; a
  zero-initialized segment embedding is added only when it is given, so existing
  single-segment call sites are byte-identical. Additive, backward-compatible.

v1.2.0 — Text Embedder & Contrastive Training

Text Embedder & Contrastive Training. A mean-pooled bi-encoder embedding model
(TextEmbedder) and the InfoNCE objective to train it (in-batch and optional hard
negatives). Additive and backward-compatible.

Added

• TextEmbedder (polaris.models): a bi-encoder tower that emits a single
  (optionally L2-normalized) embedding per text — the shared TransformerEncoder
  trunk, mask-aware mean pooling, an optional linear projection, and optional
  normalization. forward(Batch) and encode(input_ids, attention_mask).
• mean_pool (polaris.models): the mask-aware mean-pool, factored out of the two
  classifiers (which now reuse it) so the embedder can share it.
• HasEncoder (polaris.models): a structural protocol for models that wrap a
  TransformerEncoder as encoder, used to type weight transfer.
• ContrastiveBatch / collate_contrastive (polaris.collation): collate
  (anchor, positive) or (anchor, positive, hard_negatives) pairs into aligned
  batches (hard negatives flattened with a per-anchor count).
• info_nce_loss (polaris.training.losses): InfoNCE contrastive loss with
  in-batch negatives, optional per-anchor hard negatives, and an optional symmetric
  (bidirectional) term.
• train_contrastive (polaris.training): a minimal seeded driver that trains a
  TextEmbedder with info_nce_loss.
• The build_model factory now recognizes the "embedder" model type.

Changed

• MaskedLanguageModel.transfer_encoder_to accepts any HasEncoder (the
  classifier or the new TextEmbedder) rather than only the classifier — a widened
  parameter, backward-compatible.

v1.1.0 — Interactive Console

The Interactive Console. polaris console opens a msfconsole-style REPL: a
block-letter ASCII banner in a randomly chosen color scheme (bundled art +
local randomness — no network, the Metasploit pattern), a polaris > prompt, and
commands to load a saved model bundle once and predict on raw text many times —
instant after the first load, unlike the one-shot polaris predict. Standard
library only (cmd + raw ANSI); color respects NO_COLOR and non-TTY output.

Added

• polaris.console: PolarisConsole (a cmd.Cmd REPL with load, predict,
  probs, info, exit/quit, and built-in help), render_banner (bundled
  block-letter art, five ANSI color schemes, seedable), and run.
• polaris console [--model BUNDLE]: open the console, optionally pre-loading a
  bundle.

v1.0.1 — First PyPI Release

First PyPI release — Polaris is now installable with pip install polaris-nlp
(the import and CLI name remain polaris). No code changes.

Added

• A Release to PyPI workflow (.github/workflows/release.yml): pushing a version
  tag builds the sdist/wheel and publishes to PyPI via trusted publishing
  (OIDC) — no API token is stored in the repository.

Changed

• Package classifier updated from Development Status :: 3 - Alpha to
  5 - Production/Stable, matching the v1.0 release.

v1.0.0 — Polaris Stable

Polaris Stable — the first public release. No new features: this is the
graduation pass over everything built across v0.1–v0.12. The full from-scratch NLP
lifecycle — data → tokenization → collation → models (a from-scratch transformer) →
training → evaluation → experiment tracking → pretrained embeddings →
self-supervised pretraining → deployment — is runnable end to end, documented, and
frozen under semantic versioning.

Changed

• API freeze. The public API is each module's __all__; these surfaces are now
  stable under semver. The top-level polaris package intentionally stays minimal
  (only __version__) so importing it is cheap and needs no optional extras.
• Documentation pass. Every implemented module now has an accurate README (fixed
  the core, models, evaluation, experiments, and utils stubs; added
  collation and embeddings). The top-level README gains Installation and
  Quickstart sections, and the badges/intro reflect the stable release.
• CI hardening. The workflow now runs a matrix — Python 3.12 & 3.13 on Ubuntu
  and macOS — with the fail_under = 90 coverage gate enforced.

v0.12.0 — Deployment & CLI

Deployment & CLI — the final feature phase before v1.0. A trained model becomes a
usable artifact: save a self-describing bundle (model type + config + weights +
vocabulary + labels), reload it anywhere with no training code, and run it from the
command line (polaris predict), over HTTP (polaris serve → FastAPI
POST /predict), or in a container (Dockerfile). This closes the end-to-end
story: a person can now train a real model on a real dataset and serve it.

Added

• polaris.inference: turn a trained model into a usable artifact. save_bundle /
  load_bundle persist a self-describing bundle (model type + config + weights +
  vocabulary + labels) and reload it as a ready Predictor; Predictor runs the
  whole raw-text path (tokenize → collate → forward → softmax) and returns a
  Prediction (label + per-class probabilities); build_model reconstructs a
  model from its saved type and config.
• polaris.deployment: a thin HTTP serving layer — create_app wraps a
  Predictor (loaded from a bundle) in a FastAPI app exposing GET /health and
  POST /predict. FastAPI/uvicorn are an optional serving extra, imported lazily.
• CLI commands: polaris predict TEXT --model BUNDLE [--probs] (classify raw text)
  and polaris serve --model BUNDLE (serve it over HTTP).
• A Dockerfile to serve a model in a container (model bundle mounted at run time).
• Vocabulary.to_dict / Vocabulary.from_dict: a validated serialization
  round-trip, used by the bundle format.
• examples/pretrain_finetune_imdb.py now saves a bundle at the end of a run.

v0.11.0 — Self-Supervised Pretraining (Masked Language Modeling)

Self-Supervised Pretraining (Masked Language Modeling). Pretrain Polaris' own
transformer on unlabeled text with an MLM objective, then transfer the trunk into
a classifier and fine-tune — the BERT recipe, from scratch, no downloads. A
controlled ablation (same vocabulary and 4-layer architecture, pretraining the only
difference) shows pretraining works in the expected direction — a large warm start
(epoch-1 validation 0.810 vs 0.736), faster convergence, higher best validation
(0.864 vs 0.851) — but converges to the same ~86% test ceiling (0.853 vs 0.852),
because 25k labels already suffice and the small in-domain corpus injects little new
knowledge. Four from-scratch levers (transformer, BPE, GloVe, MLM pretraining) now
all land at ~85-86%: the ceiling is the task and data/compute regime, not any one
component.

Added

• polaris.pretraining: masked-language-model pretraining from scratch —
  mask_tokens / MaskedLMBatch (BERT-style 80/10/10 masking with dynamic,
  seedable corruption), MaskedLanguageModel (shared trunk + vocabulary head,
  with transfer_encoder_to for moving a pretrained trunk into a classifier),
  and pretrain (the MLM loop: masked-position cross-entropy, warmup scheduling,
  per-epoch loss and masked-token accuracy).
• TransformerEncoder (polaris.models): the shared, headless transformer trunk
  (embedding + positional + encoder blocks + final norm → per-token hidden
  states), now reused by both the classifier and the masked-language model.
• mask_token / mask_id on Vocabulary, and a mask_token argument on
  build_vocabulary.
• IMDB's "unsupervised" split (50,000 unlabeled reviews) is now loadable via
  IMDBDataset, for pretraining.
• examples/pretrain_finetune_imdb.py: the full pretrain → transfer → fine-tune
  thread on IMDB.

Changed

• TransformerEncoderClassifier is refactored to compose the shared
  TransformerEncoder trunk (behavior-preserving; its trunk weights now live
  under the encoder submodule).

v0.10.0 — Pretrained Embeddings (GloVe)

Pretrained Embeddings. A from-scratch GloVe loader and vocabulary-aligned
embedding-matrix builder, wired into both models. The benchmark honestly shows
GloVe does not break the ceiling on IMDB (+0.001 for the pooling model; the
transformer overfits harder and slips to 0.849) — with 25,000 labeled reviews the
model learns good embeddings from scratch anyway. Pretrained word vectors help
when labeled data is scarce; the lever that actually moves this task is
self-supervised (contextual) pretraining, which is the next milestone.

Added

• load_glove and build_embedding_matrix (polaris.embeddings): load pretrained GloVe vectors and align them to a Vocabulary (GloVe vector per known word, seeded-random for out-of-vocabulary, zeros for padding).
• pretrained_embeddings / freeze_embeddings parameters on MeanPoolingClassifier and TransformerEncoderClassifier, initializing the embedding layer from a matrix.

Changed

• The IMDB example has a GLOVE_PATH setting: point it at a GloVe file (with the whitespace tokenizer) to initialize the model's embeddings from pretrained vectors.
• README benchmark table gains a GloVe column and the third honest finding (pretrained word embeddings do not break the ~86% ceiling here).

v0.9.0 — Subword Tokenization (BPE)

A from-scratch Byte Pair Encoding tokenizer — the second real tokenizer — plus an honest re-run of the IMDB benchmark.

Added

• train_bpe learns subword merges from a corpus (word-level BPE, Sennrich et al.), and BPETokenizer applies them. It satisfies the v0.3 Tokenizer protocol (validating that contract) and splits rare/unseen words into known subwords instead of <unk>.
• A TOKENIZER switch in the example ("bpe" | "whitespace"); runs are recorded per model and tokenizer.

Benchmark (honest)

Re-running IMDB (25k/25k, seed 0) with BPE:

Model	Whitespace (20k)	BPE (10k)
Mean-pooling baseline	0.856	0.839
Transformer (from scratch)	0.855	0.838

BPE slightly hurt here. IMDB sentiment lives in common whole words, which BPE fragments (diluting the signal and losing more to truncation) — subwording pays off for out-of-vocabulary / morphology, not this task. This disproved the earlier hypothesis that tokenization was the ceiling: the real ceiling (~85–86%) is the model class — simple, from-scratch, no pretraining. The lever that breaks it (pretrained representations) is the next milestone.

BPE remains a correct, valuable tokenizer; the benchmark honestly shows when it helps.

See CHANGELOG.md for full details.