Quality-of-life: tokenizer persistence and a corpus-embedding helper. Additive and
backward-compatible.
Added
BPETokenizer.save/BPETokenizer.load(andto_dict/from_dict):
serialize a trained BPE tokenizer (vocabulary + merges + end-of-word marker) to a
versioned JSON file and reconstruct it exactly, encoding/decoding identically.encode_texts(polaris.inference): batch-embed a sequence of texts with a
TextEmbedder, returning a(len(texts), embedding_dim)NumPy array — the
corpus-embedding step of a dense retriever.