Repository navigation
v0.3.2
New features and fixes, and nothing that worked before stops working: a patch release, as
CONTRIBUTING.md describes. Without a key, tokens are numbered as before.
Added
- Stable pseudonyms for search indexes and RAG:
Anonymizer(guards, pseudonyms=Pseudonyms(key))
computes every new token with HMAC-SHA256 under a secret key of at least 32 bytes, such as
<OSOBA_482193047556>, instead of numbering it per conversation. The same value gets the same
token in every document, process and question, with no shared state. A person's name is read in
the nominative where its first name and surname agree, so "Jan Kowalski", "Jana Kowalskiego" and
"z Janem Kowalskim" share a token, and an identifier is one value with or without its separators
and country prefix. Tokens keep the<LABEL_number>shape, with 12 digits by default. The key is
never shown, pickled or copied;key_idandsettingssay what an index was built with, and
scope=keeps two indexes under one key apart. A collision gets four digits more and is recorded
inPseudonyms(collisions=...). - Masking for an index:
anonymizer.index_chunks(passages, store)masks the passages of a
document with one mapping and puts what each token hides into aMappingStore, a two-method
protocol over the application's own encrypted table (InMemoryMappingStorefor tests and
examples), with a person's name in the nominative and a fingerprint to tell a collision.
anonymizer.mask_query(question, store)gives a question the documents' tokens, also for a
surname written alone, which no rule reads and the store keeps for every indexed person.mask_chunks,stored_mapping
andtokens_inmask without a store and restore found passages and answers, also while they
stream; each passage also comes with its mapping as written, to restore it exactly. - Haystack 3 components, with
pip install "polanon[haystack]":PolanonDocumentMaskermasks
documents before the splitter and the embedder of an indexing pipeline,PolanonQueryMasker
gives a question the documents' tokens,PolanonRestorerrestores the replies and the documents
found, andPolanonStreamRestorerrestores a streamed answer as a streaming callback. Nothing
that names a value reaches the document store: not the content, notmeta, and noidthat is
a plain hash of an original text. What the tokens stand for goes to the application's mapping
store only. The key is a HaystackSecret, read fromPOLANON_KEYand saved by name only.
examples/haystack_rag.pyruns offline;examples/haystack_openai.pyuses OpenAI and is paid.
Fixed
- A token the mapping holds is left whole when masked text is masked again, such as a passage
from an index or history sent back. The digits of<OSOBA_50810112354>read as a PESEL, a
language model readOSOBA_8as a place, a PIN label took the token after it for the PIN, and a
known company called "Firma" was found inside<FIRMA_1>; each corrupted the token so restore no
longer gave the value back. A placeholder that stands for itself still shelters nothing. - Titles stay out of a foreign name after a label. "z. Hd. Frau Wiśniewska" made "Frau" the
first name, so every later "Frau" in the letter was masked as her, and "Attn: Mr Oliver Bennett"
got a token apart from the "Oliver Bennett" of its sign-off. With two titles, as in "Frau Prof.
Anna Schmidt", the second one was masked as the person wherever it stood.