Repository navigation
Releases
v0.3.6
Two modes for the language model, a benchmark of an AI agent's calls, and far fewer words masked
for nothing; nothing that worked before stops working, so this is a patch release, as
CONTRIBUTING.md describes. The spaCy guard now leaves out what it reads as a name in English
prose, code and identifiers, a place typed in lowercase among them, so an agent's prompt, tools
and results stay readable: in the benchmark's calls 5 of 307 token uses hide nothing, against 110
of 410 in 0.3.5. mode="paranoid", polanon-gateway --paranoid and polanon --ner --paranoid
mask all of it instead. A birthday under a JSON key and a login made of a known surname are now
masked, and the gateway can send images and files on unmasked when told to.
Added
-
polanon --ner --paranoidmasks with the spaCy guard inmode="paranoid". Without--ner
the flag is refused, since it changes nothing without the model. -
polanon-gateway --paranoidmasks with the spaCy guard inmode="paranoid", and the banner
says so. Without the model the flag changes nothing, and the banner says that too. -
mode="paranoid"masks every name the spaCy model finds. The filters that keep an AI
agent's English, code and identifiers readable let a little through by design: a name opening
an English sentence, a word in inline code, a place typed in lowercase.default_guards(mode= "paranoid")andNerGuard(mode="paranoid")keep every span with a letter in it instead. On
the agent's calls of the benchmark only what no guard finds leaks, 1 of 243 marked
occurrences against 3 in the defaultmode="normal", and 386 of 696 token uses hide nothing,
against 5 of 307. On the sample
documents it masks the same 187 values, with 8 over-masked spans instead of 6. The rules are
the same in both modes.benchmarks/run.pyandbenchmarks/run_agent.pytake--paranoid. -
A benchmark of an AI agent's calls, and of what comes back.
benchmarks/agent_corpus.py
holds eight invented calls of a personal assistant as its client sends them to a model, seven
to Chat Completions and one to the Responses API: an English system prompt with a Polish memory
of its user, sixteen tool definitions, and the JSON its tools return from a smart home, a web
search, a shell, a mailbox, a contact book and a file, with code, a traceback, a system log,
keys, a lease with two PESEL numbers, and mail in German and Ukrainian among it. The 103 values
that must not reach the model are marked where they are written.benchmarks/run_agent.py
masks each call as the gateway does and counts what still reaches the model and the tokens
that hide nothing, with their uses. It then restores the model's answer to each call, written
with the call's tokens and the cases the Polish note asks for, as text, as a whole response and
as a stream cut at every place, and scores each line and argument against what a reader should
get back. Withpl_core_news_md, 3 of 243 occurrences reach the model and 5 of 307 uses hide
nothing, against 5 and 110 of 410 in 0.3.5; 39 of 40 lines come back exact and none wrong.
tests/test_agent_benchmark.pyholds all of it as bounds, and CI prints the tables. -
The gateway can send what it cannot mask, when told to.
polanon-gateway --pass-unsupportedsends an image, a file or another part it cannot mask to the provider as it
is, unmasked, and masks the text around it, for an operator who wants the model to see a photo
or a document sent to an agent. The banner says so.MaskedTurn(unknown_blocks="keep")and
mask_responses_request(pass_unsupported=True)do the same in code. Without the flag such a
call is still refused, and--drop-unsupportedstill leaves the content out; the two flags
exclude each other.
Changed
-
WELL_KNOWN_NAMESholds hardware an agent names: Dell, Lenovo, Samsung, Roborock and HACS,
the store of Home Assistant's add-ons. In the agent's calls of the benchmark "Dell" was masked
as a company eight times in one call, "Roborock" as a person and "HACS" as a place. -
WELL_KNOWN_NAMESholds the platforms an AI agent names: OpenRouter, DeepSeek, Hugging
Face, Ollama, LM Studio, Claude Code, Hermes Agent, Nous Research, Home Assistant and its "HA",
GitHub, GitLab, Docker, Linux and Ubuntu. An agent driving a smart home named "HA" 117 times in
one call, and the Polish model took it for a place each time: the model saw a token in every
tool description and result in place of Home Assistant. With the list, as the gateway uses it,
they go out as written. -
spaCy reads a text once. A chat sends its whole history with every call, and an agent its
system prompt and the descriptions of its tools too: masking an agent's call of 105 messages
took 28 seconds, 20 of them in spaCy reading some two thousand texts it had read in the call
before.NerGuardnow remembers what it found in up to 20 000 texts (cache_texts, 0 turns it
off), under each text's SHA-256 digest, never the text itself; the same call took 13 seconds. -
The search for known values is prepared once per set of values. Every text of a call is
masked and checked against the whole mapping, and the search for its few hundred values, with
each value's keys, was built anew for each of some 2,700 texts. It is now kept for the latest
values, and the gateway drops it after each call (polanon._matching.forget), so a call's
values stay in memory no longer than the call. Masking the agent's call of 105 messages now
takes 8 seconds and its last check 1 instead of 13 and 3.5.
Fixed
- The gateway's last check no longer refuses an agent's history over names from code. With
Home Assistant's tools, a word of a tool's JSON result the masking took for a name, "service"
or "text", also stood in a JSON key, a tool call's id, a type such as "reasoning.text", a
tool's name or the parameters a tool requires, which the masking leaves as they are, and every
call was refused. The check now reads the payload as the masking does: a JSON text value by
value, and those names from code, written without a space, not at all. Every other text, every
value and a key with a space in it are searched as before. - A value found while earlier messages are masked once more is masked in all of them. A
message masked before the mapping last grew is masked again with the whole mapping, and doing
so can make the mapping grow once more: in an agent's history of 115 messages a piece left over
from a losing detection became a new value then, a message re-masked just before kept it, and
the gateway's last check refused the call.MaskedTurn.mask_messagesand the gateway's
Responses API masking now repeat the pass until the mapping stops growing, at most four times
(polanon.turn.MASKING_PASSES). - A number, a quote, a literal of code and a small English word are no longer names. In an
agent's tool results the Polish model tagged "2026" as a company,File "of a Python
traceback and Python's "None" as people, and "has" and "before" on lines it read as Polish. A
span of digits alone, one with a double quote, a literal such as "None", "True" or "null", and
one English function word in lowercase are now dropped on a line of any language; the rules
still find the numbers that matter. The benchmark is unchanged. - A model's English with a Polish quote in it is read as English. An agent's model reasons in
English and quotes the user in Polish, and one Polish letter in the quote made the whole line
Polish, so "The user asks" and "the model" went out as tokens. A line is now Polish by its
Polish words, a word with a Polish letter counting as one, and what stands in quotation marks
is left out of the count. In a text that is plainly English as a whole, a model's reasoning or a
manual, a line too short to show a language, such as "Possible causes" or "Max", is read as
English too; in a Polish text it stays as it was. The benchmark is unchanged. - A piece of a longer word, a Python traceback and more small English words are no longer
names: "service" ofha_call_servicewas a company, "File" ofFile "/srv/app.py", line 12a
place, and "while" in a Polish sentence a place too. A span glued to a letter, a digit or a "_"
is dropped, but not after a line break written out as\n, as an escaped JSON string has it,
soKontakt:\nAnna Lindqviststill names her. The benchmark is unchanged. - A span that opens with an English small word is no name on a line that shows no language
either: a tool's description "That's the limit" gave the person "That's", masked 64 times in
one agent's call once it was known; "This Week" was a company. The benchmark is unchanged. - A place or a company of one word in lowercase is no longer masked, on any line. Polish
writes both with a capital letter, and in lowercase the Polish model found English words of
the manuals in an agent's tool results instead: "text", "context" and "service", masked 54, 25
and 38 times in one call once known. The price is a place or a company typed in lowercase in a
chat, "jadę do gdyni" or "pracuję w orlenie", which now goes out as written; a person in
lowercase is still masked. The benchmark masks what it did, with one stray span fewer. - An English small word ends a name on every line. A span that opens with one is dropped on
an English or a Polish line too: "That's" stood after a dash in mid-sentence on an English line
and was masked 76 times in one agent's call once known, and a Polish manual quoting English gave
"Before" and "the model". A span that holds one in lowercase is cut there, and a piece stays
when it has two capitalised words or a Polish first name: "Maria Lindqvist at Northwind
Traders" gives both names, "tokens and passwords" none. "Will" and "May" are first names too,
so a span they open with a capital letter is read as before...
v0.3.5
A new way back for names and fixes for agents, and nothing that worked before stops working: a
patch release, as CONTRIBUTING.md describes. The Polish note of token_instructions now asks the
model to write a person's case after a token; a token without one restores as before. Where the
Polish model read English as names it now finds less, so an agent's system prompt comes out with
fewer tokens, and the gateway masks the prose of tools it used to send as written.
Added
- A person comes back in the case the model asks for. A token stood for the form the text
wrote, so "wniosek pracownika Jana Kowalskiego" gave an answer "Szanowny Panie Jana
Kowalskiego". The model may now write the case after a colon,<OSOBA_1:wołacz>, and the name
comes back declined: "Szanowny Panie Janie Kowalski". The Polish note oftoken_instructions
asks for it, with<OSOBA_1:mianownik>in a tool's arguments, and every restore understands
it:MaskedTurn.restoreandrestore_data,Anonymizer.restore,StreamRestorerwhen a
token and its case arrive cut apart, the Haystack components and both APIs of the gateway, a
tool's arguments staying valid JSON. Declined are names of one or two words the rules read for
sure, a listed first name and a surname such as Kowalski, Nowak, Bednarek, Mickiewicz or Duda,
written in capitals, in lowercase or without Polish letters as the value was. A surname alone
that may be a man's or a woman's, a foreign name, a company and a place come back as written,
and the case is dropped. A token without a case restores as before, and an invented one with
a case is reported byunknown_tokensandsweepas the token it declines.
Fixed
- Ordinary English words are no longer masked as names by the Polish model. Reading an agent's
English system prompt through the gateway, it tagged "result", "NOT", "Safety" and "the user's
own" as companies, people and places: 89 tokens in one call, a prompt the model could hardly
read, and, since the same words stood in the tool descriptions, a call the gateway's last check
refused. On a plainly English line, one with English words, few Polish ones and no Polish
letter, a span the Polish model finds is kept only as a capitalised name in mid-sentence, such as
"Maria Lindqvist" in "ask Maria Lindqvist for it". A name that opens an
English sentence is left to the rules, which find a Polish first name anywhere. Polish lines,
in lowercase or without diacritics, and an English spaCy model are untouched; the benchmark is
unchanged. - Names from code and lines with no language are no longer taken for people. An agent's
tools have parameters such asreplace_all, which the Polish model tagged as a person: the tool
schema holds the name as a key, which cannot be masked, so the gateway refused every call. A
span with a backtick, or with a "_" and no Polish first name, is now code on any line, so
"jan_kowalski" stays masked. On a line too short to show a language, such as "Max results" or
"Skills for", a span with a word in lowercase is kept only when it holds a Polish first name.
A line counts as Polish when its Polish words outnumber its English ones, and "ten", "pan",
"pod" and "ale", English words too, no longer count; after a colon on an English line, a name
is kept: "Contact: John Smith". The benchmark masks what it did, with one stray span fewer. - The gateway masks the prose of tools and answer formats. Tool descriptions went out as
written, so a value they held reached the provider, and a value the messages held was found
there by the last check, which refused the call: an agent with 18 tools could not get a single
call through. Now the descriptions and titles of the tools, the legacyfunctions, a JSON-schema
response_formatand the Responses API'stext.formatare masked, and so are the values of
enum,const,defaultandexamples; a model answering with such a token gets it back in
the arguments. Tool names, types and the names of parameters stay, so the provider still checks
answers against the schema. A value found in a tool is masked in the messages, and one found in
the messages in every tool.
v0.3.4
A new extra and fixes, and nothing that worked before stops working: a patch release, as
CONTRIBUTING.md describes. The gateway comes with pip install "polanon[gateway]"; pip install polanon alone is unchanged and still needs nothing beyond the standard library. Where detection
missed or mislabelled a value it now differs, so a text may come out with other tokens: a first
name alone gives way to the language model's place or company, and a stray word left over where
two detections overlapped stays readable.
Added
-
A masking gateway for tools that speak the OpenAI API, run by whoever needs it on their own
machine:pip install "polanon[gateway]", thenpolanon-gateway --upstream https://api.openai.com/v1, and Hermes Agent, n8n, Open WebUI or LibreChat point their base URL at
http://127.0.0.1:8787/v1. It serves/v1/chat/completionsand the Responses API,
/v1/responses, streamed or not and with tool calls,/v1/embeddingsand/v1/models, and
refuses every other endpoint, images and audio. With the Responses API the whole conversation
comes with every call, as Hermes Agent sends it withstore: false, and encrypted reasoning
passes as it is; a call that leaves the conversation at the provider is refused. Every call
is masked, checked once more for any original value before it leaves, and restored on the way
back, a tool call's arguments as valid JSON. It keeps nothing on disk, listens on 127.0.0.1 and
needs a key every client sends to listen anywhere else, takes keys from the environment only, and
logs counts, never values. The library still imports no web framework: the gateway is a plain
ASGI application with httpx, served by uvicorn. -
A twelfth benchmark document, a hospital discharge summary, health data being the most
sensitive kind (#4): a patient with her PESEL, birth date, address and phone, her daughter, two
doctors behind their titles and the hospital's address. Every value is made up, and the patient
was born in 1847, so her PESEL belongs to nobody. The rules alone mask all of it but its two
places; withpl_core_news_mdall 18 values are masked, and the ward's name is read as a company.
The corpus now has 187 annotated values: 161 masked with the rules alone (86%) and 184 with the
model (98%), with 8 over-masked spans, the ward one of them.
Fixed
- A model's reasoning sent back with its answer is masked by
MaskedTurn.mask_messages:
OpenRouter returns the reasoning next to the answer, inreasoningandreasoning_details, and
a client that kept the answer sent it back with the history, restored, in the clear. Now
reasoning,reasoning_contentand the text and summary of every item ofreasoning_details
are masked, and encrypted reasoning passes as it is. Found by the gateway's last check, which
refused the call, in a test against OpenRouter. - A surname on the next line or in the next cell is masked without the language model:
"z panem Janem\nKowalskim" in a hard-wrapped e-mail, "Jan;Kowalski" in a CSV export and
"| Jan | Kowalski |" in a Markdown table kept the surname in clear (#15). On the next line it
joins the first name; in the next cell it is masked on its own, so the row keeps its columns.
Across a line or a cell the word must read as a surname, so "Imię: Jan\nNazwisko:", "Jan;Anna",
"Jan;Kraków" and "Kamil\nNowak Transport" under a signature stay apart. - A place spelled like a first name keeps the language model's label: "Wakacje na Kubie" went
to the model as a person, since a first name outranked the model wherever the two overlapped
(#16). A first name alone, with no surname, initial or form of address, now scores 0.8, below
the model's 0.85, and is still masked as a person without the model. "Asia" alone is the
continent, as "Maja" is the month; "Asia Kowalska" and "pani Asia" stay people. - A piece left over from a losing detection is treated alike in every turn (#20). Masked in
place in the first turn, it was a known value in the next, so the bare "NIP" that an address
rule left behind went to the model as an address in every sentence. A piece with no digit of a
type whose values always carry one, or with fewer than three letters and digits, now stays as
written; any other piece, such as the name of a loose span around an e-mail address, is masked
wherever it stands from the first turn on. - An academic title alone stays readable: the Polish model tagged the "n." of "dr n. med." as a
person. A span the model finds made only of titles such as dr, n., med., hab., prof., mgr, inż.
and lek. is dropped, as form labels are; "dr n. med. Ewa Zawadzka" keeps the name masked.
v0.3.3
Fixes only, and nothing that worked before stops working: a patch release, as CONTRIBUTING.md
describes. More is masked than under 0.3.2, so a text may carry more tokens, and with
WELL_KNOWN_NAMES the names of a few public institutions stay readable.
Fixed
- A token-shaped number longer than six digits in the input, such as
<OSOBA_50810112354>,
no longer sets the number of every later token, and one of thousands of digits no longer stops
masking with aValueError. A new token still skips every token the mapping holds, so
<OSOBA_999999>followed by a new person never hands out one token twice. Contributed by Sonike
in #24. - A spaCy model named as pip writes it, such as
pl-core-news-md, loads, and a model directory
withoutmeta.jsonandconfig.cfgis reported when the guards are built, without a download
command that could not work. An installed model still wins over a file or folder of the same
name in the working directory. Contributed by Sonike in
#27. - A password named by a word or two before its colon, "Hasło administratora: Zima2025!Lod",
"hasło użytkownika admin:", "Hasło do panelu klienta:", is masked: only "do", "dla" or "od" and
one word could stand there, so the value went out in clear. With words in between, the value
must look like a password, so "Hasło administratora zostało zmienione." and a date after "Hasło
ważne do:" stay as they are. - A PESEL with a typo right after its label is masked: "PESEL: 15501416181", with a wrong
check digit, and "pesel 78059938421", with day 99, went out in clear, since the checksum was
checked before the label. Without the label the checksum still decides, so an order number of
eleven digits stays as it is. - A feminine surname in another case is masked after the full name: once "Natalia Bielska"
was masked, "Bielskiej" and "Bielską" went out in clear, in the same text and in the next turn,
since no case of a surname in -ska, -cka or -dzka but the nominative begins with it. In
lowercase, "w bielskiej firmie" stays an adjective. - A street without "ul." and without a postcode is masked after an address label, "Adres:
Japczaka 17", "zam. Długa 5", "mieszka przy Długiej 5", and after "z", "na", "przy" or "do"
with a town right after the number, "z Długiej 17 w Białymstoku". Both went out in clear.
"Windows 11 w Polsce", "na Windows 11 w firmie" and "Excel 365 w biurze" stay as they are. - An address is found in linear time in a long run of capitalised words: each word started a
street that ran to the end of the run, so 5,000 words such as "Ulica Ulica ..." or "I I ..."
took several seconds, and ten times as many a hundred times longer. A street's name now takes
at most eight words and three titles, more than any Polish street needs. - ZUS, Płatnik and the institutions staff name every day are on
WELL_KNOWN_NAMES: with the
allowlist, "Jak to rozliczyć w ZUS?" and "zna program Płatnik" went out as<FIRMA_1>, since
the language model reads them as organisations. Added: ZUS, PUE ZUS, eZUS, Płatnik, KRUS, NFZ,
GUS, KSeF, ePUAP, CEIDG and PFRON. The list releases names only, so no identifier is unmasked,
and "pani Płatnik" stays a person.
v0.3.2
New features and fixes, and nothing that worked before stops working: a patch release, as
CONTRIBUTING.md describes. Without a key, tokens are numbered as before.
Added
- Stable pseudonyms for search indexes and RAG:
Anonymizer(guards, pseudonyms=Pseudonyms(key))
computes every new token with HMAC-SHA256 under a secret key of at least 32 bytes, such as
<OSOBA_482193047556>, instead of numbering it per conversation. The same value gets the same
token in every document, process and question, with no shared state. A person's name is read in
the nominative where its first name and surname agree, so "Jan Kowalski", "Jana Kowalskiego" and
"z Janem Kowalskim" share a token, and an identifier is one value with or without its separators
and country prefix. Tokens keep the<LABEL_number>shape, with 12 digits by default. The key is
never shown, pickled or copied;key_idandsettingssay what an index was built with, and
scope=keeps two indexes under one key apart. A collision gets four digits more and is recorded
inPseudonyms(collisions=...). - Masking for an index:
anonymizer.index_chunks(passages, store)masks the passages of a
document with one mapping and puts what each token hides into aMappingStore, a two-method
protocol over the application's own encrypted table (InMemoryMappingStorefor tests and
examples), with a person's name in the nominative and a fingerprint to tell a collision.
anonymizer.mask_query(question, store)gives a question the documents' tokens, also for a
surname written alone, which no rule reads and the store keeps for every indexed person.mask_chunks,stored_mapping
andtokens_inmask without a store and restore found passages and answers, also while they
stream; each passage also comes with its mapping as written, to restore it exactly. - Haystack 3 components, with
pip install "polanon[haystack]":PolanonDocumentMaskermasks
documents before the splitter and the embedder of an indexing pipeline,PolanonQueryMasker
gives a question the documents' tokens,PolanonRestorerrestores the replies and the documents
found, andPolanonStreamRestorerrestores a streamed answer as a streaming callback. Nothing
that names a value reaches the document store: not the content, notmeta, and noidthat is
a plain hash of an original text. What the tokens stand for goes to the application's mapping
store only. The key is a HaystackSecret, read fromPOLANON_KEYand saved by name only.
examples/haystack_rag.pyruns offline;examples/haystack_openai.pyuses OpenAI and is paid.
Fixed
- A token the mapping holds is left whole when masked text is masked again, such as a passage
from an index or history sent back. The digits of<OSOBA_50810112354>read as a PESEL, a
language model readOSOBA_8as a place, a PIN label took the token after it for the PIN, and a
known company called "Firma" was found inside<FIRMA_1>; each corrupted the token so restore no
longer gave the value back. A placeholder that stands for itself still shelters nothing. - Titles stay out of a foreign name after a label. "z. Hd. Frau Wiśniewska" made "Frau" the
first name, so every later "Frau" in the letter was masked as her, and "Attn: Mr Oliver Bennett"
got a token apart from the "Oliver Bennett" of its sign-off. With two titles, as in "Frau Prof.
Anna Schmidt", the second one was masked as the person wherever it stood.
v0.3.1
Fixed
- A known value that only partly overlaps a new detection is masked whole: "Nowak z
Kowalskich" known from an earlier turn met the name rule's "Anna Nowak", and "z Kowalskich" went
out in the clear. The same holds for two known values that overlap. - Two findings of the judge that overlap are both masked. A small model that quoted "Ngozi
Okonkwo-Brzęczyszczykiewicz" and "Okonkwo-Brzęczyszczykiewicz z kancelarii ..." had its first
name sent, whilejudge_hitsreported success. - A PIN that equals a year stays out of two dates joined by "-" or "/", a span of years such as
2024/2025and a date ending in2025r.: in01.01.<SEKRET_1>-31.12.<SEKRET_1>the dates around
the token gave it away. - A known short number in one long line of numbers, such as a minified JSON array, no longer
makes masking quadratic: 50 KB took most of a minute, and now takes under a second.
v0.3.0
Added
MaskedTurn, fromanonymizer.turn(): one object for everything a call to a model sends and
gets back. It masks the history, the question and the results of tools with one mapping, so a
value keeps one token across them, restores the arguments of tool calls and the answer, also
while it streams, and reports invented tokens. Values the caller already knows are sensitive,
such as its clients' names, go inknown=and are masked wherever they appear. When a detector
fails, everymaskmethod raisesMaskingUnavailableErrorand returns nothing half masked.
PolCRM and PolSales each kept a copy of this as aMaskerof their own.MaskedTurn.mask_messages()masks a whole history written for the Anthropic or the OpenAI
API: text,tool_useinput,tool_resultand"tool"content, the arguments oftool_calls.
Tool results and arguments written as JSON stay valid JSON, and ids and tool names are left
alone. A value first recognised in a later message is masked in the earlier ones too. A block
nothing can mask, such as an image or a model's signed reasoning, is refused, or left out with
unknown_blocks="drop".token_instructions("pl" | "en"): a note for the system prompt that tells the model what the
tokens are and to copy them exactly, the one both apps wrote out for themselves.MaskingUnavailableError, the one error an app catches to refuse sending text it could not
mask, andNerModelUnavailableError, the one of them raised when spaCy or its model is
missing.ner_available()runs the same check asdefault_guards()without loading
anything, for a settings page.
Changed
default_guards(with_ner=True)without spaCy or its model raisesNerModelUnavailableError
instead of a plainModuleNotFoundError. It is a subclass ofModuleNotFoundErrorwith the same
message, soexcept ModuleNotFoundErrorandexcept ImportErrorstill catch it. Through
MaskingUnavailableErrorit is also aRuntimeError, so anexcept RuntimeErroraround
default_guards()now catches it too, and a check of the exact type, such as
type(exc) is ModuleNotFoundError, no longer matches. Nothing else in the 0.2.2 API changed:
every name and signature is still there, and every benchmark document masks to the same text,
tokens and spans, with the language model and without it.
v0.2.2
Found by masking synthetic CVs, profiles written as JSON and lists of repositories with the rules
alone.
Added
- Links to a person's profile are a new built-in type,
LINK(tokens<LINK_1>): LinkedIn
(linkedin.com/in/...), GitHub, GitLab, Bitbucket, Codeberg, GitHub and GitLab Pages, Stack
Overflow, Kaggle, Behance, Dribbble, Medium, Instagram, Facebook, X, YouTube and a few more.
Only the link up to the handle is masked, so<LINK_1>/weather-appstill names the project.
A LinkedIn link carried the person's name, and a GitHub login reappeared in every repository.
The paths of a site's own pages (github.com/features,facebook.com/groups/...,
youtube.com/watch) stay readable. A guard that declaresLINKas its own type keeps working. - The surname of a masked person and the handle of a masked profile link are masked on their
own wherever they appear in the text: "Kowalski" alone after "Jan Kowalski", and
"login": "jkowalski",@jkowalskiandjkowalski.github.ioaftergithub.com/jkowalski, went
out in the clear. A lone surname in lowercase stays an ordinary word, as any known one-word name
does; a name only the language model found gives its surname only next to a first name the rules
know, and the handles of well-known brands (github.com/microsoft/...) stay readable. - Popular first names missing from the list: Oliwier, Cyprian, Maurycy, Celina, Ksenia,
Lilianna, Matylda, Melania and Otylia, with their surnames: "Oliwier Nowak" went out whole.
Fixed
- An e-mail address with its "@" written out is masked whole:
jan.kowalski [at] example.com,
(at),(małpa), also with[dot]. Only "jan" was masked, as a first name. - A street written without "ul." is masked when a postcode follows it: in
Długa 5 m. 4, 00-238 Warszawaonly the postcode was. A word and a number with no postcode
after them, such asWindows 11, stay readable. - A birth date written in English is masked after its label:
Date of birth: 15 March 1985,
DOB: March 15, 1985andborn 3rd May 1985went out whole. WithDATEturned on, such dates
are dates too.
v0.2.1
Added
polanon --versionprints the installed version. Contributed by barkya28 in
#29.- An eleventh benchmark document, a monthly roster of surnames and periods, which keeps names
beside ordinary numbers masked without a language model. The corpus now has 169 annotated
values: the rules alone fully mask 86% of them and the rules withpl_core_news_md98%.
Fixed
- E-mail addresses with Polish letters are masked whole:
zofia.żak@example.comwas masked
as a name and half an address, and an address on an internationalised domain not at all.
Contributed by ShahadatTest in #14. - A phone number after "tel." is labelled a phone, not a REGON: one nine-digit number in ten
passes the REGON checksum, and the model was told it was a company's register number.
Contributed by ShahadatTest in #22; "kom."
counts as a phone label as well, and "hotel" no longer does. - A phone number written with a bare 48 in front is masked whole:
48123456789, as SMS
gateways and CRM exports write it, went out unmasked, and in48 123 456 789the country code
stayed in the clear. A valid PESEL that starts with 48 keeps its type, and the minutes of a time
such as10:48are not pulled into the phone. Contributed by Aaqibhafeezkhan in
#26. - A restored name with a case ending is not reported as new: "dla <OSOBA_1>a" restored as
"Jan Kowalskia" was flagged by the output sweep as personal data the model made up. A person,
company or place known from the conversation now covers its Polish case endings, while a longer
different name, letters glued on the left, and other types such as an address with a letter
added are still reported. Contributed by bfs2021 in
#25. - A month spelled like a first name is left alone in a date written with dashes, slashes or
dots: "Jan" in15-Jan-2024,2024/Jan/15or15.Jan.2024was masked as a person. The same
separator has to stand on both sides, so a name beside an ordinary number, as in
Nowak Anna 03/2024, is still masked. Contributed by ShahadatTest in
#23.
v0.2.0
Before this release a pre-release review went through the library looking for ways personal data
could get past it. Everything under Fixed is what it found; each fix comes with a test that
fails without it.
Added
--file PATHon the command line. polanon reads the file's bytes itself and tells UTF-8,
UTF-16 and cp1250 apart, so Windows PowerShell 5.1 cannot garble Polish letters on the way in.
Contributed by jay23606 in #5.--mapping PATHwrites the mapping to a file as UTF-8 JSON, created readable by its owner
only, instead of to stderr, which Windows PowerShell's2>wraps in error records.
Fixed
- A UTF-16 file went out unmasked. Windows PowerShell 5.1 writes UTF-16 with
>; decoded
as cp1250 it was full of NUL characters that hid every value from the guards, and the command
line printed it unmasked with exit status 0. Input is now decoded by its byte order mark,
UTF-16 is recognised without one, a UTF-8 file with one cp1250 line is decoded line by line,
and bytes that are none of these are refused with exit status 2. - The second of two IBANs went out whole when they were grouped and written one per line or
joined by "oraz". - A double surname was read by its last part only, so "mają kowalską-nowak" went out where
"mają kowalską" was masked. - A name on the cut of a long single-line text was missed by spaCy. Text over 100,000
characters with no line break was cut at exactly that length, through the middle of a name. - A card number after the year of a date, "15.03.2024 4111 1111 1111 1111", went out whole.
- Settings secrets named
SECRET_KEY,APP_KEYorDB_PASSwere not recognised, and a
labelled password containing "," or ";" was cut short with the rest sent. - A birth date with the year abbreviation glued on, "ur. 15.03.1985r.", went out whole.
- An address after a street word in an oblique case, "przy ulicy Długiej 5", went out
whole; KRS is now also found after "pod numerem" and "No.". - A denylist term copied from Word, a PDF or macOS (an en dash, decomposed letters, a soft
hyphen) matched nothing, not even the same text. - A judge error carried the HTTP error it came from, so a logged traceback could show a
proxy password or an API key. Judge replies with code fences inside their JSON, a brace in a
sentence before it, or a quote that differs from the text only in its whitespace are now read
instead of dropped, and a reply of a few thousand spaces no longer freezes the event loop. - "Pana przesyłka została nadana" no longer masks the parcel as a person.
- Masked CRLF documents no longer come out as CR CR LF on Windows, and two rules that took
quadratic time on long lines without spaces now run in linear time.
Changed
- The README no longer recommends
Get-Content -Encoding UTF8 file | polanonfor Windows
PowerShell 5.1: that pipe turns Polish letters into?. It recommends--fileand
--mappinginstead.