Skip to content

Releases

v0.3.6

Choose a tag to compare

@github-actions github-actions released this 15 Nov 14:50

Two modes for the language model, a benchmark of an AI agent's calls, and far fewer words masked
for nothing; nothing that worked before stops working, so this is a patch release, as
CONTRIBUTING.md describes. The spaCy guard now leaves out what it reads as a name in English
prose, code and identifiers, a place typed in lowercase among them, so an agent's prompt, tools
and results stay readable: in the benchmark's calls 5 of 307 token uses hide nothing, against 110
of 410 in 0.3.5. mode="paranoid", polanon-gateway --paranoid and polanon --ner --paranoid
mask all of it instead. A birthday under a JSON key and a login made of a known surname are now
masked, and the gateway can send images and files on unmasked when told to.

Added

  • polanon --ner --paranoid masks with the spaCy guard in mode="paranoid". Without --ner
    the flag is refused, since it changes nothing without the model.

  • polanon-gateway --paranoid masks with the spaCy guard in mode="paranoid", and the banner
    says so. Without the model the flag changes nothing, and the banner says that too.

  • mode="paranoid" masks every name the spaCy model finds. The filters that keep an AI
    agent's English, code and identifiers readable let a little through by design: a name opening
    an English sentence, a word in inline code, a place typed in lowercase. default_guards(mode= "paranoid") and NerGuard(mode="paranoid") keep every span with a letter in it instead. On
    the agent's calls of the benchmark only what no guard finds leaks, 1 of 243 marked
    occurrences against 3 in the default mode="normal", and 386 of 696 token uses hide nothing,
    against 5 of 307. On the sample
    documents it masks the same 187 values, with 8 over-masked spans instead of 6. The rules are
    the same in both modes. benchmarks/run.py and benchmarks/run_agent.py take --paranoid.

  • A benchmark of an AI agent's calls, and of what comes back. benchmarks/agent_corpus.py
    holds eight invented calls of a personal assistant as its client sends them to a model, seven
    to Chat Completions and one to the Responses API: an English system prompt with a Polish memory
    of its user, sixteen tool definitions, and the JSON its tools return from a smart home, a web
    search, a shell, a mailbox, a contact book and a file, with code, a traceback, a system log,
    keys, a lease with two PESEL numbers, and mail in German and Ukrainian among it. The 103 values
    that must not reach the model are marked where they are written. benchmarks/run_agent.py
    masks each call as the gateway does and counts what still reaches the model and the tokens
    that hide nothing, with their uses. It then restores the model's answer to each call, written
    with the call's tokens and the cases the Polish note asks for, as text, as a whole response and
    as a stream cut at every place, and scores each line and argument against what a reader should
    get back. With pl_core_news_md, 3 of 243 occurrences reach the model and 5 of 307 uses hide
    nothing, against 5 and 110 of 410 in 0.3.5; 39 of 40 lines come back exact and none wrong.
    tests/test_agent_benchmark.py holds all of it as bounds, and CI prints the tables.

  • The gateway can send what it cannot mask, when told to. polanon-gateway --pass-unsupported sends an image, a file or another part it cannot mask to the provider as it
    is, unmasked, and masks the text around it, for an operator who wants the model to see a photo
    or a document sent to an agent. The banner says so. MaskedTurn(unknown_blocks="keep") and
    mask_responses_request(pass_unsupported=True) do the same in code. Without the flag such a
    call is still refused, and --drop-unsupported still leaves the content out; the two flags
    exclude each other.

Changed

  • WELL_KNOWN_NAMES holds hardware an agent names: Dell, Lenovo, Samsung, Roborock and HACS,
    the store of Home Assistant's add-ons. In the agent's calls of the benchmark "Dell" was masked
    as a company eight times in one call, "Roborock" as a person and "HACS" as a place.

  • WELL_KNOWN_NAMES holds the platforms an AI agent names: OpenRouter, DeepSeek, Hugging
    Face, Ollama, LM Studio, Claude Code, Hermes Agent, Nous Research, Home Assistant and its "HA",
    GitHub, GitLab, Docker, Linux and Ubuntu. An agent driving a smart home named "HA" 117 times in
    one call, and the Polish model took it for a place each time: the model saw a token in every
    tool description and result in place of Home Assistant. With the list, as the gateway uses it,
    they go out as written.

  • spaCy reads a text once. A chat sends its whole history with every call, and an agent its
    system prompt and the descriptions of its tools too: masking an agent's call of 105 messages
    took 28 seconds, 20 of them in spaCy reading some two thousand texts it had read in the call
    before. NerGuard now remembers what it found in up to 20 000 texts (cache_texts, 0 turns it
    off), under each text's SHA-256 digest, never the text itself; the same call took 13 seconds.

  • The search for known values is prepared once per set of values. Every text of a call is
    masked and checked against the whole mapping, and the search for its few hundred values, with
    each value's keys, was built anew for each of some 2,700 texts. It is now kept for the latest
    values, and the gateway drops it after each call (polanon._matching.forget), so a call's
    values stay in memory no longer than the call. Masking the agent's call of 105 messages now
    takes 8 seconds and its last check 1 instead of 13 and 3.5.

Fixed

  • The gateway's last check no longer refuses an agent's history over names from code. With
    Home Assistant's tools, a word of a tool's JSON result the masking took for a name, "service"
    or "text", also stood in a JSON key, a tool call's id, a type such as "reasoning.text", a
    tool's name or the parameters a tool requires, which the masking leaves as they are, and every
    call was refused. The check now reads the payload as the masking does: a JSON text value by
    value, and those names from code, written without a space, not at all. Every other text, every
    value and a key with a space in it are searched as before.
  • A value found while earlier messages are masked once more is masked in all of them. A
    message masked before the mapping last grew is masked again with the whole mapping, and doing
    so can make the mapping grow once more: in an agent's history of 115 messages a piece left over
    from a losing detection became a new value then, a message re-masked just before kept it, and
    the gateway's last check refused the call. MaskedTurn.mask_messages and the gateway's
    Responses API masking now repeat the pass until the mapping stops growing, at most four times
    (polanon.turn.MASKING_PASSES).
  • A number, a quote, a literal of code and a small English word are no longer names. In an
    agent's tool results the Polish model tagged "2026" as a company, File " of a Python
    traceback and Python's "None" as people, and "has" and "before" on lines it read as Polish. A
    span of digits alone, one with a double quote, a literal such as "None", "True" or "null", and
    one English function word in lowercase are now dropped on a line of any language; the rules
    still find the numbers that matter. The benchmark is unchanged.
  • A model's English with a Polish quote in it is read as English. An agent's model reasons in
    English and quotes the user in Polish, and one Polish letter in the quote made the whole line
    Polish, so "The user asks" and "the model" went out as tokens. A line is now Polish by its
    Polish words, a word with a Polish letter counting as one, and what stands in quotation marks
    is left out of the count. In a text that is plainly English as a whole, a model's reasoning or a
    manual, a line too short to show a language, such as "Possible causes" or "Max", is read as
    English too; in a Polish text it stays as it was. The benchmark is unchanged.
  • A piece of a longer word, a Python traceback and more small English words are no longer
    names: "service" of ha_call_service was a company, "File" of File "/srv/app.py", line 12 a
    place, and "while" in a Polish sentence a place too. A span glued to a letter, a digit or a "_"
    is dropped, but not after a line break written out as \n, as an escaped JSON string has it,
    so Kontakt:\nAnna Lindqvist still names her. The benchmark is unchanged.
  • A span that opens with an English small word is no name on a line that shows no language
    either: a tool's description "That's the limit" gave the person "That's", masked 64 times in
    one agent's call once it was known; "This Week" was a company. The benchmark is unchanged.
  • A place or a company of one word in lowercase is no longer masked, on any line. Polish
    writes both with a capital letter, and in lowercase the Polish model found English words of
    the manuals in an agent's tool results instead: "text", "context" and "service", masked 54, 25
    and 38 times in one call once known. The price is a place or a company typed in lowercase in a
    chat, "jadę do gdyni" or "pracuję w orlenie", which now goes out as written; a person in
    lowercase is still masked. The benchmark masks what it did, with one stray span fewer.
  • An English small word ends a name on every line. A span that opens with one is dropped on
    an English or a Polish line too: "That's" stood after a dash in mid-sentence on an English line
    and was masked 76 times in one agent's call once known, and a Polish manual quoting English gave
    "Before" and "the model". A span that holds one in lowercase is cut there, and a piece stays
    when it has two capitalised words or a Polish first name: "Maria Lindqvist at Northwind
    Traders" gives both names, "tokens and passwords" none. "Will" and "May" are first names too,
    so a span they open with a capital letter is read as before...
Read more

v0.3.5

Choose a tag to compare

@github-actions github-actions released this 14 Nov 19:03

A new way back for names and fixes for agents, and nothing that worked before stops working: a
patch release, as CONTRIBUTING.md describes. The Polish note of token_instructions now asks the
model to write a person's case after a token; a token without one restores as before. Where the
Polish model read English as names it now finds less, so an agent's system prompt comes out with
fewer tokens, and the gateway masks the prose of tools it used to send as written.

Added

  • A person comes back in the case the model asks for. A token stood for the form the text
    wrote, so "wniosek pracownika Jana Kowalskiego" gave an answer "Szanowny Panie Jana
    Kowalskiego". The model may now write the case after a colon, <OSOBA_1:wołacz>, and the name
    comes back declined: "Szanowny Panie Janie Kowalski". The Polish note of token_instructions
    asks for it, with <OSOBA_1:mianownik> in a tool's arguments, and every restore understands
    it: MaskedTurn.restore and restore_data, Anonymizer.restore, StreamRestorer when a
    token and its case arrive cut apart, the Haystack components and both APIs of the gateway, a
    tool's arguments staying valid JSON. Declined are names of one or two words the rules read for
    sure, a listed first name and a surname such as Kowalski, Nowak, Bednarek, Mickiewicz or Duda,
    written in capitals, in lowercase or without Polish letters as the value was. A surname alone
    that may be a man's or a woman's, a foreign name, a company and a place come back as written,
    and the case is dropped. A token without a case restores as before, and an invented one with
    a case is reported by unknown_tokens and sweep as the token it declines.

Fixed

  • Ordinary English words are no longer masked as names by the Polish model. Reading an agent's
    English system prompt through the gateway, it tagged "result", "NOT", "Safety" and "the user's
    own" as companies, people and places: 89 tokens in one call, a prompt the model could hardly
    read, and, since the same words stood in the tool descriptions, a call the gateway's last check
    refused. On a plainly English line, one with English words, few Polish ones and no Polish
    letter, a span the Polish model finds is kept only as a capitalised name in mid-sentence, such as
    "Maria Lindqvist" in "ask Maria Lindqvist for it". A name that opens an
    English sentence is left to the rules, which find a Polish first name anywhere. Polish lines,
    in lowercase or without diacritics, and an English spaCy model are untouched; the benchmark is
    unchanged.
  • Names from code and lines with no language are no longer taken for people. An agent's
    tools have parameters such as replace_all, which the Polish model tagged as a person: the tool
    schema holds the name as a key, which cannot be masked, so the gateway refused every call. A
    span with a backtick, or with a "_" and no Polish first name, is now code on any line, so
    "jan_kowalski" stays masked. On a line too short to show a language, such as "Max results" or
    "Skills for", a span with a word in lowercase is kept only when it holds a Polish first name.
    A line counts as Polish when its Polish words outnumber its English ones, and "ten", "pan",
    "pod" and "ale", English words too, no longer count; after a colon on an English line, a name
    is kept: "Contact: John Smith". The benchmark masks what it did, with one stray span fewer.
  • The gateway masks the prose of tools and answer formats. Tool descriptions went out as
    written, so a value they held reached the provider, and a value the messages held was found
    there by the last check, which refused the call: an agent with 18 tools could not get a single
    call through. Now the descriptions and titles of the tools, the legacy functions, a JSON-schema
    response_format and the Responses API's text.format are masked, and so are the values of
    enum, const, default and examples; a model answering with such a token gets it back in
    the arguments. Tool names, types and the names of parameters stay, so the provider still checks
    answers against the schema. A value found in a tool is masked in the messages, and one found in
    the messages in every tool.

v0.3.4

Choose a tag to compare

@github-actions github-actions released this 14 Nov 01:20

A new extra and fixes, and nothing that worked before stops working: a patch release, as
CONTRIBUTING.md describes. The gateway comes with pip install "polanon[gateway]"; pip install polanon alone is unchanged and still needs nothing beyond the standard library. Where detection
missed or mislabelled a value it now differs, so a text may come out with other tokens: a first
name alone gives way to the language model's place or company, and a stray word left over where
two detections overlapped stays readable.

Added

  • A masking gateway for tools that speak the OpenAI API, run by whoever needs it on their own
    machine: pip install "polanon[gateway]", then polanon-gateway --upstream https://api.openai.com/v1, and Hermes Agent, n8n, Open WebUI or LibreChat point their base URL at
    http://127.0.0.1:8787/v1. It serves /v1/chat/completions and the Responses API,
    /v1/responses, streamed or not and with tool calls, /v1/embeddings and /v1/models, and
    refuses every other endpoint, images and audio. With the Responses API the whole conversation
    comes with every call, as Hermes Agent sends it with store: false, and encrypted reasoning
    passes as it is; a call that leaves the conversation at the provider is refused. Every call
    is masked, checked once more for any original value before it leaves, and restored on the way
    back, a tool call's arguments as valid JSON. It keeps nothing on disk, listens on 127.0.0.1 and
    needs a key every client sends to listen anywhere else, takes keys from the environment only, and
    logs counts, never values. The library still imports no web framework: the gateway is a plain
    ASGI application with httpx, served by uvicorn.

  • A twelfth benchmark document, a hospital discharge summary, health data being the most
    sensitive kind (#4): a patient with her PESEL, birth date, address and phone, her daughter, two
    doctors behind their titles and the hospital's address. Every value is made up, and the patient
    was born in 1847, so her PESEL belongs to nobody. The rules alone mask all of it but its two
    places; with pl_core_news_md all 18 values are masked, and the ward's name is read as a company.
    The corpus now has 187 annotated values: 161 masked with the rules alone (86%) and 184 with the
    model (98%), with 8 over-masked spans, the ward one of them.

Fixed

  • A model's reasoning sent back with its answer is masked by MaskedTurn.mask_messages:
    OpenRouter returns the reasoning next to the answer, in reasoning and reasoning_details, and
    a client that kept the answer sent it back with the history, restored, in the clear. Now
    reasoning, reasoning_content and the text and summary of every item of reasoning_details
    are masked, and encrypted reasoning passes as it is. Found by the gateway's last check, which
    refused the call, in a test against OpenRouter.
  • A surname on the next line or in the next cell is masked without the language model:
    "z panem Janem\nKowalskim" in a hard-wrapped e-mail, "Jan;Kowalski" in a CSV export and
    "| Jan | Kowalski |" in a Markdown table kept the surname in clear (#15). On the next line it
    joins the first name; in the next cell it is masked on its own, so the row keeps its columns.
    Across a line or a cell the word must read as a surname, so "Imię: Jan\nNazwisko:", "Jan;Anna",
    "Jan;Kraków" and "Kamil\nNowak Transport" under a signature stay apart.
  • A place spelled like a first name keeps the language model's label: "Wakacje na Kubie" went
    to the model as a person, since a first name outranked the model wherever the two overlapped
    (#16). A first name alone, with no surname, initial or form of address, now scores 0.8, below
    the model's 0.85, and is still masked as a person without the model. "Asia" alone is the
    continent, as "Maja" is the month; "Asia Kowalska" and "pani Asia" stay people.
  • A piece left over from a losing detection is treated alike in every turn (#20). Masked in
    place in the first turn, it was a known value in the next, so the bare "NIP" that an address
    rule left behind went to the model as an address in every sentence. A piece with no digit of a
    type whose values always carry one, or with fewer than three letters and digits, now stays as
    written; any other piece, such as the name of a loose span around an e-mail address, is masked
    wherever it stands from the first turn on.
  • An academic title alone stays readable: the Polish model tagged the "n." of "dr n. med." as a
    person. A span the model finds made only of titles such as dr, n., med., hab., prof., mgr, inż.
    and lek. is dropped, as form labels are; "dr n. med. Ewa Zawadzka" keeps the name masked.

v0.3.3

Choose a tag to compare

@github-actions github-actions released this 12 Nov 22:28

Fixes only, and nothing that worked before stops working: a patch release, as CONTRIBUTING.md
describes. More is masked than under 0.3.2, so a text may carry more tokens, and with
WELL_KNOWN_NAMES the names of a few public institutions stay readable.

Fixed

  • A token-shaped number longer than six digits in the input, such as <OSOBA_50810112354>,
    no longer sets the number of every later token, and one of thousands of digits no longer stops
    masking with a ValueError. A new token still skips every token the mapping holds, so
    <OSOBA_999999> followed by a new person never hands out one token twice. Contributed by Sonike
    in #24.
  • A spaCy model named as pip writes it, such as pl-core-news-md, loads, and a model directory
    without meta.json and config.cfg is reported when the guards are built, without a download
    command that could not work. An installed model still wins over a file or folder of the same
    name in the working directory. Contributed by Sonike in
    #27.
  • A password named by a word or two before its colon, "Hasło administratora: Zima2025!Lod",
    "hasło użytkownika admin:", "Hasło do panelu klienta:", is masked: only "do", "dla" or "od" and
    one word could stand there, so the value went out in clear. With words in between, the value
    must look like a password, so "Hasło administratora zostało zmienione." and a date after "Hasło
    ważne do:" stay as they are.
  • A PESEL with a typo right after its label is masked: "PESEL: 15501416181", with a wrong
    check digit, and "pesel 78059938421", with day 99, went out in clear, since the checksum was
    checked before the label. Without the label the checksum still decides, so an order number of
    eleven digits stays as it is.
  • A feminine surname in another case is masked after the full name: once "Natalia Bielska"
    was masked, "Bielskiej" and "Bielską" went out in clear, in the same text and in the next turn,
    since no case of a surname in -ska, -cka or -dzka but the nominative begins with it. In
    lowercase, "w bielskiej firmie" stays an adjective.
  • A street without "ul." and without a postcode is masked after an address label, "Adres:
    Japczaka 17", "zam. Długa 5", "mieszka przy Długiej 5", and after "z", "na", "przy" or "do"
    with a town right after the number, "z Długiej 17 w Białymstoku". Both went out in clear.
    "Windows 11 w Polsce", "na Windows 11 w firmie" and "Excel 365 w biurze" stay as they are.
  • An address is found in linear time in a long run of capitalised words: each word started a
    street that ran to the end of the run, so 5,000 words such as "Ulica Ulica ..." or "I I ..."
    took several seconds, and ten times as many a hundred times longer. A street's name now takes
    at most eight words and three titles, more than any Polish street needs.
  • ZUS, Płatnik and the institutions staff name every day are on WELL_KNOWN_NAMES: with the
    allowlist, "Jak to rozliczyć w ZUS?" and "zna program Płatnik" went out as <FIRMA_1>, since
    the language model reads them as organisations. Added: ZUS, PUE ZUS, eZUS, Płatnik, KRUS, NFZ,
    GUS, KSeF, ePUAP, CEIDG and PFRON. The list releases names only, so no identifier is unmasked,
    and "pani Płatnik" stays a person.

v0.3.2

Choose a tag to compare

@github-actions github-actions released this 12 Nov 16:05

New features and fixes, and nothing that worked before stops working: a patch release, as
CONTRIBUTING.md describes. Without a key, tokens are numbered as before.

Added

  • Stable pseudonyms for search indexes and RAG: Anonymizer(guards, pseudonyms=Pseudonyms(key))
    computes every new token with HMAC-SHA256 under a secret key of at least 32 bytes, such as
    <OSOBA_482193047556>, instead of numbering it per conversation. The same value gets the same
    token in every document, process and question, with no shared state. A person's name is read in
    the nominative where its first name and surname agree, so "Jan Kowalski", "Jana Kowalskiego" and
    "z Janem Kowalskim" share a token, and an identifier is one value with or without its separators
    and country prefix. Tokens keep the <LABEL_number> shape, with 12 digits by default. The key is
    never shown, pickled or copied; key_id and settings say what an index was built with, and
    scope= keeps two indexes under one key apart. A collision gets four digits more and is recorded
    in Pseudonyms(collisions=...).
  • Masking for an index: anonymizer.index_chunks(passages, store) masks the passages of a
    document with one mapping and puts what each token hides into a MappingStore, a two-method
    protocol over the application's own encrypted table (InMemoryMappingStore for tests and
    examples), with a person's name in the nominative and a fingerprint to tell a collision.
    anonymizer.mask_query(question, store) gives a question the documents' tokens, also for a
    surname written alone, which no rule reads and the store keeps for every indexed person. mask_chunks, stored_mapping
    and tokens_in mask without a store and restore found passages and answers, also while they
    stream; each passage also comes with its mapping as written, to restore it exactly.
  • Haystack 3 components, with pip install "polanon[haystack]": PolanonDocumentMasker masks
    documents before the splitter and the embedder of an indexing pipeline, PolanonQueryMasker
    gives a question the documents' tokens, PolanonRestorer restores the replies and the documents
    found, and PolanonStreamRestorer restores a streamed answer as a streaming callback. Nothing
    that names a value reaches the document store: not the content, not meta, and no id that is
    a plain hash of an original text. What the tokens stand for goes to the application's mapping
    store only. The key is a Haystack Secret, read from POLANON_KEY and saved by name only.
    examples/haystack_rag.py runs offline; examples/haystack_openai.py uses OpenAI and is paid.

Fixed

  • A token the mapping holds is left whole when masked text is masked again, such as a passage
    from an index or history sent back. The digits of <OSOBA_50810112354> read as a PESEL, a
    language model read OSOBA_8 as a place, a PIN label took the token after it for the PIN, and a
    known company called "Firma" was found inside <FIRMA_1>; each corrupted the token so restore no
    longer gave the value back. A placeholder that stands for itself still shelters nothing.
  • Titles stay out of a foreign name after a label. "z. Hd. Frau Wiśniewska" made "Frau" the
    first name, so every later "Frau" in the letter was masked as her, and "Attn: Mr Oliver Bennett"
    got a token apart from the "Oliver Bennett" of its sign-off. With two titles, as in "Frau Prof.
    Anna Schmidt", the second one was masked as the person wherever it stood.

v0.3.1

Choose a tag to compare

@github-actions github-actions released this 11 Nov 21:30

Fixed

  • A known value that only partly overlaps a new detection is masked whole: "Nowak z
    Kowalskich" known from an earlier turn met the name rule's "Anna Nowak", and "z Kowalskich" went
    out in the clear. The same holds for two known values that overlap.
  • Two findings of the judge that overlap are both masked. A small model that quoted "Ngozi
    Okonkwo-Brzęczyszczykiewicz" and "Okonkwo-Brzęczyszczykiewicz z kancelarii ..." had its first
    name sent, while judge_hits reported success.
  • A PIN that equals a year stays out of two dates joined by "-" or "/", a span of years such as
    2024/2025 and a date ending in 2025r.: in 01.01.<SEKRET_1>-31.12.<SEKRET_1> the dates around
    the token gave it away.
  • A known short number in one long line of numbers, such as a minified JSON array, no longer
    makes masking quadratic: 50 KB took most of a minute, and now takes under a second.

v0.3.0

Choose a tag to compare

@github-actions github-actions released this 10 Nov 23:10

Added

  • MaskedTurn, from anonymizer.turn(): one object for everything a call to a model sends and
    gets back. It masks the history, the question and the results of tools with one mapping, so a
    value keeps one token across them, restores the arguments of tool calls and the answer, also
    while it streams, and reports invented tokens. Values the caller already knows are sensitive,
    such as its clients' names, go in known= and are masked wherever they appear. When a detector
    fails, every mask method raises MaskingUnavailableError and returns nothing half masked.
    PolCRM and PolSales each kept a copy of this as a Masker of their own.
  • MaskedTurn.mask_messages() masks a whole history written for the Anthropic or the OpenAI
    API: text, tool_use input, tool_result and "tool" content, the arguments of tool_calls.
    Tool results and arguments written as JSON stay valid JSON, and ids and tool names are left
    alone. A value first recognised in a later message is masked in the earlier ones too. A block
    nothing can mask, such as an image or a model's signed reasoning, is refused, or left out with
    unknown_blocks="drop".
  • token_instructions("pl" | "en"): a note for the system prompt that tells the model what the
    tokens are and to copy them exactly, the one both apps wrote out for themselves.
  • MaskingUnavailableError, the one error an app catches to refuse sending text it could not
    mask, and NerModelUnavailableError, the one of them raised when spaCy or its model is
    missing. ner_available() runs the same check as default_guards() without loading
    anything, for a settings page.

Changed

  • default_guards(with_ner=True) without spaCy or its model raises NerModelUnavailableError
    instead of a plain ModuleNotFoundError. It is a subclass of ModuleNotFoundError with the same
    message, so except ModuleNotFoundError and except ImportError still catch it. Through
    MaskingUnavailableError it is also a RuntimeError, so an except RuntimeError around
    default_guards() now catches it too, and a check of the exact type, such as
    type(exc) is ModuleNotFoundError, no longer matches. Nothing else in the 0.2.2 API changed:
    every name and signature is still there, and every benchmark document masks to the same text,
    tokens and spans, with the language model and without it.

v0.2.2

Choose a tag to compare

@github-actions github-actions released this 08 Nov 21:52

Found by masking synthetic CVs, profiles written as JSON and lists of repositories with the rules
alone.

Added

  • Links to a person's profile are a new built-in type, LINK (tokens <LINK_1>): LinkedIn
    (linkedin.com/in/...), GitHub, GitLab, Bitbucket, Codeberg, GitHub and GitLab Pages, Stack
    Overflow, Kaggle, Behance, Dribbble, Medium, Instagram, Facebook, X, YouTube and a few more.
    Only the link up to the handle is masked, so <LINK_1>/weather-app still names the project.
    A LinkedIn link carried the person's name, and a GitHub login reappeared in every repository.
    The paths of a site's own pages (github.com/features, facebook.com/groups/...,
    youtube.com/watch) stay readable. A guard that declares LINK as its own type keeps working.
  • The surname of a masked person and the handle of a masked profile link are masked on their
    own
    wherever they appear in the text: "Kowalski" alone after "Jan Kowalski", and
    "login": "jkowalski", @jkowalski and jkowalski.github.io after github.com/jkowalski, went
    out in the clear. A lone surname in lowercase stays an ordinary word, as any known one-word name
    does; a name only the language model found gives its surname only next to a first name the rules
    know, and the handles of well-known brands (github.com/microsoft/...) stay readable.
  • Popular first names missing from the list: Oliwier, Cyprian, Maurycy, Celina, Ksenia,
    Lilianna, Matylda, Melania and Otylia, with their surnames: "Oliwier Nowak" went out whole.

Fixed

  • An e-mail address with its "@" written out is masked whole: jan.kowalski [at] example.com,
    (at), (małpa), also with [dot]. Only "jan" was masked, as a first name.
  • A street written without "ul." is masked when a postcode follows it: in
    Długa 5 m. 4, 00-238 Warszawa only the postcode was. A word and a number with no postcode
    after them, such as Windows 11, stay readable.
  • A birth date written in English is masked after its label: Date of birth: 15 March 1985,
    DOB: March 15, 1985 and born 3rd May 1985 went out whole. With DATE turned on, such dates
    are dates too.

v0.2.1

Choose a tag to compare

@github-actions github-actions released this 08 Nov 20:21

Added

  • polanon --version prints the installed version. Contributed by barkya28 in
    #29.
  • An eleventh benchmark document, a monthly roster of surnames and periods, which keeps names
    beside ordinary numbers masked without a language model. The corpus now has 169 annotated
    values: the rules alone fully mask 86% of them and the rules with pl_core_news_md 98%.

Fixed

  • E-mail addresses with Polish letters are masked whole: zofia.żak@example.com was masked
    as a name and half an address, and an address on an internationalised domain not at all.
    Contributed by ShahadatTest in #14.
  • A phone number after "tel." is labelled a phone, not a REGON: one nine-digit number in ten
    passes the REGON checksum, and the model was told it was a company's register number.
    Contributed by ShahadatTest in #22; "kom."
    counts as a phone label as well, and "hotel" no longer does.
  • A phone number written with a bare 48 in front is masked whole: 48123456789, as SMS
    gateways and CRM exports write it, went out unmasked, and in 48 123 456 789 the country code
    stayed in the clear. A valid PESEL that starts with 48 keeps its type, and the minutes of a time
    such as 10:48 are not pulled into the phone. Contributed by Aaqibhafeezkhan in
    #26.
  • A restored name with a case ending is not reported as new: "dla <OSOBA_1>a" restored as
    "Jan Kowalskia" was flagged by the output sweep as personal data the model made up. A person,
    company or place known from the conversation now covers its Polish case endings, while a longer
    different name, letters glued on the left, and other types such as an address with a letter
    added are still reported. Contributed by bfs2021 in
    #25.
  • A month spelled like a first name is left alone in a date written with dashes, slashes or
    dots: "Jan" in 15-Jan-2024, 2024/Jan/15 or 15.Jan.2024 was masked as a person. The same
    separator has to stand on both sides, so a name beside an ordinary number, as in
    Nowak Anna 03/2024, is still masked. Contributed by ShahadatTest in
    #23.

v0.2.0

Choose a tag to compare

@github-actions github-actions released this 03 Nov 03:02

Before this release a pre-release review went through the library looking for ways personal data
could get past it. Everything under Fixed is what it found; each fix comes with a test that
fails without it.

Added

  • --file PATH on the command line. polanon reads the file's bytes itself and tells UTF-8,
    UTF-16 and cp1250 apart, so Windows PowerShell 5.1 cannot garble Polish letters on the way in.
    Contributed by jay23606 in #5.
  • --mapping PATH writes the mapping to a file as UTF-8 JSON, created readable by its owner
    only, instead of to stderr, which Windows PowerShell's 2> wraps in error records.

Fixed

  • A UTF-16 file went out unmasked. Windows PowerShell 5.1 writes UTF-16 with >; decoded
    as cp1250 it was full of NUL characters that hid every value from the guards, and the command
    line printed it unmasked with exit status 0. Input is now decoded by its byte order mark,
    UTF-16 is recognised without one, a UTF-8 file with one cp1250 line is decoded line by line,
    and bytes that are none of these are refused with exit status 2.
  • The second of two IBANs went out whole when they were grouped and written one per line or
    joined by "oraz".
  • A double surname was read by its last part only, so "mają kowalską-nowak" went out where
    "mają kowalską" was masked.
  • A name on the cut of a long single-line text was missed by spaCy. Text over 100,000
    characters with no line break was cut at exactly that length, through the middle of a name.
  • A card number after the year of a date, "15.03.2024 4111 1111 1111 1111", went out whole.
  • Settings secrets named SECRET_KEY, APP_KEY or DB_PASS were not recognised, and a
    labelled password containing "," or ";" was cut short with the rest sent.
  • A birth date with the year abbreviation glued on, "ur. 15.03.1985r.", went out whole.
  • An address after a street word in an oblique case, "przy ulicy Długiej 5", went out
    whole; KRS is now also found after "pod numerem" and "No.".
  • A denylist term copied from Word, a PDF or macOS (an en dash, decomposed letters, a soft
    hyphen) matched nothing, not even the same text.
  • A judge error carried the HTTP error it came from, so a logged traceback could show a
    proxy password or an API key. Judge replies with code fences inside their JSON, a brace in a
    sentence before it, or a quote that differs from the text only in its whitespace are now read
    instead of dropped, and a reply of a few thousand spaces no longer freezes the event loop.
  • "Pana przesyłka została nadana" no longer masks the parcel as a person.
  • Masked CRLF documents no longer come out as CR CR LF on Windows, and two rules that took
    quadratic time on long lines without spaces now run in linear time.

Changed

  • The README no longer recommends Get-Content -Encoding UTF8 file | polanon for Windows
    PowerShell 5.1: that pipe turns Polish letters into ?. It recommends --file and
    --mapping instead.