You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
MaskedTurn, from anonymizer.turn(): one object for everything a call to a model sends and
gets back. It masks the history, the question and the results of tools with one mapping, so a
value keeps one token across them, restores the arguments of tool calls and the answer, also
while it streams, and reports invented tokens. Values the caller already knows are sensitive,
such as its clients' names, go in known= and are masked wherever they appear. When a detector
fails, every mask method raises MaskingUnavailableError and returns nothing half masked.
PolCRM and PolSales each kept a copy of this as a Masker of their own.
MaskedTurn.mask_messages() masks a whole history written for the Anthropic or the OpenAI
API: text, tool_use input, tool_result and "tool" content, the arguments of tool_calls.
Tool results and arguments written as JSON stay valid JSON, and ids and tool names are left
alone. A value first recognised in a later message is masked in the earlier ones too. A block
nothing can mask, such as an image or a model's signed reasoning, is refused, or left out with unknown_blocks="drop".
token_instructions("pl" | "en"): a note for the system prompt that tells the model what the
tokens are and to copy them exactly, the one both apps wrote out for themselves.
MaskingUnavailableError, the one error an app catches to refuse sending text it could not
mask, and NerModelUnavailableError, the one of them raised when spaCy or its model is
missing. ner_available() runs the same check as default_guards() without loading
anything, for a settings page.
Changed
default_guards(with_ner=True) without spaCy or its model raises NerModelUnavailableError
instead of a plain ModuleNotFoundError. It is a subclass of ModuleNotFoundError with the same
message, so except ModuleNotFoundError and except ImportError still catch it. Through MaskingUnavailableError it is also a RuntimeError, so an except RuntimeError around default_guards() now catches it too, and a check of the exact type, such as type(exc) is ModuleNotFoundError, no longer matches. Nothing else in the 0.2.2 API changed:
every name and signature is still there, and every benchmark document masks to the same text,
tokens and spans, with the language model and without it.