oblitus 1.1.1

The README's quick start names signup_date and age as the sample table's quasi-identifiers, matching what scan prints
since 1.1.0. No code changes.

oblitus 1.1.0

• Readable output. oblitus propose -o plan.json prints each column's technique and the
  reason, the quasi-identifiers, and the apply command to run after review. oblitus apply
  prints the level reached, why it fell short, and the share of each column's cells blanked
  to protect rare rows. AnonymizationPlan.describe() and AnonymizationResult.describe()
  return the same text, and printing a result no longer dumps the whole table.
• Noise explorer page. The risk map, histogram and strength slider work from the
  keyboard: the histogram is one Tab stop read with the arrow keys, and on the risk map the
  arrows walk the points and Enter selects one. Risk-map techniques differ by marker shape as
  well as colour. A tap on a point selects that point, not the nearest point of the selected
  technique. Selecting laplace_noise shows that the page bounded the noise with the column's
  own minimum and maximum and that a release needs public bounds. Strengths carry units and a
  direction ("max_distance_percent = 5% of rows", "epsilon = 1 (smaller is stronger)"), the
  risk map gives chance as odds ("1 in 2,000"), the table keeps its first column in view on a
  phone, and the metrics are defined under it. Muted text is darker and axis labels larger.
• oblitus --help shows the usual path and the commands grouped by task. A missing input file
  is reported as ": no such file". Problem texts count "distinct customer_id values"
  instead of "entities", and a plain date's scan evidence says it can single people out.
• Event dates are quasi-identifiers. A date that is not a birth date (a visit, a purchase,
  an admission) is now counted with the other quasi-identifiers, and the anonymization
  proposal keeps only its year. Before, it was kept to the day and not counted, so city, sex
  and one visit date singled out 20 to 87% of people in a release labelled anonymized.
  Pseudonymization proposals keep the date and count it. A column whose dates are all the
  first of a month, the time axis of a monthly table, is a new reporting_period category:
  also a quasi-identifier, but kept as it is with statistical fidelity, so a monthly pattern
  survives.
• Ages are in five-year bands under anonymization with statistical fidelity, instead of
  means of groups of five. Group means give nearly every group its own age, which beside a
  date left rows unique and got the ages blanked: on the quick start table, the bands and the
  year keep 87% of ages and nearly every date, where month and means blanked 99% of ages.
• Random numeric codes are record IDs whatever their shape. Unique whole numbers spread
  evenly within each digit length are codes, which covers codes of mixed lengths, negative
  codes, short codes and codes with a few repeats. 1.0.2 only caught codes of one length, and
  such columns were still rounded and linked for about 60% of people. The spread is tested
  over the range the codes cover, so a sequence that stops short of a power of ten (1 to
  90,000) counts, and a few short placeholder values do not hide the codes. A column named
  after a quantity, or with values on both sides of zero, stays a number.
• The protection guide lists three more limits: consistent pseudonyms follow the value's
  frequency, rounded columns can join two releases, and deterministic encrypt leaks length.

A review of the library's statistical claims, measured by simulation, led to these fixes:

• The synthetic-data check no longer fails fresh data on exact copies by chance. The 1%
  limit on the excess of copies over the holdout now also needs a one-sided sign test (p ≤
  0.01). On small discrete tables the plain limit failed 27 to 51% of independent draws; it now
  fails about 1 to 4%, and copies of unique training rows are still found.
• The answer key says what its best score assumes. A new best_achievable_assumes list
  names hidden drivers that a model cannot learn back, drivers with blank cells and a random
  split of a panel. The best score is described as the truth's score on the test set, which a
  model can beat slightly by chance, not as a ceiling.
• Answer-key roles. Blanks that follow the target are a leak; a column computed from a
  driver (a noisy proxy of a hidden one) or correlated with a drawn target is
  correlated_but_not_causal, not a distractor; an earlier period of a column built from the
  target is listed with lags_target.
• majority_class_accuracy is the share of the more common class. It was one minus the
  positive rate, which is wrong when positives are the majority.
• Date shifts agree across tables when an ID is an integer in one and a float in another.
  Keyed draws now hash a whole-number float as its integer, as pandas stores an integer ID as a
  float once one value is missing. Before, only 1% of such rows got the same shift.
• The utility report adds added_blank_share per column, the cells a release blanked,
  since local suppression blanks rare cells and the filled cells alone can look unchanged. The
  mean change of a date column is now its shift in standard deviations, not a ratio to days
  since 1970.
• Differentially private integers decode into their own bin. When the bins did not split
  the integers evenly, up to 10% of decoded values fell in a neighbouring bin.
• top_bottom_code and multiplicative_noise are labelled distorts, not preserves distribution. Proposal reasons no longer say "shape kept" for an index or rank, or "3
  significant figures kept" where small values round to zero. The linkage problem text gives
  the number of rows it compared.
• With an entity_column, a person in several small quasi-identifier groups is counted once
  among the people below the target. The share below the target could exceed 100%.
• bin_numeric with a fixed width puts a value on an edge in the bin that edge opens (0.3 in
  [0.3, 0.4) with a width of 0.1); the edges were float sums such as 0.30000000000000004.
• round_numeric rounds the largest float below 0.5 down, and gives 0 instead of -0.0.
• One infinite cell no longer turns a column into NaN under gaussian_noise or normalize:
  the statistics leave infinite cells out, and the cell itself stays infinite.
• The docs now state: randomized response's estimator; suppression's bias on the filled cells;
  that DP accuracy covers the clamped values; noise mass in empty DP bins; the conservative
  zCDP conversion; date shifts break calendar alignment; truncation moves the declared mean;
  Spearman targets are latent for tied columns; infeasible correlation sets shrink; the learned
  copula loses non-monotone relations and dependent blanks; difficulty does not lower the best
  score of a target without noise.

oblitus 1.0.2

• Sparse numeric codes are record IDs. A column of unique integers of at least 6 digits,
  95% of one length (account, phone or medical record numbers), is now a record ID, whatever
  its name. Before, only densely numbered keys were: an account number was rounded to 3
  significant figures, and the release said anonymized while the rounded number still
  pointed to one row for about 6 in 10 customers of a 400-row table. Epoch timestamps stay
  numeric. Column names ending in account, account_number, conta, msisdn, matricula
  or prontuario are record IDs too.
• Long integers never stop a CSV read. Any column that Polars would read as a 128-bit
  integer (more than 19 digits, including values too long for 128 bits and digits of other
  scripts) is read as text, in whole-file, lazy and chunked reads alike. 1.0.1 handled only
  20 to 38 digits, and apply --chunk-rows still crashed on them.
• A listed empty string matches an empty CSV cell in dp-synthesize and dp-aggregate.

oblitus 1.0.1

• A numeric domain no longer echoes a private value in its error. Text in a column declared
  numeric (for MST, AIM or any release with a NumericDomain) raised pandas' own error, which
  quotes the offending value. It now raises SchemaError naming only the column, with the
  original error left out of the traceback.
• The same fix for aggregates and the synthetic-data check. Text in a summed or averaged
  column (aggregate_private, oblitus dp-aggregate) and text in a column that is numeric in
  the training table (check_synthetic, oblitus check-synthetic) raise SchemaError without
  the value. Bytes in a category column are compared as text instead of decoded, so a
  malformed byte no longer appears in an error with its position.
• Numeric bounds must be less than a float's range apart. Bounds such as -1e308 and 1e308
  are refused when declared. Before, a valid record at the upper bound crashed the release,
  so whether it crashed told an observer that the record was there, without spending budget.
• CSV input for dp-synthesize and dp-aggregate no longer depends on Polars' type guess.
  Columns with listed values are read as text and matched as written, so one record such as
  other beside true and false no longer turns a refusal into a release. Listed numbers and
  booleans match their JSON form.
• A CSV cell that cannot be parsed is refused without the cell. Polars' error quoted the
  cell and its offset in the file; every command now reports only the file name. Integers of 20
  to 39 digits are read as text instead of crashing the conversion to pandas.
• These refusals carry no trace of the original error, not even as the exception's context.

oblitus 1.0.0

• AIM synthesis. synthesize_private(..., method="aim") and oblitus dp-synthesize --method aim
  build a differentially private synthetic table with AIM, the iterative successor of MST. Each
  round measures the column group the model gets most wrong; a workload (--workload a,b,c)
  names groups of three or more columns whose joint distribution matters. The mechanism and its
  budget accounting are in oblitus, on the declared domains. Private-PGM fits the model. It
  needs the new aim extra (mbi and JAX, no PyTorch).
• PrivateSynthesis.measured_marginals lists every column group a synthesizer measured, for
  MST and AIM. oblitus dp-synthesize prints "column groups measured" instead of "column pairs
  measured".
• A sample of a Spark table (the noise explorer, the sampled checks) now always has the
  requested number of rows. It kept each row with a probability before, and could come up short.
• The roadmap is complete.

oblitus 0.4.0

• Short labels with a name are person names. A column of cells such as Paciente: Ana Silva
  (a listed name and at most one other word) is classed as a person name, so the default
  proposal replaces it instead of keeping it.

• Names in prose match in any capitalization. ana Silva and Ana silva are found when
  both the first name and the surname are in the bundled lists, as ana silva already was.

• apply_many checks every connected sub-join. A third table holding only some of the
  entities no longer hides a pair of tables that singles a person out. Groups of up to 6 tables
  are checked in every connected subset, larger ones in every connected pair and the full join.

• Row-count links cover columns that are also in a named link. A table that copies account
  no longer hides the link between account and a differently named patient column.

• k-anonymity counts people when the plan has an entity_column. The reported k and the
  rows-below-target figure count distinct entities per combination of every released
  quasi-identifier, joint microaggregation columns included, so a group the extra column
  splits down to two people is at_risk (PrivacyReport.counts_entities says which unit).

• The skew check covers text columns of any cardinality. A diagnosis column with more than
  50 distinct values no longer switches the check off; only numeric columns above 50 values are
  skipped.

• Anonymization drops free text by default. No list or pattern finds every name in prose
  (lowercase and foreign names pass them all), so the proposal suppresses free-text columns
  under anonymization. A kept redact_entities column stays at_risk, and the problem suggests
  dropping it or reviewing its text. Pseudonymization still redacts.

• apply_many joins the whole graph of tables. Tables linked through a bridge table are
  joined together, and two columns with different names that code the same entities are linked
  when their values occur equally often (frequency_links.py). k-anonymity and inference are
  checked over each connected group of tables, per entity.

• Joint microaggregation counts people when the plan has an entity_column. MDAV groups
  hold at least group_size distinct entities, an entity's rows share one vector, and the
  isolated-record and skew checks count entities, so five identical visits of one person are no
  longer a group of five.

• Joint microaggregation columns count as quasi-identifiers. The k-anonymity and skew
  checks now group by the aggregated vector too, so five records that share one released vector
  and one diagnosis make the release at_risk.

• Mondrian does not name rare categories. A category held by fewer than k_target rows in
  the table reads other inside a released set instead of showing its name.

• apply_many checks the join. Anonymized tables that release the same column with
  consistent codes are joined and checked for k-anonymity and inference over entities. A join that
  singles out an entity makes every table of it at_risk, naming the other table and the link.

• Joint microaggregation handles missing values. Records missing a listed column are
  microaggregated among records missing the same ones, or blanked when too few exist, instead
  of keeping their other values. The protection check now looks for isolated records over the
  listed columns.

• Skewed groups are flagged. Under anonymization, a quasi-identifier group where one sensitive
  value is at least 90% of the group and at least 30 points above its share in the table makes
  the release at_risk, even when l-diversity passes. The limits are the plan fields
  inference_confidence and inference_gain.

• Mondrian no longer publishes extreme values. A group of numbers or dates is described by
  the cuts that bound it (under 40, 40 to under 60, 60 or more), and the midpoint style
  publishes the group's median, so one outlier does not show in the release.

• check_synthetic catches copies of rare records. A generator that appends a few rare
  training records verbatim to thousands of fresh rows passed the whole-table tests. The check
  now also counts copies among the 10% most isolated training records, against the same
  count for the isolated holdout records. A copy is a synthetic row within 1% of the record's
  isolation distance on every column (exact or jittered) or on more than half of them (partial
  copy). Repeated ordinary rows no longer relax the rule; chance is measured from real holdout
  rows instead.
  SyntheticPrivacyReport.isolated_records and a line of oblitus check-synthetic show it.

• Names and cities in prose are redacted without the ner extra. A first name from the
  bundled IBGE list followed by capitalized words, and a municipality name, are found in free
  text. A column of notes that mentions them gets redact_entities in the default plan, and the
  release check reports any that remain. Names separated by several spaces or tabs are found,
  and so are lowercase names (a listed first name followed by a listed surname) and lowercase
  municipalities of several words after "em", "para", "na", "no" or "cidade de". Everyday-noun municipality names and a cell that holds
  only a name or address are left alone.

• Tagged identifiers and lowercase passport zones are detected. Identifier patterns no
  longer treat an underscore as part of a word, so wallet_0x..., cpf_5299... and
  0x..._backup are found in text and redacted. Letters or digits touching an identifier
  still stop a match. A passport machine-readable zone is matched and checked in any case.

• Synthetic-data checks. Anonymizer.check_synthetic and oblitus check-synthetic SYNTHETIC --real TRAIN [--holdout HOLDOUT] measure the share of exact copies, the distance to the closest
  real record (Gower-like, for mixed columns) and the nearest-neighbour distance ratio. With a
  holdout, they add the DCR share and a distance-based membership-inference attack with a
  bootstrap interval, and give a pass or fail verdict. See docs/synthesis.md.

• Files larger than memory. oblitus apply --chunk-rows N and Anonymizer.apply_file
  process a CSV or Parquet file in chunks and write the output incrementally. Keyed per-row
  techniques release exactly what an in-memory run releases. Techniques that read the whole
  column, k-anonymity and l-diversity enforcement, joint microaggregation, a missing row_key
  and an unshuffled-order anonymization plan are refused with a PlanError listing every
  reason. The protection level comes from the same whole-table checks, run lazily over the
  written file. Technique has a new whole_column_reason field.

• Spark tables. Anonymizer takes PySpark DataFrames (extra spark) through the same
  engine layer as DuckDB: the work stays lazy as Spark SQL, keyed noise is computed with
  sha2 and equals the pandas, Polars and DuckDB values, and the plan needs a row_key. The
  engine parity tests run on sqlframe, and a local PySpark session checks the same values
  in CI; real PySpark tests are skipped without Java.

• Tokenization with a vault. The tokenize technique replaces values with random tokens
  (optionally keeping length and format, or with a prefix) and keeps the mapping in a vault you
  pass to apply and reveal. MemoryVault and the SQLite-backed SqliteVault are built in,
  and TokenVault is the interface for your own. The CLI takes --vault PATH. It is
  pseudonymization, so anonymization plans refuse it.

• Several tables in one call. Anonymizer.apply_many and oblitus apply-many MANIFEST
  protect related tables together: the same person gets the same pseudonym, token and date
  offset in every table, and anonymizing plans share one throwaway key for the call. Protection
  is assessed per table. date_shift takes a stream name so date columns of different names
  share per-entity offsets.

• Key generation from several entropy sources. oblitus keygen [--source ...] and
  generate_key mix os.urandom (always included) with hardware RNG, microphone, camera, CPU
  timing jitter and, opt-in, public randomness beacons, with a health check and a per-source
  report. New extras audio and camera. Anonymizer(throwaway_key_source=...) draws the
  anonymization throwaway key from the mix.

• Differential privacy. Anonymizer.synthesize_private and oblitus dp-synthesize build a
  synthetic table with MST under a declared (epsilon, delta) or rho budget, and
  Anonymizer.aggregate_private and oblitus dp-aggregate release noisy counts, sums and means
  per group with OpenDP (new opendp extra), with the confidence-interval half-width of every
  value. Column domains and bounds must be declared, never read from the data. Every release
  returns an itemized budget report, and a spend above the declared budget raises
  PrivacyBudgetError. See docs/differential-privacy.md.

• Class imbalance. A question's positive_share sets how rare the positive class is by
  shifting the log-odds of its truth_column. The answer key gains a class_balance section
  with the PR-AUC baseline and the best achievable PR-AUC.

• Missing not at random. A column's missingness makes its blank cells follow its own
  value (mnar) or another column (mar). The key records each column's mechanism under
  missingness.

• Related tables. A schema with a tables list generates tables joined by foreign keys,
  with child counts per parent, child columns that read parent columns, and aggregates of
  child rows that a parent's target can depend on. oblitus challenge writes every child
  table, oblitus synthesize writes a folder of tables, and Anonymizer.synthesize_tables
  returns them. Treatments and test shifts are refused for related tables. See
  examples/customers_orders_challenge.json.

• Difficulty levels. A q...

data-anonymizer 0.2.0

data-anonymizer 0.2.0 detects sensitive columns, proposes a plan, anonymizes or pseudonymizes tables, and generates synthetic data and data-science challenges. It works on pandas, Polars and DuckDB tables.

Install

pip install "data-anonymizer @ git+https://github.com/rmonteiro-pereira/data-anonymizer.git@v0.2.0"
pip install "data-anonymizer[duckdb] @ git+https://github.com/rmonteiro-pereira/data-anonymizer.git@v0.2.0"

The wheel and source archive are attached below. Python 3.12 or later.

Highlights

• Three engines. pandas, Polars and DuckDB tables in, the same kind out. Noise is keyed per row, so the same row gets the same noise on every engine. A 1.07M-row table is anonymized in about 20 seconds.
• Secure defaults. propose then apply reaches anonymized on everyday tables. Identifiers are replaced. Account and card columns get fresh codes for each release. Timestamps and amounts that would single out rows are generalized, and local suppression enforces k-anonymity on every column the uniqueness check counts.
• Honest protection reports. The level (anonymized, at_risk, pseudonymized or identifiable) comes from checks on the released data:
  • surviving identifiers;
  • fingerprint and key columns;
  • unique rows;
  • linkage of noisy numbers, including several columns jointly;
  • free-text residue;
  • dominance of one contributor in a segment;
  • rescaling that a known value could undo.
• Business confidentiality. Revenue and rent are indexed per store by default, so no known value plus a public total gives the money back. Segments are opaque codes, and a segment with too few stores is merged into OTHER.
• Synthesis and challenges. Synthetic data comes from a schema or a reference table. Challenges come with causal graphs and true treatment effects, drift, label noise, leakage traps, and panels with time-based splits. Rare categories and extreme values of the reference are not copied.
• Noise explorer. An interactive page shows, column by column, what each noise setting does to the values.

Known limits

The checks are rules, not proofs. These limits are documented:

• "What keyed noise cannot hide" and "What the checks cannot catch" in docs/protection.md;
• the business guide, docs/business-confidentiality.md.

The main ones:

• Keys below the thresholds. The report misses an untreated key with fewer than 20 values, a key made of three or more columns, or a pair of columns that each have fewer than 20 values.
• Group sizes. Fresh codes hide the value but not the group. Someone who knows how many rows they have, or a few of their own amounts, can find their code.
• Seeds and keys. A release made with a seed alone or with a guessable key can be undone. Re-releasing under the same key shows what changed, and averaging many releases weakens the noise.
• Rescaled money. robust scaling, chosen to compare store sizes, is undone by one known value plus a true total. The report marks it at_risk.

See CHANGELOG.md for every change.