DataFog Core 0.3.0
DataFog Core 0.3.0 adds structured PERSON discovery and improves performance
when processing many findings. Rust, Python, Node.js, and browser WASM share
the detection and stateless protection behavior.
Structured PERSON support
- Discover explicit name fields such as
first_name,last_name, and
fullNamewithout a model or dictionary download. - Supply concrete JSON Pointer mappings for fields that automatic discovery
leaves unresolved, such as/customer/name. - Scan JSON string values with the original seven detectors and return each
finding with its field path. Protect the findings with the existing policies. - Use structured pseudonymization, tokenization, and restoration through
Rust, Python, and Node provider integrations. Browser WASM retains its
existing restriction on provider-backed operations.
PERSON detection uses field context. It does not recognize arbitrary names in
prose, and passing serialized JSON to scan(text) does not enable discovery.
See Discover and protect person fields.
Performance
- Index duplicate findings and resolve ordinary overlaps with ordered interval
selection. Built-in findings use an O(m log m) selection path, where m is the
finding count. - Reuse lazy text indexes during validation and byte/code-point/UTF-16 range
conversion, maintain running output positions, and avoid copying a whole
field for every Node transformation record. - Expose Rust's reusable
TextIndexfor converting multiple ranges from the
same string. See the Rust reference.
On one local macOS ARM64 benchmark, selection of 4,096 disjoint findings fell
from 60.3 ms to 0.56 ms. Separately, the bookkeeping changes reduced a complete
Node structured scan-and-protect request with 1,024 findings in one Unicode
field from 242.3 ms to 3.2 ms. These measure different stages and baselines;
they are not a universal speedup guarantee. Short-field workloads changed
little, and one sparse scan-only case was about 10% slower.
The selection benchmark notes
and bookkeeping benchmark notes
include inputs, methodology, and limitations.
Compatibility and publishing
Existing validation, offset semantics, transformation policies, and finding
preferences are preserved. Caller-supplied overlapping findings that mix
scored and unscored confidence can retain the original quadratic selection
algorithm to preserve its behavior. See Findings and ranges.
The Node and WASM release workflows publish to npm from GitHub Actions using
trusted publishing. Node builds cover macOS ARM64/x64, Linux GNU ARM64/x64,
and Windows x64. Node.js 24.x remains required.