Skip to content

Release v1.6.2: Selector & Filter Boundary Fixes; Broader PII Redaction; Faster Dedup, Loading & Selectors

Latest

Choose a tag to compare

@cmgzn cmgzn released this 04 Nov 01:57
· 43 commits to main since this release
782ebbc

📊 Stats: 30 PRs merged, from 9 PR authors
📈 Code diff: 64 files changed, with 3,829 insertions and 803 deletions

Enhancements

  • Hash-column-only deduplication: document_deduplicator, document_simhash_deduplicator, image_deduplicator and video_deduplicator read the hash columns once, compute the kept indices from them, and return dataset.select(keep) instead of running a stateful Dataset.filter that decoded every column. document_line_deduplicator counts line hashes from the hash column in 10,000-row batches. The kept-row rules are unchanged. #1123 #1127 #1128
  • MinHash clustering: each band table maps a hash to the first sample that has it and unions later collisions during the scan, instead of allocating a set per sample per band and walking every table again. #1119
  • Batched column reads in field selectors: new iter_column(dataset, column, batch_size=10000) reads a column in slices instead of decoding one row at a time; topk_, frequency_, range_ and tags_specified_field_selector use it. #1125
  • Loading: unify_format checks Arrow null_count and skips the None-text filter when no text column has nulls; when nulls exist, the filter reads only the text columns in batches. #1129
  • Test coverage: restored discovery of tests/utils/job/, added CPU regression coverage for GPU preflight planning, job utilities and CLI entry points, arXiv extraction, and HaWoR/DWPose helpers; unit coverage is now above 70%. No production code changed. #1102
  • Source distribution: the PyPI sdist now ships data_juicer, tools, the build hook and metadata, so it can build a working wheel; before this release it carried only root metadata files and no package sources. The wheel is unchanged. #1131

Fixed Bugs

  • range_specified_field_selector: an omitted lower bound now defaults to 0 and an omitted upper bound to the dataset size, so one-sided upper_rank/lower_percentile bounds select a range instead of returning the whole dataset. No bounds still returns the dataset unchanged, in its original order; zero- and one-row datasets are returned as-is. #1100
  • topk_specified_field_selector: top-level field values now go through the same number conversion as nested keys, so numeric strings rank numerically ("100" > "10" > "9"), None no longer raises TypeError, and large integers are compared exactly. #1099 #1125
  • Zero top_ratio: topk_specified_field_selector and frequency_specified_field_selector distinguish an omitted ratio from an explicit 0, so top_ratio: 0 selects zero rows instead of the full dataset. #1120
  • random_selector: select_ratio of 0, or one that rounds down to zero rows, now returns an empty dataset with the original features instead of all rows or a TypeError. #1106
  • specified_numeric_field_filter: 0 and 0.0 are numeric and are kept when inside the configured range; False is treated as 0.0; list and dict values are filtered out instead of raising TypeError. #1111
  • suffix_filter: configured suffixes are normalized to the formatter's stored form, so txt, .TXT and .txt all match .txt samples. #1114
  • clean_email_mapper: the default pattern matches uppercase domains and TLDs, e.g. JANE@EXAMPLE.ORG. #1112
  • clean_ip_mapper: the default pattern matches compressed IPv6 (2001:db8::ff00:42:8329, ::1) and IPv6 addresses ending in an IPv4 address, without touching std::cout, times, MAC addresses or longer tokens. #1115
  • pii_redaction_mapper: CN mobile and ID numbers adjacent to CJK text are redacted, and so are JWTs, international phone numbers, MAC addresses, Feishu/DingTalk open IDs and bracketed IPv6 addresses ([2001:db8::1]) next to text, while [123], [cafe] and [12:01:33] are left alone. Secret values are redacted whole, including quoted values, placeholders, and the credential after Authorization: Bearer/Basic/Digest/Token; ID key-values such as user_id: 42, trace_id:'t1' and {"user_id": "u-123"} keep the key, separator and quotes and redact the value. Full-width : separators are accepted, and a value stays on its line unless the next line is indented. #1113 #1117 #1130 #1132 #1133 #1134
  • Image OPs: image_blur_mapper, image_face_blur_mapper, image_remove_background_mapper and image_diffusion_mapper write results back to the image bytes field as PNG bytes instead of raw pixel data, so chained image OPs can decode them. #1118
  • remove_comments_mapper: inline comments no longer consume the character before %, and full-line comments stay under the multiline option, including CRLF input. #1104
  • expand_macro_mapper: macros at the end of the text or directly followed by another macro are now expanded; the word-boundary check uses a lookahead instead of consuming the next character. #1138
  • Lenient JSONL: .jsonl, .jsonl.gz and .jsonl.zst inputs are decoded as UTF-8 with an optional BOM, so the first record of a BOM-prefixed file is kept; compressed shards whose suffix is uppercase or mixed-case (.JSONL.GZ, .Jsonl.Gz) now load as gzip/zstd instead of failing as plain text. #1103 #1136
  • Video frame processing: process_each_frame demuxes the input audio streams together with the video and muxes their packets into the output, so video_face_blur_mapper and video_remove_watermark_mapper keep the audio track. #1122
  • Job progress: completed-operation duration, input/output rows, throughput and reduction ratio are read from the completion event's metadata instead of missing event fields. #1109
  • UnionFind: find uses an iterative path compression, so long parent chains no longer hit the recursion limit. #1119

Compatibility Notes

  • Selectors and filters now treat explicit zero as zero: top_ratio: 0 and select_ratio: 0 return no rows, and specified_numeric_field_filter keeps 0, 0.0 and False. #1120 #1106 #1111
  • range_specified_field_selector applies one-sided bounds; recipes that relied on a single bound returning the full dataset must set both bounds or remove the selector. #1100
  • suffix_filter accepts suffixes with or without a leading dot and in any case; previously such values matched nothing. #1114
  • Default cleaning and redaction patterns cover more inputs: uppercase email domains, compressed IPv6, bracketed IPv6, PII next to CJK text, JWTs/MACs/open IDs, full-width key-value separators, and ID key-values in JSON or dict text. pii_redaction_mapper no longer treats the next line as a secret or ID value unless that line is indented. #1112 #1115 #1113 #1117 #1130 #1132 #1133 #1134
  • Image OPs store PNG bytes, which are lossless but can be larger than the original JPEG bytes. #1118
  • JobUtils no longer calls logger.remove()/logger.add() at startup, so it stops overriding the application's loguru configuration. #1109
  • nltk is bumped from 3.10.3 to 3.10.4 to address GHSA-j456-xh4h-cpf2, and GitPython from 3.1.44 to 3.2.1, which clears the advisories reported for the pinned version (e.g. GHSA-23mf-xhv8-69c2). #1124 #1135

Acknowledgements

New Contributors

Full Changelog: v1.6.1...v1.6.2