Repository navigation
Release v1.6.2: Selector & Filter Boundary Fixes; Broader PII Redaction; Faster Dedup, Loading & Selectors
Latest📊 Stats: 30 PRs merged, from 9 PR authors
📈 Code diff: 64 files changed, with 3,829 insertions and 803 deletions
Enhancements
- Hash-column-only deduplication:
document_deduplicator,document_simhash_deduplicator,image_deduplicatorandvideo_deduplicatorread the hash columns once, compute the kept indices from them, and returndataset.select(keep)instead of running a statefulDataset.filterthat decoded every column.document_line_deduplicatorcounts line hashes from the hash column in 10,000-row batches. The kept-row rules are unchanged. #1123 #1127 #1128 - MinHash clustering: each band table maps a hash to the first sample that has it and unions later collisions during the scan, instead of allocating a set per sample per band and walking every table again. #1119
- Batched column reads in field selectors: new
iter_column(dataset, column, batch_size=10000)reads a column in slices instead of decoding one row at a time;topk_,frequency_,range_andtags_specified_field_selectoruse it. #1125 - Loading:
unify_formatchecks Arrownull_countand skips the None-text filter when no text column has nulls; when nulls exist, the filter reads only the text columns in batches. #1129 - Test coverage: restored discovery of
tests/utils/job/, added CPU regression coverage for GPU preflight planning, job utilities and CLI entry points, arXiv extraction, and HaWoR/DWPose helpers; unit coverage is now above 70%. No production code changed. #1102 - Source distribution: the PyPI sdist now ships
data_juicer,tools, the build hook and metadata, so it can build a working wheel; before this release it carried only root metadata files and no package sources. The wheel is unchanged. #1131
Fixed Bugs
range_specified_field_selector: an omitted lower bound now defaults to 0 and an omitted upper bound to the dataset size, so one-sidedupper_rank/lower_percentilebounds select a range instead of returning the whole dataset. No bounds still returns the dataset unchanged, in its original order; zero- and one-row datasets are returned as-is. #1100topk_specified_field_selector: top-level field values now go through the same number conversion as nested keys, so numeric strings rank numerically ("100">"10">"9"),Noneno longer raisesTypeError, and large integers are compared exactly. #1099 #1125- Zero
top_ratio:topk_specified_field_selectorandfrequency_specified_field_selectordistinguish an omitted ratio from an explicit0, sotop_ratio: 0selects zero rows instead of the full dataset. #1120 random_selector:select_ratioof0, or one that rounds down to zero rows, now returns an empty dataset with the original features instead of all rows or aTypeError. #1106specified_numeric_field_filter:0and0.0are numeric and are kept when inside the configured range;Falseis treated as0.0; list and dict values are filtered out instead of raisingTypeError. #1111suffix_filter: configured suffixes are normalized to the formatter's stored form, sotxt,.TXTand.txtall match.txtsamples. #1114clean_email_mapper: the default pattern matches uppercase domains and TLDs, e.g.JANE@EXAMPLE.ORG. #1112clean_ip_mapper: the default pattern matches compressed IPv6 (2001:db8::ff00:42:8329,::1) and IPv6 addresses ending in an IPv4 address, without touchingstd::cout, times, MAC addresses or longer tokens. #1115pii_redaction_mapper: CN mobile and ID numbers adjacent to CJK text are redacted, and so are JWTs, international phone numbers, MAC addresses, Feishu/DingTalk open IDs and bracketed IPv6 addresses ([2001:db8::1]) next to text, while[123],[cafe]and[12:01:33]are left alone. Secret values are redacted whole, including quoted values, placeholders, and the credential afterAuthorization: Bearer/Basic/Digest/Token; ID key-values such asuser_id: 42,trace_id:'t1'and{"user_id": "u-123"}keep the key, separator and quotes and redact the value. Full-width:separators are accepted, and a value stays on its line unless the next line is indented. #1113 #1117 #1130 #1132 #1133 #1134- Image OPs:
image_blur_mapper,image_face_blur_mapper,image_remove_background_mapperandimage_diffusion_mapperwrite results back to the image bytes field as PNG bytes instead of raw pixel data, so chained image OPs can decode them. #1118 remove_comments_mapper: inline comments no longer consume the character before%, and full-line comments stay under themultilineoption, including CRLF input. #1104expand_macro_mapper: macros at the end of the text or directly followed by another macro are now expanded; the word-boundary check uses a lookahead instead of consuming the next character. #1138- Lenient JSONL:
.jsonl,.jsonl.gzand.jsonl.zstinputs are decoded as UTF-8 with an optional BOM, so the first record of a BOM-prefixed file is kept; compressed shards whose suffix is uppercase or mixed-case (.JSONL.GZ,.Jsonl.Gz) now load as gzip/zstd instead of failing as plain text. #1103 #1136 - Video frame processing:
process_each_framedemuxes the input audio streams together with the video and muxes their packets into the output, sovideo_face_blur_mapperandvideo_remove_watermark_mapperkeep the audio track. #1122 - Job progress: completed-operation duration, input/output rows, throughput and reduction ratio are read from the completion event's metadata instead of missing event fields. #1109
- UnionFind:
finduses an iterative path compression, so long parent chains no longer hit the recursion limit. #1119
Compatibility Notes
- Selectors and filters now treat explicit zero as zero:
top_ratio: 0andselect_ratio: 0return no rows, andspecified_numeric_field_filterkeeps0,0.0andFalse. #1120 #1106 #1111 range_specified_field_selectorapplies one-sided bounds; recipes that relied on a single bound returning the full dataset must set both bounds or remove the selector. #1100suffix_filteraccepts suffixes with or without a leading dot and in any case; previously such values matched nothing. #1114- Default cleaning and redaction patterns cover more inputs: uppercase email domains, compressed IPv6, bracketed IPv6, PII next to CJK text, JWTs/MACs/open IDs, full-width key-value separators, and ID key-values in JSON or dict text.
pii_redaction_mapperno longer treats the next line as a secret or ID value unless that line is indented. #1112 #1115 #1113 #1117 #1130 #1132 #1133 #1134 - Image OPs store PNG bytes, which are lossless but can be larger than the original JPEG bytes. #1118
JobUtilsno longer callslogger.remove()/logger.add()at startup, so it stops overriding the application's loguru configuration. #1109nltkis bumped from 3.10.3 to 3.10.4 to address GHSA-j456-xh4h-cpf2, andGitPythonfrom 3.1.44 to 3.2.1, which clears the advisories reported for the pinned version (e.g. GHSA-23mf-xhv8-69c2). #1124 #1135
Acknowledgements
- @vamsibvv fixed image bytes write-back, video audio, and email/IP/PII/ID redaction, including bracketed IPv6, JWT/MAC/open-ID handling next to CJK text, and full-width key-value separators. #1112 #1113 #1115 #1117 #1118 #1122 #1130 #1132 #1133 #1134
- @apex-mochen sped up deduplication, field selectors, and dataset loading. #1119 #1123 #1125 #1127 #1128 #1129
- @xlb3055 fixed BOM-prefixed JSONL, inline LaTeX comments, zero-row random selection, and zero
top_ratio. #1103 #1104 #1106 #1120 - @MohammadHijjawi97 fixed top-k value conversion, zero numeric values, and suffix normalization. #1099 #1111 #1114
- @katsugtgz bumped
nltkandGitPythonfor their security advisories. #1124 #1135 - @BSAIPF fixed one-sided range bounds. #1100
- @ShryukGrandhi fixed macro expansion at the end of the text and next to other macros. #1138
- @bosprimigenious fixed decoding of compressed JSONL files with uppercase or mixed-case suffixes. #1136
New Contributors
- @BSAIPF made their first contribution in #1100
- @xlb3055 made their first contribution in #1103
- @vamsibvv made their first contribution in #1112
- @apex-mochen made their first contribution in #1123
- @ShryukGrandhi made their first contribution in #1138
- @bosprimigenious made their first contribution in #1136
Full Changelog: v1.6.1...v1.6.2