Release v1.5.5: External OP Plugins; HDFS I/O & Ray Data Optimizations; Elastic Multi-node Sharding

Major Updates

📊 Stats: 18 PRs merged, from 5 contributors

📈 Code diff: 84 files changed, with 11,991 insertions and 389 deletions

🧩 External operator plugins — Support loading external operators via Python entry points, allowing third-party OPs to be shipped as independent pip packages and used with zero extra config. #1026

🗄️ HDFS protocol support — Data-Juicer can now read datasets from and export processed results to HDFS (hdfs://), consistent with the existing S3 remote-storage workflow. #1014

⚡ Leaner & faster Ray Data — Removed eager count()/columns()/schema() actions that triggered premature Ray Dataset execution, replaced the private get_compute_strategy helper with the public TaskPoolStrategy/ActorPoolStrategy APIs, and made PartitionedRayExecutor run partitions concurrently. #1022 #1024 #1025 #1032 #1036

🌐 Elastic multi-node sharding — Available as a runnable reference workflow (scripts + sample configs under demos/elastic_sharding/) that pre-splits a large JSONL dataset into shards and processes them elastically across multiple nodes with a node-local Data-Juicer Ray executor, including retries, stale-claim recovery, status inspection, and ordered merge. #1015

📊 RayAnalyzer for distributed analysis — New RayAnalyzer computes filter stats via Ray map_batches and aggregates overall statistics (count/mean/std/min/max) using Ray native aggregation operators, without pandas materialization. It is the distributed counterpart to the local Analyzer. #1016

Enhancements

• External operator plugins via entry points: third-party OPs can be shipped as independent pip packages and are loaded into the global OPERATORS registry at import data_juicer.ops time. #1026
• HDFS protocol support for dataset loading and exporting. #1014
• Public Ray compute strategies: replaced the private get_compute_strategy helper with the public TaskPoolStrategy/ActorPoolStrategy APIs. #1024
• Reduced eager Ray dataset actions: removed premature count(), switched to take(k)/take_all(), and avoided repeated columns()/schema() calls. #1025
• Elastic multi-node sharding: available as a runnable reference workflow (scripts + sample configs under demos/elastic_sharding/) that pre-splits a large JSONL dataset into shards and processes them elastically across multiple nodes with a node-local Data-Juicer Ray executor (retries, stale-claim recovery, status inspection, and ordered merge). #1015
• RayAnalyzer for distributed data analysis: computes filter stats via Ray map_batches and aggregates overall statistics without pandas materialization. #1016
• Membership operators (in/not in) in general_field_filter for expressions like lang in ['en', 'zh']. #1000
• Streamed n-gram frequency counting: count character and word n-grams as they are generated instead of retaining an occurrence-sized intermediate list, cutting peak memory in character_repetition_filter and word_repetition_filter while preserving ratios, caching, and filtering decisions. #1029
• Bounded MinHash permutation workspace: reduce token-by-permutation matrices in 8 MiB blocks and fold exact minima in document_minhash_deduplicator, cutting RSS from 768.0 to 271.9 MiB at 4x while preserving byte signatures. #1035
• Test refactor & coverage: enhanced tests for base_op, job utils, and config. #1018

Fixed Bugs

• Concurrent partitioned Ray execution: PartitionedRayExecutor now drives partitions concurrently via a driver-side ThreadPoolExecutor instead of serially, preserving partition order. #1022
• Elastic actor pool concurrency: ActorPoolStrategy now honors (min, max) num_proc ranges instead of collapsing them to a fixed size. #1032
• Partition checkpoint preservation: partition checkpoints now survive Ray block-layout changes, so resumed runs no longer reprocess completed partitions. #1036
• text_chunk_mapper visible delimiters: split_pattern no longer emits visible delimiters (e.g. _@@@_) as standalone chunks. #999
• calibrate_response_mapper output pattern: now honors output_pattern for regex extraction instead of ignoring it. #1017
• image_diffusion_mapper null captions: handles missing or empty caption fields gracefully instead of crashing. #1020
• Silent no-op on empty OP list: DefaultExecutor and RayExecutor now warn when the process operator list is empty. #1020
• pandas 3 incompatibility with Ray Data: pinned pandas<3 in the distributed extra to avoid AttributeError during JSON export. #1023
• pyarrow>=17 compatibility: bumped fsspec accordingly. #1020
• Python 3.13+ dependency resolution: removed the numpy<2.0 upper bound (blocked tf-keras/ml-dtypes needing numpy>=2.1.0) and upgraded label-studio to >=1.23.0. #1027

Acknowledgements

• @LiuGuH contributed HDFS protocol support for dataset loading and exporting. #1014
• @macroguo-ghy contributed streamed n-gram frequency counting and a bounded MinHash permutation workspace to cut peak memory. #1029 #1035

New Contributors

• @LiuGuH made their first contribution in #1014

Full Changelog: v1.5.4...v1.5.5