Skip to content

DataFlow v1.0.11 Release Note

Choose a tag to compare

View all tags
@SunnyHaze SunnyHaze released this 28 Sep 10:36
v1.0.11
99edfcc

DataFlow v1.0.11 更新日志（v1.0.10 → v1.0.11）

🚀 DataFlow v1.0.11 — 并行加速、稳定性提升与工程完善

v1.0.11 是一次围绕「执行效率、核心稳定性与工程体验」展开的迭代版本。

相比 v1.0.10 对稳定性和细节的优化，这一版本进一步聚焦于：

  • 引入 RayOrch 并增强 DataFlow operator 的并行与 GPU 加速能力
  • 完善 PDF、文本处理、评测与推理相关链路
  • 修复存储、数据对齐、去重和 lazy 输出中的边界问题
  • 强化运行时错误处理、依赖安全与开发测试体验
  • 持续完善 README、Awesome DataFlow 与社区文档

这是一次让 DataFlow 更快、更稳、更适合实际生产使用的版本更新。

🔑 主要更新

⚡ RayOrch 加速与执行能力增强

  • 集成 RayOrch，支持 DataFlow operator 的并行数据处理
  • 新增 Ray 加速 operator、内存存储类及相关测试能力
  • 基于 RayOrch 0.1.1 支持 DataFlow operator 的 GPU 加速
  • 实现 pdf2model pipeline，并支持 DataFlex 训练后端

感谢 @SunnyHaze、@Heinz217。

🛠️ 核心模块稳定性修复

  • 增强 PromptedFilter 与 Text2MultiHopQAGenerator 的数据校验
  • 修复多处由未绑定变量导致的运行时崩溃
  • 修复 reasoning pipeline 中 branch root filter 覆盖步骤输出的问题
  • 修复 kbcleaning 中清洗文本泄漏问题
  • 在 NgramSampleEvaluator 中自动检测 CJK 文本
  • 重构 BenchDatasetEvaluatorQuestion，并修复非默认索引和结果校验问题
  • 修正 ChunkedPromptedGenerator 中 seperator 参数拼写错误

感谢 @memoryforget、@MBemera、@xuuuuuuu9、@zrh805、@ZhaoyangHan04、@dragonfrank666。

📦 存储、文本处理与数据质量优化

  • 保留 chunked 输出路径中的多后缀文件名
  • 为远程数据集正确应用 batch slicing
  • 修复无效 prompt 导致的 Text2QA 源数据错位问题
  • 保留长度小于 n-gram 大小的不同文本
  • 刷新 lazy 输出时保留源文件
  • 拒绝刷新 lazy source step，避免产生不完整结果

感谢 @zoedsy、@SunnyHaze。

🧩 Operator、依赖与开发体验改进

  • 新增 mmd operator
  • 通过 AST 扫描检查未进行懒加载的 operator 类
  • 将 transformers 最低版本提升至 >=4.51.0，修复 CVE-2025-3263 风险
  • 修复文件句柄泄漏，并改进异常处理
  • 为 pip extras 安装示例补充引号，提升跨 shell 兼容性
  • 更新 DataFlow-Agent、DataFlow-WebUI 与 Awesome DataFlow 相关 README

感谢 @yibotongxue、@SunnyHaze、@nyxst4ck、@Jununn、@XJCM1、@xuuuuuuu9。

👨‍💻 新贡献者

欢迎以下新贡献者加入 DataFlow 社区：

@dragonfrank666 · @yibotongxue · @Jununn · @XJCM1 · @xuuuuuuu9 · @zrh805 · @MBemera · @nyxst4ck · @zoedsy

🔗 完整更新记录

👉 v1.0.10...v1.0.11

DataFlow v1.0.11 Release Notes（v1.0.10 → v1.0.11）

🚀 DataFlow v1.0.11 — Parallel Acceleration, Stability Improvements & Engineering Refinements

v1.0.11 is an iteration focused on execution efficiency, core stability, and engineering quality.

Following the stability and refinement work in v1.0.10, this release emphasizes:

  • Introducing RayOrch-based parallel and GPU acceleration for DataFlow operators
  • Strengthening PDF, text processing, evaluation, and reasoning workflows
  • Fixing edge cases in storage, data alignment, deduplication, and lazy outputs
  • Improving runtime error handling, dependency security, and developer tooling
  • Continuing to improve README documentation, Awesome DataFlow, and community resources

This release makes DataFlow faster, more reliable, and better suited for production workloads.

🔑 Highlights

⚡ RayOrch Acceleration & Execution Improvements

  • Integrated RayOrch for parallel data processing across DataFlow operators
  • Added Ray-accelerated operators, in-memory storage classes, and validation tests
  • Added GPU acceleration for DataFlow operators with RayOrch 0.1.1
  • Implemented the pdf2model pipeline with a DataFlex training backend

Thanks to @SunnyHaze and @Heinz217.

🛠️ Core Stability Fixes

  • Enhanced data validation in PromptedFilter and Text2MultiHopQAGenerator
  • Fixed multiple runtime crashes caused by unbound names
  • Prevented the reasoning branch root filter from overwriting step outputs
  • Fixed cleaned text leakage in kbcleaning
  • Added automatic CJK text detection in NgramSampleEvaluator
  • Refactored BenchDatasetEvaluatorQuestion and fixed non-default index handling and validation issues
  • Corrected the seperator parameter typo in ChunkedPromptedGenerator

Thanks to @memoryforget, @MBemera, @xuuuuuuu9, @zrh805, @ZhaoyangHan04, and @dragonfrank666.

📦 Storage, Text Processing & Data Quality

  • Preserved dotted filenames in chunked output paths
  • Applied batch slicing correctly to remote datasets
  • Preserved Text2QA source alignment after invalid prompts
  • Preserved distinct texts shorter than the n-gram size
  • Preserved source files when flushing lazy outputs
  • Rejected flushing lazy source steps to prevent incomplete results

Thanks to @zoedsy and @SunnyHaze.

🧩 Operator, Dependency & Developer Experience Improvements

  • Added the mmd operator
  • Added AST scanning for operator classes that are not lazily loaded
  • Raised the transformers lower bound to >=4.51.0 to address CVE-2025-3263
  • Fixed file handle leaks and improved exception handling
  • Added quotes to pip extras installation examples for better cross-shell compatibility
  • Updated README documentation for DataFlow-Agent, DataFlow-WebUI, and Awesome DataFlow

Thanks to @yibotongxue, @SunnyHaze, @nyxst4ck, @Jununn, @XJCM1, and @xuuuuuuu9.

👨‍💻 New Contributors

Welcome our new contributors:

@dragonfrank666 · @yibotongxue · @Jununn · @XJCM1 · @xuuuuuuu9 · @zrh805 · @MBemera · @nyxst4ck · @zoedsy

🔗 Full Changelog

👉 v1.0.10...v1.0.11

Raw Release Note

What's Changed

  • feat(core_text): enhance data validation in PromptedFilter and Text2MultiHopQAGenerator by @memoryforget in #497
  • fix: correct parameter name from 'seperator' to 'separator' in ChunkedPromptedGenerator by @dragonfrank666 in #498
  • [test]: implement AST scanning for operator classes which not-lazzyload by @SunnyHaze in #500
  • feat: add mmd operator by @yibotongxue in #499
  • [pipeline & req]: implement pdf2model with DataFlex training backend; add pdf2model-dataflex dependency group by @Heinz217 in #502
  • [rayorch] integrate RayOrch for parallel data processing with new operators and storage classes by @SunnyHaze in #501
  • Fix kbcleaning cleaned text leakage by @ZhaoyangHan04 in #505
  • [deps] Raise transformers lower bound to >=4.51.0 (CVE-2025-3263) by @SunnyHaze in #507
  • Update README for DataFlow-Agent and DataFlow-WebUI upgrades by @Jununn in #509
  • Add dataflow-speech-quality to awesome DataFlow by @XJCM1 in #518
  • [README] integrate Awesome DataFlow into bilingual READMEs by @SunnyHaze in #519
  • Fixes #516: replace bare except with Exception and fix file handle leaks by @xuuuuuuu9 in #517
  • Fixes #396: auto-detect CJK text in NgramSampleEvaluator scoring by @zrh805 in #515
  • fix: five runtime crashes from names that were never bound by @MBemera in #513
  • docs: quote pip extras install examples by @nyxst4ck in #512
  • refactor(core_text): split BenchDatasetEvaluatorQuestion.run and fix row indexing by @MBemera in #514
  • fix(reasoning): prevent branch root filter from overwriting step output by @xuuuuuuu9 in #521
  • fix(core_text): preserve dotted filenames in chunked output paths by @zoedsy in #528
  • fix(storage): apply batch slicing to remote datasets by @zoedsy in #529
  • fix(core_text): preserve Text2QA source alignment after invalid prompts by @zoedsy in #525
  • fix(dedup): preserve distinct texts shorter than the n-gram size by @zoedsy in #526
  • fix(storage): preserve source files when flushing lazy outputs by @zoedsy in #527
  • fix(storage): reject flushing lazy source step by @SunnyHaze in #530
  • feat: accelerate DataFlow operators with GPUs by RayOrch 0.1.1 by @SunnyHaze in #533

New Contributors

Full Changelog: v1.0.10...v1.0.11

Contributors

zoedsy, SunnyHaze, and 11 other contributors
Assets 2
Loading