Skip to content

DataFlow v1.0.9 Release Note

Choose a tag to compare

View all tags
@SunnyHaze SunnyHaze released this 27 Feb 05:41
· 86 commits to main since this release
v1.0.9
01dfaec

DataFlow v1.0.9 更新日志（v1.0.8 → v1.0.9）

🚀 DataFlow v1.0.9 — 工程能力全面升级 & 全新 WebUI 发布

v1.0.9 是一次面向「规模化与工程化」的重要升级。

这一版本，我们不只是增加功能，而是系统性地强化了 DataFlow 的执行能力与使用体验：

  • 正式引入 Batched Pipeline，为大规模数据处理与长链路任务提供稳定的批量执行框架
  • 重构 Serving 架构，统一参数体系并增强可扩展性
  • 重构并发布全新的 DataFlow-WebUI，带来更现代化的可视化流水线体验

如果说 1.0.8 解决的是功能完整性问题，
那么 1.0.9 解决的是工程可用性与规模化能力。

DataFlow 正在从「研究工具」走向「工程系统」。

🔑 主要更新

🔄 Batched Pipeline 正式发布

  • 引入 Batched Pipeline，支持批量执行与分步处理
  • Storage 支持缓存、streamread、xlsx 读写、覆盖写入与 index 重置
    感谢 @fatty-belly、@wongzhenhao、@SunnyHaze。

🤖 Serving 架构重构

  • 重构 api_llm_serving，统一参数结构并增加自动测试
  • 支持自定义 generation configs
  • 修复 LiteLLM embedding 与跨平台一致性问题
    感谢 @SunnyHaze、@fatty-belly、@Libao-Yang。

🧠 Agentic & RAG

📄 MinerU & PDF 重构

🧪 Evaluation & Operator 优化

  • 重构评测 ops 与 pipelines
  • 批量化 Code Quality Evaluator
  • 重命名 PromptTemplatedGenerator → FormatStrPromptedGenerator
    感谢 @scuuy、@J1zz、@SunnyHaze、@haolpku、@MOLYHECI。

🌐 全新 WebUI 发布（重构版）

我们重构并发布了全新的 DataFlow-WebUI：
👉 https://github.com/OpenDCAI/DataFlow-WebUI

新版 WebUI 提供更稳定的前后端架构、更清晰的交互逻辑与更完善的流水线可视化能力，显著提升 DataFlow 的工程可用性与用户体验。

衷心感谢 WebUI 项目的所有贡献者：

@aleversn · @MOLYHECI · @SunnyHaze · @scuuy · @HeRunming · @TechNomad-ds · @zzy1127 · @duanchy3 · @wongzhenhao · @haolpku

👨‍💻 新贡献者

@Libao-Yang · @Aed-p · @ckane · @Lavender1 · @haosenwang1018

DataFlow v1.0.9 Release Notes（v1.0.8 → v1.0.9）

🚀 DataFlow v1.0.9 — Engineering Upgrade & New WebUI Launch

v1.0.9 marks a major step toward scalability and production readiness.

This release is not just about adding features — it is about strengthening DataFlow’s core execution architecture and user experience:

  • Introducing the official Batched Pipeline for scalable and resumable execution
  • Refactoring the Serving architecture with unified parameters and improved extensibility
  • Rebuilding and launching a brand-new DataFlow-WebUI for a modernized visual workflow experience

If v1.0.8 focused on feature completeness,
v1.0.9 focuses on engineering robustness and scalability.

DataFlow is evolving from a research toolkit into a production-grade system.

🔑 Highlights

🔄 Official Batched Pipeline

🤖 Serving Refactor

🧠 Agentic & RAG

📄 MinerU & PDF Refactor

🧪 Evaluation Updates

🌐 New DataFlow-WebUI (Fully Refactored)

We rebuilt and released a brand-new DataFlow-WebUI:
👉 https://github.com/OpenDCAI/DataFlow-WebUI

This release delivers a cleaner architecture, improved UX, and stronger pipeline visualization support.

Special thanks to all WebUI contributors:

@aleversn · @MOLYHECI · @SunnyHaze · @scuuy · @HeRunming · @TechNomad-ds · @zzy1127 · @duanchy3 · @wongzhenhao · @haolpku

What's Changed

  • Standardize the default values of the parameter SQL by @TechNomad-ds in #409
  • feat: add empty dataset for synthesis pipelines by @zzy1127 in #410
  • add mineru operator via api calling by @ZhaoyangHan04 in #411
  • Add empty jsonl dataset and update the source data for text2sql pipeline by @TechNomad-ds in #413
  • fix simple text pipelines & prompted generator by @haolpku in #415
  • [op & pipeline] fix bug in promptTemplatedGenerator by @SunnyHaze in #416
  • Batched pipeline and storage by @fatty-belly in #418
  • [Scaffold] add scaffold for create dataflow-extension by dataflow init repo CLI by @SunnyHaze in #419
  • Batched storage will cache the dataframe after reading for the first time by @fatty-belly in #420
  • [Batched Pipeline] Add tqdm bar. Delete batch_size argument in storage. by @fatty-belly in #421
  • remove pdf2model pipeline lang='en' and support the operator of ReasoningPretrainFormatConvertGenerator by @YalinFeng01 in #423
  • [op] Rename PromptTemplatedGenerator to FormatStrPromptedGenerator by @SunnyHaze in #424
  • [debug] fix wrong term in sample data of FormatStrPromptedGenerator by @SunnyHaze in #425
  • update entry file path by @wongzhenhao in #426
  • remove agenticrag_gpu,rename text2qa_apipipeline by @wongzhenhao in #427
  • Add Multihop AgenticRAG Pipeline by @YqjMartin in #399
  • [Fix Bug] fix schema parameters naming by @wongzhenhao in #429
  • [Batched Storage] Overwrite file when batch_step=0 by @fatty-belly in #432
  • [feature] streamread added for batchedpipeline by @wongzhenhao in #431
  • [storage] add support for read/write xlsx by @SunnyHaze in #433
  • [serving] refactor the api_llm_serving and add auto test for it by @SunnyHaze in #434
  • [serving] fix same behavior on win/linux by @SunnyHaze in #435
  • Reframe model evaluation ops and pipelines & Fix bug in compiled pl with local llm_serving by @scuuy in #430
  • fix: correct raw code input template by @J1zz in #438
  • Refactor: use batch generation in code quality evaluator by @J1zz in #440
  • [debug] update consistentchat & condor text operators for prompt_template arguments by @MOLYHECI in #441
  • [Serving] Add customized generation configs by @fatty-belly in #442
  • [Batched Storage] Reset index after slicing dataframe in storage.py by @fatty-belly in #444
  • [serving] revise parameter list for api serving by @SunnyHaze in #447
  • PDF2VQA 重构 by @fatty-belly in #443
  • Fix: embedding serving setup error in LiteLLMServing by @Libao-Yang in #448
  • [Readme] add awesome dataflow section by @SunnyHaze in #449
  • [cli] add support for automatically download and run dataflow-webui by @SunnyHaze in #451
  • [cli] add win support for webui cli by @SunnyHaze in #452
  • update get_desc by @wongzhenhao in #453
  • [Batched pipeline] update last_success_step.txt path include file prefix by @fatty-belly in #454
  • Kcenter desc by @wongzhenhao in #457
  • Correct QAExtractor operator action by @Aed-p in #459
  • [webui] fix default URL of webui to meet change by @SunnyHaze in #461
  • Fix bugs and optimize for user usage for Text2SQL pipeline by @TechNomad-ds in #460
  • [cli] make clear logic for dataflow webui; fix import issue for df init repo by @SunnyHaze in #465
  • Dev/fix python executor by @gty1829 in #466
  • [webui] make default cli no-interactive by @SunnyHaze in #467
  • [action] fix depend for python 3.12, with setuptools<=81.0.0 by @SunnyHaze in #471
  • Change fasttext-wheel to fasttext in requirements.txt by @ckane in #470
  • [MinerU Ops] integrate 3 mineru operators for clarity by @Lavender1 in #473
  • [mineru op] fix typo and update all pdf pipelines to flash-mineru by @SunnyHaze in #474
  • [pdf2vqa] 改为单输入，可以是一个pdf path或者pdf path list。同时适配了Mineru API的返回格式。 by @fatty-belly in #464
  • Remove duplicate import of 're' module by @HeRunming in #475
  • fix: replace 43 bare excepts with except Exception by @haosenwang1018 in #477

New Contributors

Full Changelog: v1.0.8...v1.0.9

Contributors

ckane, aleversn, and 19 other contributors
Assets 2
Loading