Skip to content

Issues: modelscope/data-juicer

Beta
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

35 Open 215 Closed
35 Open 215 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

sandbox 的trainModelHook是否支持训练本地模型和自定义模型 question Further information is requested
#649 opened Apr 22, 2025 by fengzx99
3 tasks done
[Bug]: 使用data-juicer中language_id_score_filter算子处理中文，会占满主机内存 bug Something isn't working
#647 opened Apr 21, 2025 by hxdsdu
3 tasks done
how to skip to last step to generate jsonl from arrow format question Further information is requested
#646 opened Apr 19, 2025 by gongysh2004
3 tasks done
A typo in readme news part question Further information is requested
#640 opened Apr 15, 2025 by finger1517
3 tasks done
1
[Bug]: 运行data-juicer-main\demos\process_code_data\app.py数据处理demo时出现ImportError，无法从data_juicer.core导入Executor bug Something isn't working
#631 opened Apr 2, 2025 by WangJunjie97
3 tasks done
2
后微调工具 question Further information is requested
#629 opened Mar 31, 2025 by heningsu
3 tasks done
是否支持ShareGPT多轮对话数据清洗 question Further information is requested
#628 opened Mar 29, 2025 by MLikeWater
3 tasks done
配好环境之后尝试运行python tools/process_data.py --config demos/process_on_ray/configs/demo.yaml，遇到疑似卡住无任何日志输出的情况 question Further information is requested
#624 opened Mar 18, 2025 by butterbutterflies
3 tasks done
6
参考demo, Ray 模式下只是通过多个CPU 读了数据，但是mapper 处理数据的时候只用了一个CPU question Further information is requested
#623 opened Mar 17, 2025 by Cheendfdf
3 tasks done
算子的使用方法 question Further information is requested
#621 opened Mar 15, 2025 by tiandidatongJLR
3 tasks done
get_init_configs issue in sandbox bug Something isn't working dj:core issues/PRs about the core functions of Data-Juicer
#612 opened Mar 7, 2025 by HYLcool
@HYLcool @BeachWang @yxdyc
希望增加针对金融数据处理或者遮蔽PII相关信息的算子 enhancement New feature or request good first issue Good for newcomers
#605 opened Mar 4, 2025 by ellie77ovo
2 tasks done
@HYLcool
4
FT-Data Ranker_大语言模型微调数据赛, 是否可以分享该比赛的数据用于对Data-Juicer项目的使用。 question Further information is requested
#603 opened Mar 3, 2025 by user2311717757
@HYLcool
1
运行 python tools/process_data.py --config configs/demo/process.yaml 报错 question Further information is requested
#602 opened Feb 28, 2025 by wqdta
@HYLcool
5
RAY模式下程序一直报OOM question Further information is requested
#601 opened Feb 28, 2025 by charonkk
3 tasks done
@cyruszhang @chenyushuo
@pan-x-c @garyzhang99
4
image_caption_mapper等类似算子使用前怎么处理自己的数据格式 question Further information is requested
#600 opened Feb 28, 2025 by Crazy-JY
3 tasks done
@HYLcool
7
搭建好环境后执行python tools/process_data.py --config configs/demo/process.yaml 命令报错 question Further information is requested
#580 opened Feb 18, 2025 by ctgushiwei
@HYLcool
1
一点小问题改进
#579 opened Feb 18, 2025 by 976311200
process_data.py pre-start is too slow 数据处理脚本启动过慢 dj:efficiency regarding to efficiency issues and enhancements question Further information is requested
#578 opened Feb 18, 2025 by hhhhsc701
3 tasks done
1
Installation progress could be optimzed. (Cmake error during installation) enhancement New feature or request environment related to third-party dependency, DJ-pypi, DJ-docker, etc.
#576 opened Feb 14, 2025 by zhenqincn
2 tasks done
以ray模式启动，当内存不足的时候,会溢写到磁盘吗？ question Further information is requested
#574 opened Feb 11, 2025 by javapythonphp
3 tasks done
2
Support others LLMs & APIs for the OP generate_qa_from_text_mapper dj:op issues/PRs about some specific OPs enhancement New feature or request
#535 opened Jan 9, 2025 by yxdyc
2 tasks done
[BUG]: inappropriate arguments for map_batches in ray mode bug Something isn't working dj:dist issues/PRs about distributed data processing
#533 opened Jan 8, 2025 by HYLcool
Can the cleaning statistics be viewed after creating the config file and performing the cleaning? question Further information is requested
#499 opened Nov 27, 2024 by Tendo33
3 tasks done
@HYLcool
Guidance on Monitoring Task Execution with Ray Executor in Data Juicer dj:dist issues/PRs about distributed data processing question Further information is requested
#496 opened Nov 24, 2024 by Fatima-0SA
3 tasks done
@chenyushuo @pan-x-c @garyzhang99
ProTip! What’s not been updated in a month: updated:<2025-03-24.