feat: add multiprocessing support for dataset loading and processing

[Ki-Seki]

No description provided.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[Copilot]

Pull Request Overview

This PR adds multiprocessing support to dataset loading and processing operations to improve performance. The changes enable parallel processing across multiple CPU cores when loading datasets from Hugging Face and applying transformations.

• Adds num_proc=os.cpu_count() parameter to load_dataset, map, and filter operations
• Updates dataset output structure to organize files by dataset name and subset
• Removes old upload script and adds new one for the restructured dataset format

Reviewed Changes

Copilot reviewed 13 out of 13 changed files in this pull request and generated 2 comments.

File	Description
dataset/utils.py	Updates save_dataset function to support nested directory structure and configurable dataset names
dataset/upload_masked_io.py	Removes old dataset upload script
dataset/upload_gim_sft.py	Adds new upload script for restructured GIM-SFT dataset format
dataset/mask_*.py (13 files)	Adds os import and multiprocessing support to dataset operations

---

_{Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.}