Added Curator and Spark example #220
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: RanjitR <ranjitr@nvidia.com>
sarahyurick
requested changes
Sep 4, 2024
| "NeMo Curator is a Python library that consists of a collection of scalable data processing modules for curating natural language processing (NLP) data for training large language models (LLMs). The modules within the NeMo Data Curator enable NLP researchers to mine high-quality text at scale from massive uncurated web corpora. \n", | ||
| "\n", | ||
| "NeMo Curator includes the following modules to perform data curation:\n", | ||
| "- Data download and Extraction\n", |
There was a problem hiding this comment.
Suggested change
| "- Data download and Extraction\n", | |
| "- Data download and text extraction\n", |
| "- Quality filtering\n", | ||
| "- Document-level deduplication\n", | ||
| "- Multilingual downstream-task decontamination\n", | ||
| "- Distributed Data Classification\n", |
There was a problem hiding this comment.
Suggested change
| "- Distributed Data Classification\n", | |
| "- Distributed data classification\n", |
| "## About this notebook\n", | ||
| "\n", | ||
| "\n", | ||
| "This notebook will use the **Tiny Stories [Dataset](https://huggingface.co/datasets/roneneldan/TinyStories)** as an example to demonstrate how you can run data curation using NeMo Curator, integrate additional data processing you may already have on PySpark and build and end to end curation pipeline. \n", |
There was a problem hiding this comment.
Suggested change
| "This notebook will use the **Tiny Stories [Dataset](https://huggingface.co/datasets/roneneldan/TinyStories)** as an example to demonstrate how you can run data curation using NeMo Curator, integrate additional data processing you may already have on PySpark and build and end to end curation pipeline. \n", | |
| "This notebook will use the **Tiny Stories [Dataset](https://huggingface.co/datasets/roneneldan/TinyStories)** as an example to demonstrate how you can run data curation using NeMo Curator, integrate additional data processing you may already have on PySpark, and build an end-to-end curation pipeline. \n", |
| "3. Perform additional processing using PySpark\n", | ||
| "4. Deduplication using Nemo Curator\n", | ||
| "\n", | ||
| "For a full working example of Nemo Curator, please refer this [tutorial](https://github.com/NVIDIA/NeMo-Curator/blob/main/tutorials/single_node_tutorial/single_gpu_tutorial.ipynb)\n", |
There was a problem hiding this comment.
Suggested change
| "For a full working example of Nemo Curator, please refer this [tutorial](https://github.com/NVIDIA/NeMo-Curator/blob/main/tutorials/single_node_tutorial/single_gpu_tutorial.ipynb)\n", | |
| "For a full working example of NeMo Curator, please refer this [tutorial](https://github.com/NVIDIA/NeMo-Curator/blob/main/tutorials/single_node_tutorial/single_gpu_tutorial.ipynb).\n", |
| "## Prerequisites\n", | ||
| "\n", | ||
| "### System Requirements\n", | ||
| "Here is the hardware setting for this notebook\n", |
There was a problem hiding this comment.
Suggested change
| "Here is the hardware setting for this notebook\n", | |
| "Here are the hardware settings for this notebook:\n", |
| "\n", | ||
| "#ignores checksum and marker files created by Spark job\n", | ||
| "processed_files = [filename for filename in get_all_files_paths_under(PROCESSED_DIR) \n", | ||
| " if not filename.endswith('.crc') or filename.endswith('_SUCCESS')]\n" |
There was a problem hiding this comment.
Suggested change
| " if not filename.endswith('.crc') or filename.endswith('_SUCCESS')]\n" | |
| " if not filename.endswith(".crc") or filename.endswith("_SUCCESS")]\n" |
| "source": [ | ||
| "t0 = time.time()\n", | ||
| "# Read input dataset from Spark output\n", | ||
| "input_dataset = DocumentDataset.read_parquet(processed_files, backend='cudf')\n", |
There was a problem hiding this comment.
Suggested change
| "input_dataset = DocumentDataset.read_parquet(processed_files, backend='cudf')\n", | |
| "input_dataset = DocumentDataset.read_parquet(processed_files, backend="cudf")\n", |
| "!mkdir -p {LOG_DIR}\n", | ||
| "!mkdir -p {EXACT_DEDUP_OUT_DIR}\n", | ||
| "\n", | ||
| "#ignores checksum and marker files created by Spark job\n", |
There was a problem hiding this comment.
Suggested change
| "#ignores checksum and marker files created by Spark job\n", | |
| "# Ignores checksum and marker files created by Spark job\n", |
| "# Read input dataset from Spark output\n", | ||
| "input_dataset = DocumentDataset.read_parquet(processed_files, backend='cudf')\n", | ||
| "\n", | ||
| "#Run exact deduplication to the input\n", |
There was a problem hiding this comment.
Suggested change
| "#Run exact deduplication to the input\n", | |
| "# Run exact deduplication on input_dataset\n", |
| " id_field=\"id\",\n", | ||
| " text_field=\"text\",\n", | ||
| " hash_method=\"md5\",\n", | ||
| " cache_dir=EXACT_DEDUP_OUT_DIR #Duplicated document ID list is output to the cache_dir\n", |
There was a problem hiding this comment.
Suggested change
| " cache_dir=EXACT_DEDUP_OUT_DIR #Duplicated document ID list is output to the cache_dir\n", | |
| " cache_dir=EXACT_DEDUP_OUT_DIR # Duplicated document ID list is outputted to the cache_dir\n", |
|
As discussed, moving this example to docs with a new PR https://github.com/NVIDIA/NeMo-Curator/pull/261 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description
Adds a simple example to demonstrate how Apache Spark can interoperate with NeMo Curator modules using intermediate parquet files.
Usage
Jupyter notebook tutorial
Checklist