NVIDIA NeMo Curator 1.3.0

Changelog Details

• ci: Package version bump to 1.2.0 by @thomasdhc :: PR: #1784
• Add SlurmRayClient for multi-node Ray cluster management on SLURM by @abhinavg4 :: PR: #1712
• ci: Fix gpu coverage and add gpu test check by @thomasdhc :: PR: #1790
• Build: drop rc0 pre-release tag and add dynamic git versioning by @thomasdhc :: PR: #1788
• ci: Gpu test matrix ci_worklod check by @thomasdhc :: PR: #1793
• ci: Fix build-test-publish-wheel by maintaing previous workflow by @thomasdhc :: PR: #1799
• fix: give nemotron-cc stages distinct names by @nightcityblade :: PR: #1763
• fix: preserve utf-8 characters in JSONL output by @nightcityblade :: PR: #1762
• Add CPU requirement to Llama Nemotron tutorial by @sarahyurick :: PR: #1773
• build(deps-dev): bump pytest from 9.0.2 to 9.0.3 by @dependabot [bot] :: PR: #1805
• build(deps): bump lxml-html-clean from 0.4.3 to 0.4.4 by @dependabot [bot] :: PR: #1564
• build(deps): bump pillow from 12.1.1 to 12.2.0 by @dependabot [bot] :: PR: #1803
• Add GLiNER instruction by @sarahyurick :: PR: #1772
• Add 60 second timeout to ArXiv check by @sarahyurick :: PR: #1776
• ci: add sync-skills workflow by @ko3n1g :: PR: #1806
• [ci] fix: retry apt-get installs to handle mirror sync failures by @ko3n1g :: PR: #1814
• refactor: extract Ray Serve backend boundary - inference (1/5) by @praateekmahajan :: PR: #1813
• [benchmarking] Adds gpustat data for processes running on each GPU by @rlratzel :: PR: #1823
• [benchmarking] Increases timeout for math_preprocess_llm_cleanup benchmark by @rlratzel :: PR: #1819
• [benchmarking] Updates timeouts based on analysis of prior CI runs by @rlratzel :: PR: #1831
• feat!: Typed Serve Config + Dyamo config stub - inference (2/5) by @praateekmahajan :: PR: #1820
• fix(image): addressing nvbug6075086 (when --images-per-tar exceed --batch-size) by @huvunvidia :: PR: #1828
• docs: add nemo-curator-docs Claude Code skill by @lbliii :: PR: #1825
• Add qwen3 by @oyilmaz-nvidia :: PR: #1827
• Update NDD (nemo data-designer) version by @huvunvidia :: PR: #1755
• Enhance VideoReader initialization with path validation by @suiyoubi :: PR: #1845
• More CVE fixes by @ayushdg :: PR: #1843
• [Tutorial] Merge multiple file prefixes generated by MegatronTokenizerWriter by @asolergi-nv :: PR: #1427
• docs: add OTEL_SDK_DISABLED workaround for NVBUG 6078008 SIGSEGV by @shubhamNvidia :: PR: #1842
• Improve math tutorial README documentation by @lbliii :: PR: #1766
• docs: cross-link Nemotron-CC recipe as a production NeMo Curator example by @arhamm1 :: PR: #1767
• Fix waveform tensor leak in audio pipeline and redesign AudioDataFilterStage for all VAD/Speaker combinations by @shubhamNvidia :: PR: #1765
• ci: add base_sha to codecov/codecov-action upload step by @ko3n1g :: PR: #1864
• Fix Sortformer tutorial issues and add InferenceSortformerStage benchmark by @melllinia :: PR: #1764
• feat: Placement Group based subprocess manager - inference (3/5) by @praateekmahajan :: PR: #1833
• feat: (experimental) Dynamo aggregated backend - inference (4/5) by @praateekmahajan :: PR: #1834
• [benchmarking] Adds support for fine-grained config overrides and global ray config by @rlratzel :: PR: #1840
• Tutorials: Add interleaved getting-started tutorial by @VibhuJawa :: PR: #1774
• Add ReadSpeech audio curation benchmark by @shubhamNvidia :: PR: #1851
• Readspeech tutorial: README, tests, and lower audio data filter GPU defaults by @shubhamNvidia :: PR: #1841
• feat(tutorials/readspeech): add interactive Jupyter notebook tutorial by @shubhamNvidia :: PR: #1870
• feat: (experimental) Dynamo Disaggregated serving + validators + router polish - inference (5/5) by @praateekmahajan :: PR: #1861
• Add generic audio processing pipeline for ASR and TTS data preparation by @sushmitha-deva-09 :: PR: #1679
• 26.04 staging by @lbliii :: PR: #1655
• Regenerate uv.lock from prev version from main by @ayushdg :: PR: #1875
• Bump python requirements to 3.11 min and 3.13 max by @ayushdg :: PR: #1855
• Add Nemotron Omni Model for Video caption pipeline by @suiyoubi :: PR: #1883
• fix: Support aarch64 torch install by @thomasdhc :: PR: #1891
• benchmark: add ndd_dynamo_dp4 entry; cache weights to drop ray-serve startup time by @praateekmahajan :: PR: #1896
• Fix checkpoint crash when ManifestWriterStage returns FileGroupTask by @melllinia :: PR: #1884
• [benchmarking] Adds paths config for specifying Docker container mounts by @rlratzel :: PR: #1879
• Bump Ray 2.55.1 / CUDA 12.9.1 / torch 2.10 / vllm 0.18; HAProxy ingress by @praateekmahajan :: PR: #1895
• docs: move install into Get Started, refresh Setup & Deployment index by @lbliii :: PR: #1878
• docs(26.04): close release-note gaps and add missing feature pages by @lbliii :: PR: #1893
• Lazy import for text module by @oyilmaz-nvidia :: PR: #1888
• Lazy init for openai client by @oyilmaz-nvidia :: PR: #1890
• [benchmarking] Adds --use-host-curator-benchmarking and --skip-curator-image-build* options by @rlratzel :: PR: #1910
• ci: Update container registry by @thomasdhc :: PR: #1934
• Add translation by @rkalaniNV :: PR: #1844
• ci: Update runner name by @thomasdhc :: PR: #1958
• Add fit_data_fraction and cache_path to KMeansStage by @sarahyurick :: PR: #1927
• build(deps): bump notebook from 7.5.1 to 7.5.6 by @dependabot [bot] :: PR: #1944
• build(deps-dev): bump gitpython from 3.1.45 to 3.1.50 by @dependabot [bot] :: PR: #1962
• docs: fix broken links — redirects, GitHub source links, drift by @lbliii :: PR: #1938
• docs: add rolling main version, remove translation from v26.04 by @lbliii :: PR: #1969
• docs(26.04): port SME video encoder docs (VP9 + BYO H.264) by @lbliii :: PR: #1926
• Don't show uv sync progress when installing extras to clean up CI logs by @ayushdg :: PR: #1971
• build(deps): bump lxml from 6.0.2 to 6.1.0 by @dependabot [bot] :: PR: #1848
• build(deps): bump mistune from 3.1.4 to 3.2.1 by @dependabot [bot] :: PR: #1939
• build(deps): bump jupyter-server from 2.17.0 to 2.18.0 by @dependabot [bot] :: PR: #1928
• build(deps): bump mako from 1.3.10 to 1.3.12 by @dependabot [bot] :: PR: #1941
• build(deps): bump python-multipart from 0.0.22 to 0.0.27 by @dependabot [bot] :: PR: #1942
• chore: Update dynamo to 1.1.0 by @praateekmahajan :: PR: #1957
• fix(filters): escape - in regex_url character class (#1601 ) by @SAY-5 :: PR: #1880
• build(deps): bump urllib3 from 2.6.3 to 2.7.0 by @dependabot [bot] :: PR: #1970
• build(deps): batch Dependabot lockfile updates by @ayushdg :: PR: #1974
• Add Skill to Nemo Curator for "Getting Started" by @chris-alexiuk :: PR: #1746
• Package translation prompt yamls by @rkalaniNV :: PR: #1972
• ci: Major refactor of release-workflows by @ko3n1g :: PR: #1911
• Nemotron-CLIMB data curation recipe by @sarahyurick :: PR: #1727
• docs(fern): switch to NVIDIA global theme as source of truth by @lbliii :: PR: #2000
• build(deps): bump idna from 3.11 to 3.15 by @dependabot [bot] :: PR: #2001
• Add filtration and PnCWithLLM stages for audio tagging pipeline by @sushmitha-deva-09 :: PR: #1863
• ci: validate release branch-rules by @ko3n1g :: PR: #2009
• fix(docs): correct version label to v1.2.0 by @lbliii :: PR: #1983
• [benchmarking] Migrate test-paths.yaml to paths section, drop duplicates by @rlratzel :: PR: #2026
• [benchmarking] Add configurable GPU memory warning threshold by @rlratzel :: PR: #1966
• [benchmarking] Add ping_users_on_failure toggle, disable in nightly config by @rlratzel :: PR: #2039
• Add ray data option for video benchmarks by @oyilmaz-nvidia :: PR: #2002
• ci: Remove sync skills workflow by @chtruong814 :: PR: #2041
• ci: disable uv cache for cpu tests by @thomasdhc :: PR: #2043
• Add long-form audio cutting pipeline for ALM pretraining by @yqwangustc :: PR: #1898
• test: add image reader sample data by @nightcityblade :: PR: #2032
• fix(utils): include port 65535 in get_free_port() scan range by @miclaldogan :: PR: #2024
• docs: rewrite README and expand CONTRIBUTING per PM review and OSS audit by @lbliii :: PR: #2030
• docs: show how to resume Common Crawl downloads by @omribz156 :: PR: #2029
• docs: fix broken links surfaced in 2026-05-14 scan by @lbliii :: PR: #1991
• ci: bump _release_library.yml to v1.4.3 by @ko3n1g :: PR: #2046
• Expose translation control knobs by @rkalaniNV :: PR: #2038
• docs: retire docs/ Sphinx tree, replace with pointer to fern/ by @lbliii :: PR: #1961
• fix: gate nemo_text_processing to x86_64 to unblock aarch64 build by @thomasdhc :: PR: #2049
• build(deps): bump aiohttp from 3.13.4 to 3.14.0 by @dependabot [bot] :: PR: #2047
• [benchmarking] Multiple fixes to stabilize the nightly benchmark suite by @rlratzel :: PR: #2035
• Unify task identity: collapse _uuid into deterministic task_id by @abhinavg4 :: PR: #2036
• build(deps): bump starlette from 0.50.0 to 1.0.1 by @dependabot [bot] :: PR: #2048
• Strip user-set task_id from tutorials & getting-started script by @abhinavg4 :: PR: #2058
• fix(audio): download FLEURS via hf_hub_download to avoid HTTP 429 by @mohammadaaftabv :: PR: #2057
• Refactor empty/sentinel tasks: EmptyTask class + SentinelTask base by @abhinavg4 :: PR: #2062
• feat(benchmarking): config-driven CI timeouts (default/cleanup/min) by @thomasdhc :: PR: #2068
• fix(audio): lazy-load audio stages and fix tutorial notebook paths by @mohammadaaftabv :: PR: #1835
• Fix EmptyTask references in benchmark and tutorials by @rlratzel :: PR: #2070
• Nemotron OCR SDG Pipeline by @suiyoubi :: PR: #1899
• caption quality evaluation by @weijiac0619 :: PR: #1980
• Minor updates to CLIMB tutorial by @sarahyurick :: PR: #2072
• docs: add Related Topics section to Quality Assessment & Filtering by @chenopis :: PR: #2071
• fix: exclude eval/ from check-manifest sdist validation by @thomasdhc :: PR: #2081
• Update Ray Data Backend to use compute strategy + Allow min/max/initial workers to be specified by @praateekmahajan :: PR: #2077
• fix: extract arxiv archives outside download dir by @nightcityblade :: PR: #2055
• build(deps): bump tornado from 6.5.5 to 6.5.7 by @dependabot [bot] :: PR: #2082
• build(deps): bump pyjwt from 2.12.1 to 2.13.0 by @dependabot [bot] :: PR: #2079
• build(deps): bump python-multipart from 0.0.27 to 0.0.31 by @dependabot [bot] :: PR: #2083
• build(deps): bump starlette from 1.0.1 to 1.3.1 by @dependabot [bot] :: PR: #2084
• build(deps): bump bleach from 6.3.0 to 6.4.0 by @dependabot [bot] :: PR: #2085
• build(deps): bump cryptography from 46.0.7 to 48.0.1 by @dependabot [bot] :: PR: #2087
• build(deps): bump aiohttp from 3.14.0 to 3.14.1 by @dependabot [bot] :: PR: #2088
• build(deps): bump jupyter-server from 2.18.0 to 2.20.0 by @dependabot [bot] :: PR: #2091
• build(deps): bump jupyterlab from 4.5.7 to 4.5.9 by @dependabot [bot] :: PR: #2099
• feat(nemotron-parse): richer inference metrics, engine_kwargs passthrough, fanout stage spec by @praateekmahajan :: PR: #2054
• feat: support stage config overrides via with_ by @praateekmahajan :: PR: #2086
• build(deps): bump pydantic-settings from 2.12.0 to 2.14.2 by @dependabot [bot] :: PR: #2100
• Use --reason for benchmark run reason by @rlratzel :: PR: #2103
• fix(dynamo): multimodal chat processor, hybrid KV cache compat, subprocess env passthrough by @praateekmahajan :: PR: #2053
• Fix NDD tutorial worker override by @praateekmahajan :: PR: #2104
• fix(deps): bump wandb, sqlfluff, libunbound8 for CVEs by @thomasdhc :: PR: #2108
• [benchmarking] Detect parent schedule for nightly session names by @rlratzel :: PR: #2105
• docs: clarify fuzzy dedup input blocksize by @nightcityblade :: PR: #2096
• Remove FlashAttn Deps by @suiyoubi :: PR: #2107
• [video] Remove model_does_preprocess by @suiyoubi :: PR: #2109
• fix: default workflow input extensions by filetype by @nightcityblade :: PR: #2045
• Add note about InferenceServer hang in ndd_data_generation_example tutorial by @sarahyurick :: PR: #2114
• [benchmarking] Generate stable nightly session names by @rlratzel :: PR: #2115
• [Curator Video] Enable Ray Data stage fusion via cpus=1.0 in CPU video stages by @weijiac0619 :: PR: #2110
• Add benchmark max timeout guard by @rlratzel :: PR: #2116
• chore: Bump to 1.3.0 by @chtruong814 :: PR: #2140
• build(deps): bump python-dotenv from 1.2.1 to 1.2.2 by @dependabot [bot] :: PR: #2117
• Pipeline resumability via source-level counter checkpointing by @abhinavg4 :: PR: #2063
• Add support for Slurm arrays by @sarahyurick :: PR: #2059
• build(deps): promote datasets to a core dependency (2164) by @svcnvidia-nemo-ci :: PR: #2165
• Remove lingering task_id from interleaved quickstart (2185) by @svcnvidia-nemo-ci :: PR: #2188
• Update Wikipedia URL generation and download (2183) by @svcnvidia-nemo-ci :: PR: #2191
• fix(deps): clear wheel-bundled ffmpeg CVEs from the all image (part 1/2) (2192) by @svcnvidia-nemo-ci :: PR: #2196
• Expose vllm_init_kwargs in semantic dedupe tutorials (2193) by @svcnvidia-nemo-ci :: PR: #2200
• Remove task id references (Stale) from fuzzy dedup notebook (2204) by @svcnvidia-nemo-ci :: PR: #2205
• fix(deps): bump etcd 3.5.21->3.5.30 to clear grpc-go auth-bypass CVE (2201) by @svcnvidia-nemo-ci :: PR: #2210
• fix(deps): remove system ffmpeg from container image (part 2/2) (2203) by @svcnvidia-nemo-ci :: PR: #2211
• fix(benchmarks): bump video timeout_s for ffmpeg source-build overhead (2214) by @svcnvidia-nemo-ci :: PR: #2215
• feat(deps): make opencv optional to strip ffmpeg CVEs from [all] (2216) by @svcnvidia-nemo-ci :: PR: #2221
• Port FFmpeg video codec support updates to main (2217) by @svcnvidia-nemo-ci :: PR: #2222
• fix(math): restore libmagic runtime and GPU coverage (2227) by @svcnvidia-nemo-ci :: PR: #2228
• build(deps): bump mcp from 1.26.0 to 1.28.1 by @thomasdhc :: PR: #2231
• build(deps): bump pillow from 12.2.0 to 12.3.0 (2230) by @svcnvidia-nemo-ci :: PR: #2234