NVIDIA NeMo Curator 1.4.0

Changelog Details

• ci: use NVIDIA inference for Claude review by @chtruong814 :: PR: #2152
• ci: include Fern preview metadata artifact by @lbliii :: PR: #2141
• Add option to drop deduplication id field by @nightcityblade :: PR: #2078
• build(deps): promote datasets to a core dependency by @thomasdhc :: PR: #2164
• docs: clarify deduplication input discovery by @lbliii :: PR: #2130
• docs(fern): add local library autodocs without Fern auth by @lbliii :: PR: #2102
• docs: explain memory-efficient semantic dedup fitting by @lbliii :: PR: #2149
• Add Nemotron-CLIMB data-curation recipe by @lbliii :: PR: #2138
• Document SLURM job arrays and retries by @lbliii :: PR: #2154
• Remove lingering task_id from interleaved quickstart by @sarahyurick :: PR: #2185
• docs(audio): document arm64/aarch64 platform constraints by @lbliii :: PR: #2169
• Update Wikipedia URL generation and download by @sarahyurick :: PR: #2183
• docs: document native pipeline resumability by @lbliii :: PR: #2134
• fix(deps): clear wheel-bundled ffmpeg CVEs from the all image (part 1/2) by @thomasdhc :: PR: #2192
• Expose vllm_init_kwargs in semantic dedupe tutorials by @sarahyurick :: PR: #2193
• build(deps): bump soupsieve from 2.8.1 to 2.8.4 by @dependabot[bot] :: PR: #2194
• Remove task id references (Stale) from fuzzy dedup notebook by @ayushdg :: PR: #2204
• build(deps): bump json-repair from 0.57.1 to 0.60.1 by @dependabot[bot] :: PR: #2206
• docs: add "What's Hot" section to README by @lbliii :: PR: #1992
• fix(deps): bump etcd 3.5.21->3.5.30 to clear grpc-go auth-bypass CVE by @thomasdhc :: PR: #2201
• fix(deps): remove system ffmpeg from container image (part 2/2) by @thomasdhc :: PR: #2203
• build(deps): bump mistune from 3.2.1 to 3.3.0 by @dependabot[bot] :: PR: #2195
• fix(benchmarks): bump video timeout_s for ffmpeg source-build overhead by @svcnvidia-nemo-ci :: PR: #2214
• Added optional language parameter to FastTextLangId and updated logic by @abdr17 :: PR: #2199
• docs: add long-form audio cutting guide by @lbliii :: PR: #2136
• docs: complete translation configuration reference by @lbliii :: PR: #2132
• feat(deps): make opencv optional to strip ffmpeg CVEs from [all] by @thomasdhc :: PR: #2216
• Port FFmpeg video codec support updates to main by @aotnvidia :: PR: #2217
• Document current video caption models and Qwen migration by @lbliii :: PR: #2148
• Add Lance reader stage by @VibhuJawa :: PR: #2111
• fix(math): restore libmagic runtime and GPU coverage by @praateekmahajan :: PR: #2227
• Document stage worker sizing and backend overrides by @lbliii :: PR: #2142
• fix: validate PyArrow table columns by @nightcityblade :: PR: #2218
• build(deps): bump mcp from 1.26.0 to 1.28.1 by @dependabot[bot] :: PR: #2220
• build(deps): bump nltk from 3.9.4 to 3.10.0 by @dependabot[bot] :: PR: #2226
• Add Nemotron OCR pipeline guide by @lbliii :: PR: #2137
• Rewrite InferenceServer docs for Ray Serve and Dynamo by @lbliii :: PR: #2147
• build(deps): bump pillow from 12.2.0 to 12.3.0 by @dependabot[bot] :: PR: #2230
• fix: remove redundant fanout stage overrides by @nightcityblade :: PR: #2157
• Add Lance writer stage by @VibhuJawa :: PR: #2112
• Interleaved Lance Reader by @aotnvidia :: PR: #2232
• Minor updates to Slurm tutorial by @sarahyurick :: PR: #2242
• build(deps): bump pyasn1 from 0.6.3 to 0.6.4 by @dependabot[bot] :: PR: #2249
• Expand Nemotron-Parse tuning guidance by @lbliii :: PR: #2131
• Publish 26.07 release notes, migration checklist, and cut the 26.07 GA docs train by @lbliii :: PR: #2143
• fix(audio): benchmark tagging on pre-staged AMI meetings by @mohammadaaftabv :: PR: #2187
• Add more custom metrics to minhash stage by @ayushdg :: PR: #2274
• Add benchmark viewer URL metadata and Slack summary updates by @rlratzel :: PR: #2263
• build(deps): roll up approved Dependabot updates by @ayushdg :: PR: #2275
• Audio: add Qwen-Omni in-process ASR adapter by @mohammadaaftabv :: PR: #1967
• chore(ci): pin GitHub Actions to commit SHAs by @svcnvidia-autobot :: PR: #2277
• benchmarking: remove redundant CI-owned files by @rlratzel :: PR: #2282
• Allow the inputs for a stage to support more than one input task type by @ayushdg :: PR: #2239
• chore(ci): bump preflight workflow template by @svcnvidia-autobot :: PR: #2286
• chore(ci): bump community workflow to v1.8.7 by @svcnvidia-autobot :: PR: #2288
• Update to Ray 2.56.1, Dynamo 1.3.0, and vLLM 0.22 (cu129) by @praateekmahajan :: PR: #2064
• Support GlotLID language and script identification by @ccyyy1023 :: PR: #2281
• [benchmarking] Default nightly runs to 8 GPUs by @rlratzel :: PR: #2283
• build(deps): bump cryptography from 48.0.1 to 50.0.0 by @dependabot[bot] :: PR: #2289
• build(deps-dev): bump gitpython from 3.1.54 to 3.1.57 by @dependabot[bot] :: PR: #2290
• Bump min required uv version by @ayushdg :: PR: #2297
• feat(ray-data): add scheduler observability by @praateekmahajan :: PR: #2293
• Audio: add Qwen3-ASR adapter by @mohammadaaftabv :: PR: #2257
• Audio: add NeMo FastConformer ASR adapter by @mohammadaaftabv :: PR: #2254
• ci: replace cicd PAT with GitHub App tokens by @svcnvidia-autobot :: PR: #2299
• docs: correct fuzzy deduplication strictness guidance by @nightcityblade :: PR: #2287
• chore(deps): update Ray to 2.57.0 by @praateekmahajan :: PR: #2304
• [benchmarking] Enable additional nightly benchmarks by @rlratzel :: PR: #2298
• Increase text benchmark workloads by @sarahyurick :: PR: #2302
• ci: bump SSO preflight to v1.8.10 by @svcnvidia-autobot :: PR: #2309
• feat(agents): Ray Data scheduler reference in AGENTS.md by @weijiac0619 :: PR: #2295
• Increase Image benchmark pipeline duration to ~10min by @aotnvidia :: PR: #2312
• Document batch minhash input spec by @ayushdg :: PR: #2243
• benchmarking: compare math streaming executors by @VibhuJawa :: PR: #2306
• build(deps): bump aiohttp from 3.14.1 to 3.14.3 by @dependabot[bot] :: PR: #2291
• Update video benchmark by @meatybobby :: PR: #2310
• fix(semdedup): forward parquet storage options by @praateekmahajan :: PR: #2316
• Update video benchmark path by @meatybobby :: PR: #2319
• [benchmarking] Make benchmark reporting resilient by @rlratzel :: PR: #2320
• Loosen modifier benchmark by @sarahyurick :: PR: #2321
• Limit nightly Slack replies to issues by @rlratzel :: PR: #2322
• Audio: add sound event detection inference stage by @mohammadaaftabv :: PR: #2260
• Audio: add rule-based ASR hallucination filter by @mohammadaaftabv :: PR: #2258
• build(deps-dev): bump gitpython from 3.1.57 to 3.1.58 by @dependabot[bot] :: PR: #2315
• Support remote URLs for CC WARC iterator by @ayushdg :: PR: #2300
• feat(text): bound vLLM embedding generation by @praateekmahajan :: PR: #2317
• Bump RAPIDS to 26.08 by @ayushdg :: PR: #2303
• Disable RAPIDS kvikio direct write by @ayushdg :: PR: #2327
• [benchmarking] Add benchmark SKU override config by @rlratzel :: PR: #2323
• feat(audio): select the best ASR prediction by @mohammadaaftabv :: PR: #2266
• feat(audio): add transcript normalization stages by @mohammadaaftabv :: PR: #2267
• ci: route external jobs through ephemeral v2 by @svcnvidia-autobot :: PR: #2347
• perf: use PyArrow fast path for JSONL reads by @praateekmahajan :: PR: #2325
• Add normalization to exact and fuzzy dedup by @ayushdg :: PR: #2247
• fix(docs): preserve semantic deduplication IDs by @praateekmahajan :: PR: #2358
• [benchmarking] Add public Sortformer nightly benchmark by @mohammadaaftabv :: PR: #2305
• fix(io): preserve Arrow schemas through Parquet writes by @praateekmahajan :: PR: #2360
• Scale ALM, LibriSpeech, and audio-tagging benchmarks by @mohammadaaftabv :: PR: #2307
• Update GB200 text benchmarks by @sarahyurick :: PR: #2348
• fix(docs): read semantic output with Arrow dtypes by @VibhuJawa :: PR: #2366
• Add agent file for Dynamo by @sarahyurick :: PR: #2365
• build(deps): bump tornado from 6.5.7 to 6.5.8 by @dependabot[bot] :: PR: #2363
• Wait for resources to become available between actor pool stages by @ayushdg :: PR: #2357
• perf(semantic-dedup): bound parquet reads and memory by @praateekmahajan :: PR: #2367
• fix(benchmarking): relax dedup removal IO threshold by @rlratzel :: PR: #2369
• build(deps): bump mistune from 3.3.0 to 3.3.3 by @dependabot[bot] :: PR: #2364
• Audio: add Faster-Whisper ASR adapter by @mohammadaaftabv :: PR: #2271
• fix: route partial reruns to the current attempt by @svcnvidia-autobot :: PR: #2381
• feat(semdedup): bound pairwise precision and memory by @praateekmahajan :: PR: #2318
• build(deps-dev): bump gitpython from 3.1.58 to 3.1.59 by @dependabot[bot] :: PR: #2382
• Audio: add sound event detection postprocessing by @mohammadaaftabv :: PR: #2261
• feat: add per-node stage worker sizing by @praateekmahajan :: PR: #2354
• Add LLM-as-a-judge to eval suite and bump NDD version by @sarahyurick :: PR: #2324
• Add inference-server PDF benchmark path by @praateekmahajan :: PR: #2349
• Chore: Bump nltk by @ayushdg :: PR: #2387
• perf(semdedup): default pairwise precision to fp16 by @praateekmahajan :: PR: #2388
• fix: vLLM startup on GB200 by @praateekmahajan :: PR: #2391
• perf(semantic-dedup): reduce KMeans fit memory overhead by @praateekmahajan :: PR: #2386
• [benchmarking] Add environment config for benchmark subprocesses by @rlratzel :: PR: #2390
• fix: use Triton attention for Nemotron Parse by @praateekmahajan :: PR: #2393
• docs: fix Nemotron-Parse writer examples by @VibhuJawa :: PR: #2402
• Fix tutorial bug by @svcnvidia-nemo-ci :: PR: #2437
• fix(deps): remediate straightforward container CVEs by @svcnvidia-nemo-ci :: PR: #2427
• fix: standardize Nemotron OCR API key by @VibhuJawa :: PR: #2439
• fix: support pandas string operations on interleaved batches by @VibhuJawa :: PR: #2440
• fix(interleaved): handle null WDS content types by @svcnvidia-nemo-ci :: PR: #2451
• fix(stages): floor gpu_memory_gb-derived GPU fraction at 0.1 by @svcnvidia-nemo-ci :: PR: #2448
• fix(stages): handle empty diverse QA batches by @svcnvidia-nemo-ci :: PR: #2452
• docs(tutorials): add cv2 extra to Nemotron-Parse PDF README by @svcnvidia-nemo-ci :: PR: #2453
• fix(tutorials): replace retired hosted LLM default by @svcnvidia-nemo-ci :: PR: #2463
• fix(docker): bump nats-server to 2.14.7 and etcd to 3.5.34 by @svcnvidia-nemo-ci :: PR: #2467
• fix(container): update vulnerable Curator dependencies by @svcnvidia-nemo-ci :: PR: #2476
• Bump urllib3 and pyjwt due to CVEs by @svcnvidia-nemo-ci :: PR: #2475