Add earnings config

[nithinraok]

Add Earnings21/22 Dataset Processing Pipeline with Forced Alignment

Overview

This PR introduces a complete 7-step processing pipeline for converting Earnings21 and Earnings22 datasets to NeMo format with advanced forced alignment capabilities. The pipeline supports both full dataset processing and evaluation subsets with optional speaker segmentation.

High-Level Changelog

New Features

Core Pipeline Processors:

• CreateInitialAudioAndManifest: Initial audio manifest creation with automatic audio conversion (MP3 → WAV, multi-channel → mono, any sample rate → 16kHz)
• CreateFullAudioManifestEarnings21: Ground truth text reconstruction from NLP token files with punctuation/capitalization preservation
• NeMoForcedAligner: Word-level forced alignment using NeMo ASR models with CTC heads
• CreateSentenceSegmentedManifest: Intelligent sentence-level segmentation based on CTM files with punctuation-aware splitting
• SpeakerSegmentedManifest: Speaker-change detection and segmentation with optional metadata mapping

Dataset Support:

• Earnings21 support (full dataset + eval10 subset)
• Earnings22 support
• Dual NLP file location handling for flexible dataset structures
• Speaker metadata CSV integration for name mapping

Audio Processing:

• Automatic audio format conversion (MP3/WAV → 16kHz mono WAV)
• Accurate duration calculation from audio files
• Batch processing with configurable test mode

Pipeline Configuration

7-Step Processing Workflow:

1. Initial Audio Manifest → Full audio files with duration
2. Text Population → Add ground truth transcripts from NLP files
3. Text Cleaning → Remove artifacts, brackets, special characters
4. Forced Alignment → Generate word-level CTM files with timestamps
5. Sentence Segmentation → Create sentence-level segments from CTM data
6. Speaker Segmentation → Create speaker-level segments (optional)
7. Field Filtering → Keep only required manifest fields

Key Configuration Options:

• dataset_type: "earnings21" | "earnings22"
• subset: "full" | "eval10" (earnings21 only)
• forced_alignment_model: Configurable NeMo ASR model
• preserve_punctuation / preserve_capitalization: Text processing options
• include_speaker_info / include_tags: Optional metadata inclusion

Output Formats

Sentence-Level Segments (Primary Output):

{
  "audio_filepath": "/path/to/audio.wav",
  "duration": 15.2,
  "offset": 45.3,
  "text": "This is a complete sentence with proper punctuation.",
  "alignment": [
    {"word": "This", "start": 45.3, "end": 45.6},
    {"word": "is", "start": 45.6, "end": 45.8}
  ]
}

Speaker-Level Segments (Optional):

{
  "audio_filepath": "/path/to/audio.wav", 
  "duration": 0,
  "text": "Speaker segment text...",
  "speaker": "speaker_1",
  "segment_id": 0
}

Usage Examples

# Process Earnings21 full dataset
python main.py --config-path=dataset_configs/english/earnings21 --config-name=config \
  dataset_type=earnings21 \
  dataset_root=/path/to/earnings21 \
  output_directory=/path/to/output

# Process Earnings22 with custom model
python main.py --config-path=dataset_configs/english/earnings21 --config-name=config \
  dataset_type=earnings22 \
  forced_alignment_model=nvidia/parakeet-tdt_ctc-1.1b \
  dataset_root=/path/to/earnings22 \
  output_directory=/path/to/output

[lilithgrigoryan]

Hi @nithinraok. Thanks!

Overall looks good to me. I left some comments about docs and docstrings, that need to be fixed and I will review code once more. Also, please, consider adding end2end tests.

[lilithgrigoryan]

2025

[lilithgrigoryan]

2025

[lilithgrigoryan]

2025

[lilithgrigoryan]

We already have a dedicated processor for this functionality: see FfmegConvert

Let's reuse this processor to avoid duplicating functionality.

[lilithgrigoryan]

For proper docs building we need documentation in specific format. You can refer to this config.

You need to have documentation section with dataset name. Without this section, the dataset will not appear in the generated docs.

Also, please, consider moving any file-level comments (i.e. "Expected output from this 5-step pipeline:..") into the documentation section to ensure they are captured by the doc builder.

You can test docs locally by

cd docs
make clean
make html SPHINXOPTS="-b linkcheck -W --keep-going -n

then opening docs/html/index.html in your browser to verify. There must be single warning 404 Client Error: Not Found for url: https://github.com/NVIDIA/.../dataset_configs/english/earnings21/config.yaml

[lilithgrigoryan]

Please add docstrings in the following format. Examples may be skipped if not necessary.

class DuplicateFields(BaseParallelProcessor):
    This processor duplicates fields in all manifest entries.

    It is useful for when you want to do downstream processing of a variant
    of the entry. E.g. make a copy of "text" called "text_no_pc", and
    remove punctuation from "text_no_pc" in downstream processors.

    Args:
        duplicate_fields (dict): dictionary where keys are the original
            fields to be copied and their values are the new names of
            the duplicate fields.

    Returns:
        The same data as in the input manifest with duplicated fields
        as specified in the ``duplicate_fields`` input dictionary.

    Example:
        .. code-block:: yaml

            - _target_: sdp.processors.modify_manifest.common.DuplicateFields
               input_manifest_file: ${workspace_dir}/test1.json
               output_manifest_file: ${workspace_dir}/test2.json
               duplicate_fields: {"text":"answer"}

Also, please update docs/src/sdp/api.rst with this processors.

[lilithgrigoryan]

Same here. Please, add proper docstrings and update api.rst

[lilithgrigoryan]

also, docstrings)

[lilithgrigoryan]

This one too

[lilithgrigoryan]

@Jorjeous Let’s move this change to a separate PR.
Also, please add an informative comment explaining why this needs to be installed manually, if the reasoning is clear.

[nithinraok]

Updated based on comments. @lilithgrigoryan pls have a look