Breath: Automated Breath Detection and Replacement

Breath is an audio-processing and machine learning pipeline designed to automatically detect human breathing patterns in audio recordings (e.g., voiceovers, podcasts) and replace them with natural room tone instead of harsh silence.

In voiceover production, loud or frequent breath sounds can distract listeners and disrupt the flow of speech. Traditionally, audio editors had two choices:

1. Manual Editing: Zooming into the waveform, cutting out every breath, and replacing it with room tone. This is incredibly tedious.
2. Noise Gating: Automatically silencing audio below a threshold. This is fast, but often clips words, and replacing breaths with absolute digital silence sounds unnatural and jarring.

Breath solves this by automating the detection using a Convolutional Neural Network (CNN) trained on mel-spectrograms, providing an interactive PyQt6 review interface, and replacing the breaths with tiled room tone.

---

Pipeline Architecture

graph TD
    A[Raw WAV + Audacity Labels] -->|split_audio.py| B[0.5s Audio Clips]
    B -->|convert_to_mel.py| C[Mel-Spectrogram Images]
    C -->|train_cnn.py| D[Trained SpectrogramCNN]
    E[Target WAV File] -->|inference.py| F[Sliding-Window Chunks]
    D --> F
    F -->|Batch Predictions| G[Audacity Label Output]
    G -->|waveform_viewer.py| H[PyQt6 Review GUI]
    H -->|edit_audio.py| I[Tiled Room-Tone Edited WAV]

Loading

---

Complete Workflow Components

1. Audio Splitting & Augmentation (split_audio.py)

• Label Parsing: Parses Audacity labels (.txt files containing tab-separated start_time, end_time, and labels).
• Sequential Dataset Management: Scans the Training-Images directory to find the next available dataset index (e.g., Dataset_1, Dataset_2, etc.) to prevent overwriting existing data.
• Breath Extraction & Data Augmentation: Performs translation augmentation on each labeled breath sound, extracting five overlapping 0.5-second snapshots (centered at shifts of -0.15s, -0.05s, 0.0s, +0.05s, and +0.15s) to expand the training dataset.
• Speaking Segment Extraction: Gaps between breath labels are identified as "Speaking" and sliced into continuous 0.5-second blocks. Both categories are stored as .wav files in Input/Breath/ and Input/Speaking/ respectively.

2. Mel-Spectrogram Generation (convert_to_mel.py)

• Transform Parameters: Audio files are loaded and converted into 224x224 RGB spectrograms using:
  • FFT size (n_fft): 2048
  • Hop length (hop_length): 512
  • Mel bands (n_mels): 128
  • Max frequency (f_max): 8000.0 Hz
• GPU-Accelerated Parallel Pipeline: Runs on GPU (supporting ROCm/CUDA via PyTorch) and uses a ThreadPoolExecutor (16 workers) to coordinate reading raw audio, GPU preprocessing, dynamic range scaling (decibels), and applying a magma colormap before saving them as images.

3. Model Training (train_cnn.py)

• Convolutional Network (SpectrogramCNN): Features 4 convolutional layers (32, 64, 128, and 128 channels) followed by MaxPool, ReLU activations, a fully connected layer (512 units), a dropout layer (0.5), and a classification head.
• Class Balancing: Class weights are calculated dynamically. The penalty for missing a breath is explicitly multiplied by 2.0 to prioritize breath recall.
• Validation & Early Stopping: Evaluates the model on validation data, tracking the F1-score of the "Breath" class. Training runs for a maximum of 20 epochs with a patience of 5 epochs.
• Fine-Tuning Mode: If best_mel_cnn.pth already exists, training resumes at a lower learning rate (0.0001 instead of 0.0005) to protect existing knowledge.

4. Sliding-Window Inference & Segmentation (inference.py)

• Sliding Window: Analyzes long audio using a 0.5-second sliding window with a 0.1-second hop size.
• Inference Thresholding: If prob_breath >= 0.80, the segment is labeled as a Breath.
• Heuristic Segment Cleaning:
  • Speaking segments < 0.5s: Flipped to Breath (assumed to be a short pause inside a breath).
  • Breath segments < 0.2s: Flipped to Speaking (filtered out as false-positive click noises or transients).
• Output: Saves individual breath .wav snippets and an Audacity label file (<base_name>_inference_labels.txt).

5. Interactive Review GUI (waveform_viewer.py)

• Interactive Visualizer: Renders the audio waveform using a dark-themed pyqtgraph interface.
• Review Loop: Displays the detected breath segments as adjustable bounding boxes. Users can drag boundaries, listen to selections (with volume boost up to 500%), and Accept or Reject predictions.
• Automatic Export & Edit Initiation: When a region is approved, timestamps append to Reviewed/<base_name>.txt. Once all regions are reviewed, the app cleans up Inference_Output and automatically executes edit_audio.py via a subprocess.

6. Audio Post-Editing (edit_audio.py)

• Room Tone Replacement: Reads Reviewed/Reviewed.txt and replaces the approved breath time ranges in Reviewed/Reviewed.wav by tiling a 1-second segment of natural ambient room tone (taken from a user-defined clean segment between 2.0s and 3.0s of the track).
• Result: Saves the clean recording as Reviewed/Edited.wav with seamless, natural-sounding pauses.

---

Model Performance & Evaluation Metrics

We evaluated the trained model checkpoint (best_mel_cnn.pth) on the dataset. Here is the statistical breakdown:

Dataset Overview

• Total Dataset Size: 5,406 spectrogram images (0.5s crops)
  • Speaking Class: 4,671 samples
  • Breath Class: 735 samples

Validation Set Performance (20% Split - 1,082 images)

• Overall Accuracy: 99.63% (1,078 / 1,082 correctly classified)
• Breath Class F1-Score: 0.9876
• Breath Class Precision: 0.9815 (Only 3 speaking segments mistaken for breath)
• Breath Class Recall (Sensitivity): 0.9938 (Only 1 breath missed out of 160)

Validation Confusion Matrix

True \ Predicted	Breath	Speaking
Breath	159 (TP)	1 (FN)
Speaking	3 (FP)	919 (TN)

Full Dataset Performance (100% - 5,406 images)

• Overall Accuracy: 99.69% (5,389 / 5,406 correctly classified)
• Breath Class F1-Score: 0.9886
• Breath Class Precision: 0.9787
• Breath Class Recall (Sensitivity): 0.9986 (Only 1 breath missed out of 735)

Full Dataset Confusion Matrix

True \ Predicted	Breath	Speaking
Breath	734 (TP)	1 (FN)
Speaking	16 (FP)	4,655 (TN)

---

Preventing Model Collapse

Because the dataset is built from only two raw audio recordings and has a heavy class imbalance (4,671 Speaking samples vs. 735 Breath samples), preventing the model from collapsing into a state where it only predicts the majority class ("Speaking") was a primary engineering challenge.

If the model collapses, it will simply output "Speaking" for every sample. While this would still yield a deceptively high baseline accuracy of ~86%, it would render the breath detection completely useless.

To prevent model collapse, the training pipeline implements several key machine learning techniques:

1. Cross-Entropy Class Weighting: We dynamically compute class weights based on the inverse frequency of the classes in the dataset. This penalizes the model heavily when it misclassifies the rare class.
2. Aggressive Breath Penalty Scaling: On top of the default inverse frequency weights, the loss penalty for misclassifying a breath is multiplied by 2.0 (class_weights[breath_idx] *= 2.0). This forces the optimizer to prioritize the minority class, ensuring that the model does not default to the majority class.
3. Data Augmentation (Translation Shifting): Without the 5-shift augmentation process during data preparation, the speaking-to-breath ratio would have been ~31:1 (making model collapse almost inevitable). The 5-shift augmentation artificially scaled the breath samples to bring the ratio down to a manageable ~6.3:1.
4. Learning Rate Management (Fine-Tuning Safety): If a pre-trained model exists, the pipeline switches to a smaller learning rate (0.0001 instead of 0.0005). This prevents gradient explosion and protects the established features from being overwritten by massive early weight changes.
5. Dropout Regularization: Using Dropout(0.5) forces the network to learn redundant representations rather than co-adapting to simple, fragile features of the small dataset, which helps the network generalize and avoid static collapse.

---

Mitigating Overtraining (Overfitting)

With a training dataset derived from only two raw audio recordings, the network is highly susceptible to overtraining (overfitting). Instead of learning the generalized acoustic characteristics of a human breath, the model can easily memorize the specific background noise, microphone coloration, room resonance, or vocal quirks unique to those two recordings.

To combat overtraining and ensure the model generalizes, the pipeline incorporates several mitigation strategies:

1. Validation-Guided Checkpoints: The training split represents 80% of the dataset, and the remaining 20% is reserved as a validation set. The model checkpoint (best_mel_cnn.pth) is updated only when the F1-score of the validation set improves, preventing the model from over-optimizing on the training set.
2. Early Stopping (Patience): The training loop monitors validation F1-score progress. If the validation F1-score fails to improve for 5 consecutive epochs, training terminates early to prevent the model from memorizing the training samples.
3. Dropout Regularization: The SpectrogramCNN architecture places a Dropout(0.5) layer before the final classification head. This randomly disables 50% of the neural connections during training forward passes, forcing the network to learn robust, redundant visual features rather than relying on brittle pixel combinations.
4. Translation Data Augmentation: By extracting five slightly offset variations of each breath, the model is exposed to breaths at different starting points, preventing it from overfitting to specific time alignments.

---

Model Design Decisions & Signal Processing Rationale

To build an efficient model, several key choices were made regarding audio representation and training optimization:

1. Treating Audio as a Computer Vision Problem

Instead of using 1D temporal sequence models (like LSTMs or RNNs) which are computationally heavy and harder to parallelize, the pipeline converts 1D audio waveforms into 2D images (spectrograms). This allows us to frame audio classification as a visual pattern recognition problem. Standard Convolutional Neural Networks (CNNs) are extremely fast to train, scale efficiently on GPUs, and naturally learn translation-invariant spatial features (such as the diagonal energy shape of a breath).

2. The Mel-Scale & Frequency Filtering

The audio is processed using the Mel Scale rather than a standard linear frequency scale. The Mel scale's logarithmic spacing matches human pitch perception, providing higher resolution in low-to-mid frequencies and lower resolution in high frequencies.

• Frequency Bound (f_max = 8000 Hz): Human speech and breath sounds carry the most information below 8 kHz. Restricting the frequency range to 8 kHz filters out high-frequency sensor noise, microphone hiss, and background room noise.
• Window Resolution (n_fft = 2048, hop_length = 512): Provides a solid balance between frequency resolution (separating bands) and temporal resolution (locating breath boundaries).

3. Colormap Uniformity (magma) for Feature Contrast

When converting single-channel audio power values into 3-channel RGB images, the choice of colormap is critical. The pipeline uses the magma colormap because it is perceptually uniform: luminance increases monotonically with intensity. This ensures that changes in amplitude (dB) translate directly to smooth, consistent changes in pixel brightness, helping the CNN filter layers detect structural edges and shape features more reliably.

4. Training Optimization via Automatic Mixed Precision (AMP)

To optimize training speeds on consumer GPUs, the training loop utilizes PyTorch's torch.amp.autocast and GradScaler. This runs most computational steps in 16-bit floating-point (FP16) while keeping critical operations in 32-bit (FP32), nearly halving memory usage and drastically reducing training times.

---

Key Design & Engineering Highlights

• Human-in-the-Loop Workflow: By combining deep learning with a custom interactive UI, the tool acts as a powerful assistant rather than an uncontrollable black box.
• Autocast / AMP: Training uses PyTorch’s Automatic Mixed Precision, reducing GPU memory footprint and cutting training time.
• Recall-focused Loss: Doubling the loss penalty for missed breaths ensures the system catches quiet or subtle breaths, leaving the easy rejections to the user during the review phase.
• Acoustic Continuity: Tiling actual room tone from the source recording prevents "dead air" gaps, maintaining the psychological presence of the room for the listener.