[audio codec] Add support for Lhotse training format

[rfejgin]

This PR adds a Lhotse data loader for audio codec training. It also introduces mechanisms to make the training process more stable and to help debug potential NaN issues.

Data Loading

The functionality is split between built-in Lhotse capabilities and a simple custom dataset class.

Lhotse

These operations are handled directly in Lhotse:

1. Duration filtering: Only keeps cuts whose duration is at least that of the training chunk size (n_samples).
2. Random segment selection: Selects a random segment of length n_samples from the cut.

Configuring Lhotse happens in AudioCodecModel._get_lhotse_dataloader().

AudioCodecLhotseDataset

The custom dataset class, AudioCodecLhotseDataset, receives a CutSet from Lhotse and performs the following:

1. Loads and collates the audio.
2. Resamples to the target sample rate, which is the codec's output_sample_rate.
3. Performs sanity checks on the audio (details below).

Training Robustness

When training models on Lhotse datasets, we observed convergence similar to the previous training recipe. However, in one instance (out of ~4 training runs), the training ran into NaNs. To help debug this issue if it ever reoccurs and to stabilize training, the following mechanisms were added:

1. Sanity checks in the loader: Errors out if the loaded audio has an unexpected length or contains NaN or infinite values. It also warns if suspicious sample values (abs(sample) > 1.5) are encountered.
2. Gradient norm tracking (operates separately for the discriminator and generator):
   • (Optional) Skip updates: If an infinite or NaN gradient norm is detected, it triggers a warning and skips the current parameter update.
   • (Optional) Gradient clipping: Applies gradient norm clipping.
   • Logging: Logs the gradient norm both before and after clipping.

Additional notes

• I have confirmed that old YAML configs still run without error after these changes
• A companion PR is being prepared in the internal repo that correspondingly updates the training recipe.

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[rfejgin]

I've addressed the PR comments. Apologies, but the commit history is lost because I needed to force-push to repair some unsigned commits that were blocking CI from running.

[github-actions]

[🤖]: Hi @rfejgin 👋,

We wanted to let you know that a CICD pipeline for this PR just finished successfully.

So it might be time to merge this PR or get some approvals.