Resuming From Checkpoint

[jomzxc]

I was training a model with this in the cli:

python /home/sabat/NeMo/examples/asr/asr_transducer/speech_to_text_rnnt_bpe.py \
    --config-path=/home/sabat/NeMo/examples/asr/conf/contextnet_rnnt --config-name=config_rnnt_bpe \
    model.train_ds.manifest_filepath=/home/sabat/NeMo/data/train_clean_100.json \
    model.validation_ds.manifest_filepath=/home/sabat/NeMo/data/test_clean.json \
    model.tokenizer.dir=/home/sabat/NeMo/tokenizers/tokenizer_spe_bpe_v500 \
    model.tokenizer.type="bpe" \
    model.train_ds.batch_size=40 \
    model.validation_ds.batch_size=40 \
    trainer.devices=-1 \
    trainer.accelerator="gpu" \
    trainer.precision=16 \
    trainer.max_epochs=50 \
    model.optim.lr=1e-4 \
    model.optim.betas=[0.9,0.999] \
    model.optim.weight_decay=0.0001 \

However i accidentally interrupted it. I tried to resume it again using:

python /home/sabat/NeMo/examples/asr/asr_transducer/speech_to_text_rnnt_bpe.py \
    --config-path=/home/sabat/NeMo/examples/asr/conf/contextnet_rnnt --config-name=config_rnnt_bpe \
    model.train_ds.manifest_filepath=/home/sabat/NeMo/data/train_clean_100.json \
    model.validation_ds.manifest_filepath=/home/sabat/NeMo/data/test_clean.json \
    model.tokenizer.dir=/home/sabat/NeMo/tokenizers/tokenizer_spe_bpe_v500 \
    model.tokenizer.type="bpe" \
    model.train_ds.batch_size=40 \
    model.validation_ds.batch_size=40 \
    trainer.devices=-1 \
    trainer.accelerator="gpu" \
    trainer.precision=16 \
    trainer.max_epochs=50 \
    model.optim.lr=1e-4 \
    model.optim.betas=[0.9,0.999] \
    model.optim.weight_decay=0.0001 \
    exp_manager.resume_if_exists=True

And it Raises a not Found Error:

raise NotFoundError(
nemo.utils.exp_manager.NotFoundError: There were no checkpoints found in checkpoint_dir or no checkpoint folder at checkpoint_dir :/home/sabat/NeMo/nemo_experiments/ConvRNNTBPE5x1/checkpoints. Cannot resume.

[jomzxc]

I checked for checkpoints and it exists but the problem still persists.
Here is the last checkpoint: /home/sabat/NeMo/nemo_experiments/ConvRNNTBPE5x1/2025-03-07_19-27-13/checkpoints/ConvRNNTBPE5x1--val_wer=2.3703-epoch=25-last.ckpt

[jomzxc]

Fixed:

python /home/sabat/NeMo/examples/asr/asr_transducer/speech_to_text_rnnt_bpe.py \
    --config-path=/home/sabat/NeMo/examples/asr/conf/contextnet_rnnt --config-name=config_rnnt_bpe \
    model.train_ds.manifest_filepath=/home/sabat/NeMo/data/train_clean_100.json \
    model.validation_ds.manifest_filepath=/home/sabat/NeMo/data/test_clean.json \
    model.tokenizer.dir=/home/sabat/NeMo/tokenizers/tokenizer_spe_bpe_v500 \
    model.tokenizer.type="bpe" \
    model.train_ds.batch_size=40 \
    model.validation_ds.batch_size=40 \
    trainer.devices=-1 \
    trainer.accelerator="gpu" \
    trainer.precision=16 \
    trainer.max_epochs=50 \
    model.optim.lr=1e-4 \
    model.optim.betas=[0.9,0.999] \
    model.optim.weight_decay=0.0001 \
    +exp_manager.resume_from_checkpoint="/home/sabat/NeMo/nemo_experiments/ConvRNNTBPE5x1/2025-03-07_19-27-13/checkpoints/ConvRNNTBPE5x1--val_wer\=2.3703-epoch\=25-last.ckpt"

Needed to use exp_manager.resume_from_checkpoint... Additionally when it is used with exp_manager.resume_if_exists, it causes a conflict between the the two methods.