fix(trainer): load checkpoint without ckpt_path which doesn't allow for updated hyperparameters

[roedoejet]

fixes #41

@davidguzmanr - can you try this PR out and see if it fixes the issue for you. It's a little annoying because it seems to lose all information about the current epoch - so the logger reverts to epoch 0, but it appears to have correctly loaded the weights and updated hyperparameters. I find it a bit surprising that PyTorch Lightning doesn't have this figured out yet. I'm going to leave this as a draft (ok - drafts aren't possible on private repos, but please don't merge this) because I don't really want to lose the ability to statefully resume training, which this PR seems to do. But, it will let @davidguzmanr run experiments with an updated LR until we figure out a better option.

[davidguzmanr]

It seems to be fine-tuning correctly now. The only issue as you mentioned is that the logger reverts to epoch 0, I think the TensorBoard logs can be modified to add the offset in the number of epochs if you want to visualize that, I think it is more important to be able to resume training from the checkpoint.

[roedoejet]

This looks like total overkill for what we want to do, but might be worthwhile to implement: https://lightning.ai/docs/pytorch/stable/notebooks/lightning_examples/finetuning-scheduler.html

[codecov]

Codecov Report

Merging #46 (b8d520b) into dev.cli (d30c847) will decrease coverage by 0.32%.
The diff coverage is 0.00%.

@@             Coverage Diff             @@
##           dev.cli      #46      +/-   ##
===========================================
- Coverage    55.02%   54.70%   -0.32%     
===========================================
  Files           44       44              
  Lines         2588     2603      +15     
  Branches       347      350       +3     
===========================================
  Hits          1424     1424              
- Misses        1091     1106      +15     
  Partials        73       73              

Files Changed	Coverage Δ
everyvoice/base_cli/helpers.py	0.00% <0.00%> (ø)

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

[roedoejet]

ok, so this is an updated patch that is ready for review and is now rebased on to #75 as well

[davidguzmanr]

It seems it still doesn't update the training hyperparameters (blue line) and instead continues using the ones from the checkpoint (orange line). I think it is the scheduler and maybe the optimizer states that also need to be updated