Save checkpoint if train_steps is smaller than batcher's steps_per_epoch

[dantreiman]

If you try to train using a total number of steps smaller than steps_per_epoch, the model weights are not saved and ludwig throws an exception.

This is useful for doing quick tests by training for a small number of steps to generate a saved model, i.e.

trainer:
  train_steps: 50

Output:

Training: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 50/50 [02:35<00:00,  3.11s/it]Traceback (most recent call last):
  File "/Users/daniel/mambaforge/envs/ludwig39-dev/bin/ludwig", line 33, in <module>
    sys.exit(load_entry_point('ludwig', 'console_scripts', 'ludwig')())
  File "/Users/daniel/Desktop/github/dantreiman-ludwig/ludwig/cli.py", line 166, in main
    CLI()
  File "/Users/daniel/Desktop/github/dantreiman-ludwig/ludwig/cli.py", line 66, in __init__
    getattr(self, args.command)()
  File "/Users/daniel/Desktop/github/dantreiman-ludwig/ludwig/cli.py", line 71, in train
    train.cli(sys.argv[2:])
  File "/Users/daniel/Desktop/github/dantreiman-ludwig/ludwig/train.py", line 387, in cli
    train_cli(**vars(args))
  File "/Users/daniel/Desktop/github/dantreiman-ludwig/ludwig/train.py", line 181, in train_cli
    model.train(
  File "/Users/daniel/Desktop/github/dantreiman-ludwig/ludwig/api.py", line 552, in train
    train_stats = trainer.train(
  File "/Users/daniel/Desktop/github/dantreiman-ludwig/ludwig/trainers/trainer.py", line 862, in train
    self.model.load(save_path)
  File "/Users/daniel/Desktop/github/dantreiman-ludwig/ludwig/models/ecd.py", line 160, in load
    self.load_state_dict(torch.load(weights_save_path, map_location=device))
  File "/Users/daniel/mambaforge/envs/ludwig39-dev/lib/python3.9/site-packages/torch/serialization.py", line 699, in load
    with _open_file_like(f, 'rb') as opened_file:
  File "/Users/daniel/mambaforge/envs/ludwig39-dev/lib/python3.9/site-packages/torch/serialization.py", line 230, in _open_file_like
    return _open_file(name_or_buffer, mode)
  File "/Users/daniel/mambaforge/envs/ludwig39-dev/lib/python3.9/site-packages/torch/serialization.py", line 211, in __init__
    super(_open_file, self).__init__(open(name, mode))
FileNotFoundError: [Errno 2] No such file or directory: '/Users/daniel/Desktop/github/dantreiman-ludwig/examples/rotten_tomatoes/results/experiment_run/model/model_weights'

[abidwael]

LGTM once checks pass

[github-actions]

Unit Test Results

       4 files   -        2         4 suites   - 2   1h 47m 48s ⏱️ - 1h 4m 14s
2 936 tests ±       0  2 887 ✔️ ±       0    49 💤 ±  0  0 ❌ ±0 
5 872 runs   - 2 936  5 754 ✔️  - 2 871  118 💤  - 65  0 ❌ ±0 

Results for commit be107b3. ± Comparison against base commit f02e92c.