Added ability to save initial and latest models in addition to best

[sarthakpati]

Fixes #538

Proposed Changes

• added new saved models initial and latest in addition to best to improve model debugging
• added preference order for inference: best followed by latest

Checklist

• I have read the CONTRIBUTING guide
• My PR is based from the current GaNDLF master
• Non-breaking change (does not break existing functionality): provide as many details as possible for any breaking change
• Function/class source code documentation added/updated
• Code has been blacked for style consistency
• If applicable, version information has been updated in GANDLF/version.py
• If adding a git submodule, add to list of exceptions for black styling in pyproject.toml file
• Usage documentation has been updated, if appropriate
• History has been updated, if appropriate
• Tests added or modified to cover the changes; if coverage is reduced, please give explanation
• If customized dependency installation is required (i.e., a separate pip install step is needed for PR to be functional), please ensure it is reflected in all the files that control the CI, namely: python-test.yml, and all docker files [1,2,3,4]

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[codecov]

Codecov Report

Merging #539 (2ae26e0) into master (e3dc6c0) will decrease coverage by 0.01%.
The diff coverage is 86.36%.

@@            Coverage Diff             @@
##           master     #539      +/-   ##
==========================================
- Coverage   93.85%   93.83%   -0.02%     
==========================================
  Files         109      109              
  Lines        6931     6944      +13     
==========================================
+ Hits         6505     6516      +11     
- Misses        426      428       +2     

Flag	Coverage Δ
unittests	93.83% <86.36%> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
GANDLF/utils/__init__.py	100.00% <ø> (ø)
testing/test_full.py	99.00% <ø> (ø)
GANDLF/compute/training_loop.py	82.20% <75.00%> (-0.77%)	⬇️
GANDLF/compute/inference_loop.py	92.25% <100.00%> (+0.86%)	⬆️
GANDLF/utils/modelio.py	82.35% <100.00%> (+0.53%)	⬆️

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more