Added new optimizers

[AdiSir05]

Fixes #586

Proposed Changes

• Added the RAdam and LBFGS Optimizers

Checklist

• I have read the CONTRIBUTING guide.
• My PR is based from the current GaNDLF master .
• Non-breaking change (does not break existing functionality): provide as many details as possible for any breaking change.
• Function/class source code documentation added/updated.
• Code has been blacked for style consistency.
• If applicable, version information has been updated in GANDLF/version.py.
• If adding a git submodule, add to list of exceptions for black styling in pyproject.toml file.
• Usage documentation has been updated, if appropriate.
• History has been updated, if appropriate.
• Tests added or modified to cover the changes; if coverage is reduced, please give explanation.
• If customized dependency installation is required (i.e., a separate pip install step is needed for PR to be functional), please ensure it is reflected in all the files that control the CI, namely: python-test.yml, and all docker files [1,2,3].

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[sarthakpati]

This needs the key to be added in GANDLF.optimizers.__init__.py.

[codecov]

Codecov Report

Merging #646 (afd45f2) into master (c822d07) will decrease coverage by 0.02%.
The diff coverage is 100.00%.

❗ Current head afd45f2 differs from pull request most recent head bb88242. Consider uploading reports for the commit bb88242 to get more accurate results

@@            Coverage Diff             @@
##           master     #646      +/-   ##
==========================================
- Coverage   94.33%   94.32%   -0.02%     
==========================================
  Files         114      114              
  Lines        7677     7679       +2     
==========================================
+ Hits         7242     7243       +1     
- Misses        435      436       +1     

Flag	Coverage Δ
unittests	94.32% <100.00%> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
GANDLF/optimizers/__init__.py	87.50% <ø> (ø)
GANDLF/optimizers/wrap_torch.py	96.29% <100.00%> (+0.29%)	⬆️

... and 1 file with indirect coverage changes

📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more

I think the tests are not covered right now.

[Geeks-Sid]

@AdiSir05 , Please sign the CLA so that we can add this PR. The CLA bot mentioned above should provide you with the required links.

[AdiSir05]

@Geeks-Sid I have signed the CLA

[sarthakpati]

It seems LBFGS expects a separate closure arg in its step [ref]. Does it make sense to keep it or shall we discard LBFGS for simplicity?

[Geeks-Sid]

I agree with @sarthakpati . Maybe skip that optimizer for now. @AdiSir05 Please update the PR so that you can have your first contribution! Welcome to GaNDLF family.

[sarthakpati]

LGTM after tests.

[sarthakpati]

LGTM