Add support for skipping validation on resume + extend saving last ckpt test

[SeanNaren]

What does this PR do ?

When restarting training from a saved checkpoint during validation, validation is re-run as a first step (as the checkpoint was saved just before validation). This PR introduces a new loop injected via the exp_manager that skips validation if we're restarting.

There was also a mistake with the test definition from #4905 and I extended the test further to check @yaoyu-33s' case.

cc @MaximumEntropy @ericharper @titu1994

Changelog

• Skip initial validation by default when resuming from checkpoint which saved after a validation pass has occurred.

Before your PR is "Ready for review"

Pre checks:

• Make sure you read and followed Contributor guidelines
• Did you write any new necessary tests?
• Did you add or update any necessary documentation?
• Does the PR affect components that are optional to install? (Ex: Numba, Pynini, Apex etc)
  • Reviewer: Does the PR have correct import guards for all optional libraries?

PR Type:

• New Feature
• Bugfix
• Documentation

Who can review?

Anyone in the community is free to review the PR once the checks have passed.
Contributor guidelines contains specific people who can review PRs to various areas.

[SeanNaren]

Would really appreciate people running various tests to check this; it would also be great if we're able to remove https://github.com/NVIDIA/NeMo/blob/main/nemo/collections/nlp/models/language_modeling/megatron_lm_encoder_decoder_model.py#L646 and see if things work as expected.

[SeanNaren]

@MaximumEntropy just to confirm, I don't see any tests that explicitly check to see if training can resume from a nemo megatron checkpoint?

[MaximumEntropy]

@SeanNaren most megatron CI tests should have the same command twice where the first run saves a checkpoint and the subsequent one should load the saved checkpoint and continue training ex: https://github.com/NVIDIA/NeMo/blob/main/Jenkinsfile#L3261 and https://github.com/NVIDIA/NeMo/blob/main/Jenkinsfile#L3302

[ericharper]

LGTM. Thanks!