Support elastic training with pytorch

[william-wang]

What would you like to be added:

Support elastic training with pytorch

Why is this needed:

Use elastic training to improve the utilization of GPU.

Tasks

• allow elastic job to be enqueued in case of resource shortage
  Implementation: support elastic #2173
• support elastic annotation in preempt/reclaim plugin
  implementaion: support elastic annotation in preempt/reclaim plugin #2105

[william-wang]

@qiankunli, You can add the detail infomation about this subject here. Thanks.

[k82cn]

xref: #1887

[stale]

Hello 👋 Looks like there was no activity on this issue for last 90 days.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity for 60 days, this issue will be closed (we can always reopen an issue if we need!).

[stale]

Hello 👋 Looks like there was no activity on this issue for last 90 days.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity for 60 days, this issue will be closed (we can always reopen an issue if we need!).

[stale]

Closing for now as there was no activity for last 60 days after marked as stale, let us know if you need this to be reopened! 🤗