Add max running time as an optional stop criteria

[sperlingxx]

/kind feature

Describe the solution you'd like
Currently, katib has three kinds of succeed reason:

1. ExperimentGoalReached
2. ExperimentMaxTrialsReached
3. ExperimentSuggestionEndReached

In my opinion, it will bring a lot convenient to some users if we add another succeed criteria: max running time.
We can add a new field MaxRunningTimeInMinutes *uint64 in experiment spec, and judge whether experiment duration exceeds pre-configured maximum running time.

[issue-label-bot]

Issue-Label Bot is automatically applying the labels:

Label	Probability
feature	0.98

Please mark this comment with 👍 or 👎 to give our bot feedback!
Links: app homepage, dashboard and code for this bot.

[gaocegege]

LGTM.

/priority p2

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[stale]

This issue has been automatically closed because it has not had recent activity. Please comment "/reopen" to reopen it.

[tenzen-y]

/reopen
/lifecycle frozen

[google-oss-prow]

@tenzen-y: Reopened this issue.

In response to this:

/reopen
/lifecycle frozen

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.