Improve Job API to support specific types of job

[hzxuzhonghu]

Currently, when users need to do TensorFlow training, firstly they need to define the TF job, with plugin svc env set.

But if they want to run MPI job, they would need to set svc, ssh plugins. It is not friendly, so we want to improve, this will require API change to support specific types of jobs without requiring users to be aware of the inner plugins.

[hzxuzhonghu]

/kind feature

[shinytang6]

There are some questions:

1. To completely remove the configuration of job plugins, it will inevitably bring incompatibility and even affect some user behaviors. For example, users may have already implemented their own plugins based on the current plugin framework. Are we sure we want to do this? Even if we want to do so, I think we may need to release a large version because of the incompatibility?
2. If we are certain that we want to do this, I think what we should do is setting up the mapping of different types of jobs and corresponding job plugins. For example, tensorflow job -> [svc, env], MPI job -> [ssh, svc], default -> [], but there is a problem that what if users want to specify the parameters of job plugins, this may need to add more user configurable parameters

I don't know if my understanding of this feature is correct and considerate. Please correct me and welcome discussion :)

[hzxuzhonghu]

cc @william-wang @Thor-wl

[k82cn]

I'm not sure why we need this feature? any background? The purpose of vcjob is to support all kind of high performance workloads by one CRD.

[Thor-wl]

I've not taken a deep investigation about how different compute framework define vcjob to adjust to Volcano. If the users of one framework define vcjob with same plugins, i think it is a good idea to let users just set "framework" field and Volcano adjust the plugins to the framework automatically.

[shinytang6]

I'm not sure why we need this feature? any background? The purpose of vcjob is to support all kind of high performance workloads by one CRD.

For volcano users (such as me), when I first used it, I would be confused with job plugins. Taking env setting as an example. I actually didn't know what the env plugin would do. From this point it is useful to block these information from users, which l think there is no conflict with 'support all kind of high performance workload by one CRD', it stills use one CRD definition Job. What we need to do is to automatically load the corresponding job plugins according to a new field such like "framework". The function of this field is only to help users identify and facilitate us to dynamically load job plugins.

[alcorj-mizar]

This is a moment that volcano walking from the scheduler framework to batch job framework. If we use "framework" field, it may still integration with project like kubeflow and just provide some automation when creating jobs. But if we use CRD to define different types of workloads, we are trying to reinvent the kubeflow.

[stale]

Hello 👋 Looks like there was no activity on this issue for last 90 days.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity for 60 days, this issue will be closed (we can always reopen an issue if we need!).

[stale]

Closing for now as there was no activity for last 60 days after marked as stale, let us know if you need this to be reopened! 🤗