refactor(controller): make ds build a job

[anda-ren]

Description

make dataset build feature be a job

Core components introduced

1.ContainerSpec

Expose a defination language for custome container job

public class ContainerSpec {

    String image;
    String[] cmds;
    String[] entrypoint;

}

2.VirtualJobLoader

Loads controller defined jobs that contains no swcli model or swcli runtimes

public class VirtualJobLoader {

    final String virtualJobPath;

    public VirtualJobLoader(@Value("${sw.job.virtual-specs}") String virtualJobPath) {
        this.virtualJobPath = virtualJobPath;
    }

    public String loadJobStepSpecs(String jobName) throws IOException {
        return FileResourceUtil.getFileContent(virtualJobPath, MessageFormat.format("virtual/model/{0}.yaml", jobName));
    }

}

3.ContainerSpecification

Define the standard components that each type of job should expose to container based scheduler

public interface ContainerSpecification {

    Map<String, String> getContainerEnvs();

    ContainerCommand getCmd();

    String getImage();


}

Modules

• UI
• Controller
• Agent
• Client
• Python-SDK
• Others

Checklist

• run code format and lint check
• add unit test
• add necessary doc

[codecov]

Codecov Report

Merging #2626 (5b9c73a) into main (712af6f) will decrease coverage by 17.28%.
Report is 1 commits behind head on main.
The diff coverage is 66.43%.

@@              Coverage Diff              @@
##               main    #2626       +/-   ##
=============================================
- Coverage     91.40%   74.12%   -17.28%     
- Complexity        0     2774     +2774     
=============================================
  Files           105      382      +277     
  Lines         12202    13521     +1319     
  Branches         15     1556     +1541     
=============================================
- Hits          11153    10023     -1130     
- Misses         1049     2788     +1739     
- Partials          0      710      +710     

Flag	Coverage Δ
console	80.00% <ø> (ø)
controller	74.09% <66.43%> (?)
standalone	?
unittests	80.00% <ø> (-11.41%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files Changed	Coverage Δ
...java/ai/starwhale/mlops/api/DatasetController.java	74.84% <0.00%> (ø)
...domain/dataset/build/mapper/BuildRecordMapper.java	0.00% <ø> (ø)
...whale/mlops/domain/job/converter/JobConverter.java	72.89% <0.00%> (ø)
...va/ai/starwhale/mlops/domain/job/po/JobEntity.java	100.00% <ø> (ø)
...a/ai/starwhale/mlops/domain/job/spec/StepSpec.java	90.00% <ø> (ø)
...hale/mlops/domain/model/po/ModelVersionEntity.java	50.00% <0.00%> (ø)
.../impl/CustomTaskContainerSpecificationFactory.java	0.00% <0.00%> (ø)
...r/impl/OnlineEvaluationContainerSpecification.java	0.00% <0.00%> (ø)
...impl/OnlineEvaluationTaskSpecificationFactory.java	0.00% <0.00%> (ø)
.../impl/RuntimeDockerizedContainerSpecification.java	0.00% <0.00%> (ø)
... and 31 more

... and 444 files with indirect coverage changes

[jialeicui]

LGTM