Job submission v2 #2260

abuccts · 2019-03-04T07:55:21Z

Back-end part for job submission v2 #2232.

Work Items

[Docs] Finalize protocol spec: merged pai-proto branch.
[Docs] Add new yaml files that follow new protocol: [Docs] add new yaml files that follow pai protocol #2292.
[Rest Server] Protocol yaml parsing, validation and rendering: [Rest Server] Protocol yaml parsing, validation, and rendering #2274.
[Rest Server] Convert protocol to framework launcher description file: [Rest Server] Convert protocol to framework launcher description #2291.
[Rest Server] Refactor and update rest API v2: [Rest Server] Refactor and update rest API v2 #2329.

Exit Criteria

BERT end to end demo.
Distributed end to end example.

* Update PAI protocol yaml Update PAI protocol template and example according to the discussion.

Update protocol spec.

Add pai-protocol for job submission v2.

fanyangCS · 2019-03-04T10:11:23Z

docs/pai-protocol.yaml

+    taskRoles:
+      <name>: String, required # Should be in taskRoles
+        preCommands:
+          - String, required # execute before $$commands$$


Line 59, 61, 211 are still using old reference way $$. Please fix them.

Remove previous `$$...$$` references

fanyangCS · 2019-03-04T11:16:39Z

docs/pai-protocol.yaml

+  <param2>: value2Type # Can be referenced by `<% $parameters.param1 %>`, `<% $parameters.param2 %>`
+
+jobRetryCount: Integer, optional # Default is 0
+taskRoles:


please explain what taskRoles is. e.g., taskRoles defines the type of task (i.e., container). In OpenPAI protocol, one job can has one or multiple types of task, each in turn may has one or multiple tasks.

fanyangCS · 2019-03-04T11:20:05Z

docs/pai-protocol.yaml

+  <name>: String, required  # Name of the taskRole
+   protocol_version: String, optional # Protocol version, default is 2
+    instances: Integer, optional # Default is 1, instances of a taskRole
+    completion:


please explain what is "completion". e.g., Completion specifies how a job state changes in case of a task completion.
Please provide a link to our document for details.

fanyangCS · 2019-03-04T11:22:25Z

docs/pai-protocol.yaml

+    dockerImage: String, required # Should reference to a dockerimage defined in prerequisites
+    data: String, optional # Should reference to data defined in prerequisites, the whole data object in this taskRole can be referenced as `$data`
+    output: String, optional # Should reference to output defined in prerequisites, the whole output object in this taskRole can be referenced as `$output`
+    script: String, optional # Should reference to script defined in prerequisites, the whole script object in this taskRole can be referenced as `$script`


Scope of the reference: the reference is only valid inside this taskRole. User cannot reference data/output/script from another taskRole

fanyangCS · 2019-03-04T11:24:39Z

docs/pai-protocol.yaml

+# to handle that a component may interact with different component differently, user is encouraged to place the codes handling such difference in the "deployments" field.
+# e.g., a job may get input data through wget, hdfc -dfs cp, copy, or just directly read from remote storage. This logic can be placed here.
+# in summary, the deployments field is responsible to make sure the job to run properly in a deployment specific runtime environment.
+# one could have many deployments, but only the first deployment can be activated at runtime. User can choose the deployment at job submission time.


"but only the first deployment can be activated at runtime." but only one deployment can be activated at runtime (specified in "default")

* Change protocolVersion type into number. * Reserve protocolVersion field for job and prerequisite types. * Merge attachments field into defaults.

Refine docs according to comments.

coveralls · 2019-04-01T08:25:13Z

Coverage increased (+0.7%) to 53.287% when pulling 2b47b07 on feature/submission-v2 into 50ebc74 on master.

coveralls · 2019-04-01T08:25:14Z

Coverage increased (+0.7%) to 53.314% when pulling 65fa657 on feature/submission-v2 into 50ebc74 on master.

Gerhut · 2019-04-01T08:28:42Z

src/rest-server/src/middlewares/v2/protocol.js

+    }
+    let entrypoint = '';
+    commands = commands.map((command) => command.trim()).join(' && ');
+    const tokens = mustache.parse(commands, ['<%', '%>']);


There is a bug in the version of mustache we are using is that the cache key of the parsed template is only the template source, w/o the delimiters, so if we are using different delimiters in the same cache, there may be parsing mistake occurs when parsing the same text with different delimiters.

sterowang · 2019-04-01T08:36:54Z

marketplace/job/caffe2_resnet50.yaml

+      memoryMB: 8192
+      gpu: 1
+    commands:
+      - python resnet50_trainer.py --train_data null


Are we changed job description file to v2 format here?

It has been changed in #2292, hasn't been merged yet.
Added in a separate directory protocol-marketplace and keep the original one.

renamed to marketplace-v2

Create new mustache writer instance to avoid conflict.

sterowang · 2019-04-01T08:55:09Z

src/rest-server/src/routes/v2/index.js


-router.use('/template', templateRouter);
-router.use('/jobs', jobV2Router);
+router.use('/jobs', jobRouter);



Synced with Yifan, when we decide to switch to v2 api, we need to redirect the all remaining APIs to v1 before that time point so that the all components will set "api/v2" as the base URI.

will do in future PRs

sterowang

Update.

sterowang · 2019-04-01T08:58:55Z

docs/pai-job-protocol.yaml

+    script: tensorflow_cnnbenchmarks
+    extraContainerOptions:
+      shmMB: 64
+    resourcePerInstance:


resourcePerInstance [](start = 4, length = 19)

resorucesPerInstance? #WontFix

The original one is resourcePerInstance, should we change to resourcesPerInstance? #WontFix

As it contains multiple children, maybe resourcesPerInstance is more grammer correct? @fanyangCS can how do you think about it?

In reply to: 270776814 [](ancestors = 270776814)

No need to change, Yarn also call it "resource".

In reply to: 270779984 [](ancestors = 270779984,270776814)

sterowang · 2019-04-01T09:03:53Z

src/rest-server/src/middlewares/v2/protocol.js

+  if ('prerequisites' in protocolObj) {
+    for (let item of protocolObj.prerequisites) {
+      prerequisites[item.type][item.name] = item;
+    }


Will you check the name duplication for the same type of prerequisites?

Nope, the prerequisite with same type and same type will overwrite the previous one.

Do you think is it useful to prompt users name duplicated? Maybe users sometimes forget to change the name when they create a new prerequisites by copying and modify the existing ones?

In reply to: 270776412 [](ancestors = 270776412)

Updated, will throw bad request error if duplicate prerequisites or deployments exist.

revoking review

sterowang · 2019-04-01T09:12:07Z

docs/pai-job-protocol.yaml

+      memoryMB: Integer, required
+      gpu: Integer, required
+      ports:
+        <portLabel>: Integer, required, default is 0 # Only for host network, portLabel string in ^[A-Za-z0-9\-._~]+$ format.


required [](start = 30, length = 8)

Optional? #Resolved

sterowang · 2019-04-01T09:12:45Z

docs/pai-job-protocol.yaml

+      cpu: Integer, required
+      memoryMB: Integer, required
+      gpu: Integer, required
+      ports:


Units? #Resolved

sterowang · 2019-04-01T09:17:14Z

docs/pai-job-protocol.yaml

+    dockerImage: String, required # Should reference to a dockerimage defined in prerequisites.
+    # Scope of the reference `$data`, '$output', `$script`: the reference is only valid inside this task role.
+    # User cannot reference them from another task role. Reference for `$parameters` is global and shared among task roles.
+    data: String, optional # Should reference to data defined in prerequisites, the whole data object in this task role can be referenced as `$data`.


Should reference to data defined in prerequisites, the whole data object in this task role can be referenced as $data. [](start = 29, length = 120)

What about describe as "Select data defined in prerequisites, target can be referenced as $data in this task role? #Resolved

sterowang · 2019-04-01T09:18:09Z

docs/pai-job-protocol.yaml

+    data: String, optional # Should reference to data defined in prerequisites, the whole data object in this task role can be referenced as `$data`.
+    output: String, optional # Should reference to output defined in prerequisites, the whole output object in this task role can be referenced as `$output`.
+    script: String, optional # Should reference to script defined in prerequisites, the whole script object in this task role can be referenced as `$script`.
+    extraContainerOptions:


And description for output and script? @fanyangCS what your opinion? #Resolved

sterowang

🕐

Update protocol docs.

Check duplicate prerequisites and deployments in protocol.

sterowang

* Upgrade yaml to protocol in marketplace Upgrade yaml to protocol in marketplace. * Add descriptions for example yamls in marketplace Add descriptions for example yamls in marketplace. * Remove unit test using outdated yaml Remove unit test using outdated yaml. * Restore former marketplace and add a new one Restore former marketplace and add a protocol marketplace. * Update Update. * Rename protocol marketplace to marketplace v2 Rename protocol marketplace to marketplace v2.

Replace webhdfs callback with async functions.

Revert changes in previous marketplace, conflicts in #2292.

Update hdfs promises.

* Add protocol submission plugin Add protocol submission plugin: * import protocol yaml file * view/edit protocol yaml file with monaco editor Known issues: * monaco editor cannot be bundled by webpack for plugin * editor worker has the wrong base url * language highlight and some editor styles do not work in plugin * Update webpack config file Update webpack config file. * Update protocol form Update protocol form. * Add webpack public path Add webpack public path. * Support clone from job list Support clone from job list. * fix webpack public url * fix cors web worker issue * Update according to comments Update according to comments. * Use immutable update to clone state Use immutable update to clone state. * Add componentDidMount Add `componentDidMount`. * Update componentDidMount Update `componentDidMount`. * Add bootstrap css Add bootstrap css. * Fix monaco editor line number style Pick monaco-hack style and fix line number margin. * Use react-bootstrap Use react-bootstrap. * Add loading spinner Add loading spinner. * Fix tslint issue Fix tslint issue. * Use css modules in bootstrap styles Use css modules in bootstrap styles. * Remove string literal in styles Remove string literal in styles. * Add parameters textfield Add parameters textfield. * Fix tslint errors Fix tslint errors. * Update Update. * Update Co-Authored-By: abuccts <xiongyf@yandex.com> * Update Update. * Add features in monaco plugin Add one feature in monaco plugin to reduce bundle size, due to a bug in https://github.com/Microsoft/monaco-editor-webpack-plugin/blob/v1.7.0/index.js#L38. * Update Update.

abuccts · 2019-04-15T03:07:19Z

🚢, close #2232.

- [Docs] Finalize protocol spec: merged pai-proto branch. - [Docs] Add new yaml files that follow new protocol: #2292. - [Rest Server] Protocol yaml parsing, validation and rendering: #2274. - [Rest Server] Convert protocol to framework launcher description file: #2291. - [Rest Server] Refactor and update rest API v2: #2329. - [Webportal] Add submission protocol plugin: #2461.

yqwang-ms and others added 17 commits January 15, 2019 20:47

[pai-protocal]: Qi's initial

737b983

[pai-protocal]: Qi's refine

2f053e5

[pai-protocal]: Fan's refine

7c67b76

[pai-protocal]: Yuqi's refine

2425f27

rename to pai-protocol.yaml

28d6a92

refined according to Bin's suggestion

3043a64

extraContainerOptions

c42d34f

minor

a48d904

refined

33936b7

correction to comments

e4ee278

refined

c9082a5

rename taskCount to Instances

c707409

refine

d10623c

updates

85be3ae

Update PAI protocol yaml (#2086)

291ef10

* Update PAI protocol yaml Update PAI protocol template and example according to the discussion.

Update protocol spec

2ca93c9

Update protocol spec.

Merge pai-protocol into feature/submission-v2

95ad9fe

Add pai-protocol for job submission v2.

abuccts added the new feature label Mar 4, 2019

abuccts added this to the Marketplace Demo milestone Mar 4, 2019

Merge master into feature/submission-v2

97b9362

fanyangCS reviewed Mar 4, 2019

View reviewed changes

Remove previous references

9f8adc2

Remove previous `$$...$$` references

fanyangCS suggested changes Mar 4, 2019

View reviewed changes

abuccts added 2 commits March 6, 2019 15:27

Unify protocol version

ba9532c

* Change protocolVersion type into number. * Reserve protocolVersion field for job and prerequisite types. * Merge attachments field into defaults.

Refine docs according to comments.

d946fe5

Refine docs according to comments.

This was referenced Mar 8, 2019

[Rest Server] Protocol yaml parsing, validation, and rendering #2274

Merged

[Rest Server] Convert protocol to framework launcher description #2291

Merged

[Docs] add new yaml files that follow pai protocol #2292

Merged

abuccts requested review from fanyangCS and sterowang March 8, 2019 12:00

Gerhut reviewed Apr 1, 2019

View reviewed changes

sterowang reviewed Apr 1, 2019

View reviewed changes

Create new mustache writer instance

88d1c93

Create new mustache writer instance to avoid conflict.

Gerhut approved these changes Apr 1, 2019

View reviewed changes

sterowang reviewed Apr 1, 2019

View reviewed changes

sterowang previously approved these changes Apr 1, 2019

View reviewed changes

Update

44cd520

Update.

sterowang reviewed Apr 1, 2019

View reviewed changes

sterowang suggested changes Apr 1, 2019

View reviewed changes

abuccts added 2 commits April 1, 2019 17:51

Update protocol docs

44d0294

Update protocol docs.

Check duplicate prerequisites and deployments

c67075c

Check duplicate prerequisites and deployments in protocol.

sterowang approved these changes Apr 1, 2019

View reviewed changes

abuccts and others added 5 commits April 1, 2019 21:34

Replace webhdfs callback with async functions

f9929c2

Replace webhdfs callback with async functions.

Revert changes in previous marketplace

cc73e55

Revert changes in previous marketplace, conflicts in #2292.

Update hdfs promises

7fc7072

Update hdfs promises.

fanyangCS approved these changes Apr 12, 2019

View reviewed changes

abuccts merged commit abe33b2 into master Apr 15, 2019

abuccts deleted the feature/submission-v2 branch August 19, 2019 05:54

Job submission v2 #2260

Job submission v2 #2260

Conversation

abuccts commented Mar 4, 2019 • edited

Work Items

Exit Criteria

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

coveralls commented Apr 1, 2019

coveralls commented Apr 1, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sterowang left a comment

Choose a reason for hiding this comment

sterowang Apr 1, 2019 • edited

Choose a reason for hiding this comment

abuccts Apr 1, 2019 • edited by sterowang

Choose a reason for hiding this comment

sterowang Apr 1, 2019 • edited by abuccts

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sterowang Apr 1, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sterowang Apr 1, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sterowang Apr 1, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sterowang Apr 1, 2019 • edited by abuccts

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sterowang left a comment

Choose a reason for hiding this comment

sterowang left a comment

Choose a reason for hiding this comment

abuccts commented Apr 15, 2019

abuccts commented Mar 4, 2019 •

edited

coveralls commented Apr 1, 2019 •

edited

sterowang Apr 1, 2019 •

edited

abuccts Apr 1, 2019 •

edited by sterowang

sterowang Apr 1, 2019 •

edited by abuccts

sterowang Apr 1, 2019 •

edited

sterowang Apr 1, 2019 •

edited

sterowang Apr 1, 2019 •

edited

sterowang Apr 1, 2019 •

edited by abuccts