DSL refactor #619

gaoning777 · 2019-01-03T22:37:11Z

combine two sanitize name function into one;
add more comments;
Aggregate certain function with similar functionality.

This change is

…build_conventional_artifact as a nested function

qimingj

Looks good. One major comment.

qimingj · 2019-01-04T04:03:23Z

sdk/python/kfp/dsl/_pipeline.py

@@ -108,7 +105,7 @@ def add_op(self, op: _container_op.ContainerOp, define_only: bool):
      op: An operator of ContainerOp or its inherited type.
    """

-    kubernetes_name = _make_kubernetes_name(op.human_name)
+    kubernetes_name = _sanitize_k8s_name(op.human_name)


I missed the previous change, so I am adding my thoughts here: one design goal is to hide k8s as much as possible in "dsl" layer, and push the k8s stuff to compiler (I used to call it "argo compiler). This way the DSL layer is more generic, and that's why there are "dsl" and "compiler" separate directories.

I feel like we don't have to sanitize the pipeline name here; We can store it as it is (so respect user's choice) and in compiler https://github.com/kubeflow/pipelines/blob/master/sdk/python/kfp/compiler/compiler.py#L457 we can sanitize there. That way, we can move the util to compiler since it is very k8s specific.

WDYT?

I'm all for the DSL hiding k8s. However, the pipeline sanitizes the name for operators which will be part of the final argo yaml. (in Pipeline.add_op() function).
I can sanitize the op names all in the compiler codes, though.

Ah yes. Sorry I mixed pipeline name with step name. I think we can sanitize the name in compiler too as you mentioned.

That way, a different "compiler" may choose to sanitize it in a different way, or even not sanitize it at all.

The Pipeline class (do not confuse with the @pipeline decorator) is a compiler helper class. It can probably moved to DSL compiler. It's only used during the compilation and it makes sense to sanitize the names/ids at this point. The original name remains untouched in op.human_name

The Pipeline class is used in the @pipeline decorator and it might not be a good idea to move the Pipeline class to the Compiler because the dsl would depend on the compiler otherwise. Then there would be a dependency loop since the compiler depends on the dsl library. Right?

The Pipeline class is used in the @pipeline decorator

This is an implementation detail. We've talked with @qimingj and AFAIK he agreed that it was a good idea to unlink them like we did with @python_component. I've prepared the code for the most common case, but there was a slight problem for multi-pipeline file compilation. If we deprecate that feature (it's not currently used anywhere) we can unlink them easier.

That way, a different "compiler" may choose to sanitize it in a different way, or even not sanitize it at all.

I agree with that. That's why I moved the sanitization code from the ContainerOp to the compiler-relater Pipeline class. As you remember, I tried to detangle the ContainerOp from the Pipeline even more, by making it not required.

Here is a proposal: Let's break the dependency between the DSL classes and the compiler by adding generic events/hooks for events like ContainerOp creation or @pipeline application. The compiler can set the handlers for those hooks to execute some compiler-specific code. This way, the DSL does not depend on the compiler.

@Ark-kun can you send a separate change and we can discuss more details? I feel it should be a separate change from this one. Small changes are more manageable.

The reason we had a "Pipeline" class are: 1) we need a "scope" that can record all ContainerOp objects. Hence the global Pipeline._default_pipeline variable. 2) Someday we can expose the Pipeline class to support dynamic pipeline construction in DSL (we don't expose that now because we want to limit the surfacing area and promote pipeline function).

Events/hooks is one option. We can compare the approaches by 1) Keep DSL as simple as it is now.
2) Favor simplicity for compiler provider. 3) Hopefully reduce or remove global variables.

…ineparam

qimingj

/lgtm

qimingj · 2019-01-04T19:59:31Z

/lgtm

qimingj · 2019-01-04T22:27:53Z

/lgtm

gaoning777 · 2019-01-04T22:40:48Z

/test kubeflow-pipeline-build-image

gaoning777 · 2019-01-04T23:45:16Z

/test kubeflow-pipeline-e2e-test

Ark-kun · 2019-01-05T01:43:12Z

sdk/python/kfp/compiler/compiler.py

+    # Sanitize operator names and param names
+    sanitized_ops = {}
+    for op in p.ops.values():
+      sanitized_name = K8sHelper.sanitize_k8s_name(op.name)


AFAIK op.name is already sanitized and made unique when it's being added to the pipeline.

Oh. You've removed that sanitization.
I'm not sure this is a good move. It's easier to fix the name before all the output references have been generated and passed around.

The motivation was to move the k8s related functions from DSL to compilers such that the implementations of another compiler in the future will not depend on K8s. The PR aims to moves the sanitization to the compiler.

The motivation was to move the k8s related functions from DSL to compilers such that the implementations of another compiler in the future will not depend on K8s. The PR aims to moves the sanitization to the compiler.

I agree with both those ideas. What's debatable is whether the Pipeline class belongs to the compiler or DSL.

Ark-kun · 2019-01-05T01:44:53Z

sdk/python/kfp/compiler/compiler.py

+        if param.op_name:
+          param.op_name = K8sHelper.sanitize_k8s_name(param.op_name)
+      for param in op.outputs.values():
+        param.name = K8sHelper.sanitize_k8s_name(param.name)


Are you sure this will work at this stage? The argument placeholders are probably already embedded into op.command and op.args as strings.

Good point. The op.args are handled in the _op_to_template function in this PR.
However, I have not handled the op.command. Do we already support parameterized commands?

Yes. We did. See 875efea

I'll update this PR for the command as well.

…h the whole serialized param str, Verify both param name and container name

qimingj · 2019-01-09T01:34:56Z

/lgtm

gaoning777 · 2019-01-09T01:35:16Z

/approve

k8s-ci-robot · 2019-01-09T01:35:20Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gaoning777

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~sdk/OWNERS~~ [gaoning777]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot · 2019-01-09T01:35:33Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: gaoning777

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~sdk/OWNERS~~ [gaoning777]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

* There is a bug in the code to detect completed jobs. For completed jobs the condition can be of type "Complete". As a result, we aren't properly detecting when jobs have been completed and rerunning them if needed. Related to kubeflow#762

* add init github actions * Remove old comments * Update test_kfp_samples.sh * add condition for venv

gaoning777 added 3 commits January 3, 2019 11:26

add comments

e64f862

relocate functions in compiler to aggregate similar functions; move _…

92d782f

…build_conventional_artifact as a nested function

reduce sanitize functions into one in the dsl.

50eac68

k8s-ci-robot added do-not-merge/work-in-progress labels Jan 3, 2019

k8s-ci-robot requested review from hongye-sun, qimingj and Ark-kun January 3, 2019 22:37

k8s-ci-robot added size/L labels Jan 3, 2019

gaoning777 changed the title ~~[WIP] DSL refactor~~ DSL refactor Jan 3, 2019

k8s-ci-robot removed do-not-merge/work-in-progress labels Jan 3, 2019

gaoning777 added 2 commits January 3, 2019 14:41

more comments

10bf3cc

Merge branch 'master' into dsl-refactor

aa1d38f

qimingj reviewed Jan 4, 2019

View reviewed changes

gaoning777 added 2 commits January 4, 2019 10:17

move all sanitization(op name, param name) from dsl to compiler

bd11209

sanitize pipelineparam name and op_name; remove format check in pipel…

fbc56c8

…ineparam

qimingj approved these changes Jan 4, 2019

View reviewed changes

k8s-ci-robot assigned qimingj Jan 4, 2019

k8s-ci-robot added the lgtm label Jan 4, 2019

remove unit test for pipelineparam op_name format checking

f257134

k8s-ci-robot removed the lgtm label Jan 4, 2019

k8s-ci-robot added the lgtm label Jan 4, 2019

fix bug: correctly replace input in the argument list

5a8c355

k8s-ci-robot removed lgtm labels Jan 4, 2019

fix bug: replace arguments with found ones

7baa2be

k8s-ci-robot added lgtm labels Jan 4, 2019

Ark-kun reviewed Jan 5, 2019

View reviewed changes

Merge branch 'master' into dsl-refactor

caf6581

k8s-ci-robot removed the lgtm label Jan 7, 2019

gaoning777 added 3 commits January 7, 2019 13:11

Merge branch 'master' into dsl-refactor

b0affb1

Sanitize the file_output keys, Matches the param in the args/cmds wit…

5c5ced5

…h the whole serialized param str, Verify both param name and container name

loosen the containerop and param name restrictions

7183f7a

k8s-ci-robot added the lgtm label Jan 9, 2019

k8s-ci-robot added the approved label Jan 9, 2019

k8s-ci-robot merged commit d3c4add into kubeflow:master Jan 9, 2019

Ark-kun mentioned this pull request Mar 8, 2019

SDK/DSL/Compiler - Fixed compilation when using ContainerOp.after #943

Merged

HumairAK pushed a commit to red-hat-data-services/data-science-pipelines that referenced this pull request Mar 11, 2024

Migrate Travis tests to Github Actions (kubeflow#619)

67c6d43

* add init github actions * Remove old comments * Update test_kfp_samples.sh * add condition for venv

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DSL refactor #619

DSL refactor #619

gaoning777 commented Jan 3, 2019 •

edited by jlewi

Loading

qimingj left a comment

qimingj Jan 4, 2019

gaoning777 Jan 4, 2019

qimingj Jan 4, 2019

gaoning777 Jan 4, 2019

Ark-kun Jan 5, 2019 •

edited

Loading

gaoning777 Jan 7, 2019

Ark-kun Jan 8, 2019

Ark-kun Jan 8, 2019

Ark-kun Jan 8, 2019

qimingj Jan 8, 2019

qimingj left a comment

qimingj commented Jan 4, 2019

qimingj commented Jan 4, 2019

gaoning777 commented Jan 4, 2019

gaoning777 commented Jan 4, 2019

Ark-kun Jan 5, 2019 •

edited

Loading

Ark-kun Jan 5, 2019

gaoning777 Jan 7, 2019

Ark-kun Jan 8, 2019

Ark-kun Jan 5, 2019

gaoning777 Jan 7, 2019

qimingj Jan 7, 2019

gaoning777 Jan 7, 2019

gaoning777 Jan 7, 2019

qimingj commented Jan 9, 2019

gaoning777 commented Jan 9, 2019

k8s-ci-robot commented Jan 9, 2019

k8s-ci-robot commented Jan 9, 2019

DSL refactor #619

DSL refactor #619

Conversation

gaoning777 commented Jan 3, 2019 • edited by jlewi Loading

qimingj left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Ark-kun Jan 5, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qimingj left a comment

Choose a reason for hiding this comment

qimingj commented Jan 4, 2019

qimingj commented Jan 4, 2019

gaoning777 commented Jan 4, 2019

gaoning777 commented Jan 4, 2019

Ark-kun Jan 5, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

qimingj commented Jan 9, 2019

gaoning777 commented Jan 9, 2019

k8s-ci-robot commented Jan 9, 2019

k8s-ci-robot commented Jan 9, 2019

gaoning777 commented Jan 3, 2019 •

edited by jlewi

Loading

Ark-kun Jan 5, 2019 •

edited

Loading

Ark-kun Jan 5, 2019 •

edited

Loading