Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[improve] More pythonic way for parameter local_params in class task #53

Merged
merged 7 commits into from
Dec 20, 2022

Conversation

jieguangzhou
Copy link
Member

@jieguangzhou jieguangzhou commented Dec 16, 2022

Brief Summary of The Change

close: #21

# [start workflow_declare]
 r"""
 A tutorial example set local parameter in pydolphinscheduler.
 Method 1:
     task = Shell(..., input_params={"input":"a"}, output_params={"output": "b"})
 Method 2:
     task = Shell(...)
     task.add_in("input", "a")
     task.add_out("output", "b")
 """

 from pydolphinscheduler.core.parameter import ParameterType
 from pydolphinscheduler.core.workflow import Workflow
 from pydolphinscheduler.tasks.shell import Shell

 with Workflow(name="local_parameter_example", release_state="offline") as workflow:

     # Add in task __init__ func
     task_1 = Shell(
         name="task_1",
         command="echo hello pydolphinscheduler",
         input_params={
             "value_VARCHAR": "abc",
             "value_INTEGER": 123,
             "value_FLOAT": 0.1,
             "value_BOOLEAN": True,
         },
         output_params={
             "value_EMPTY": None,
         },
     )

     # Add in task instance
     task_2 = Shell(name="task_2", command="echo hello pydolphinscheduler")

     task_2.add_in("value_VARCHAR", "abc")
     task_2.add_in("value_INTEGER", 123)
     task_2.add_in("value_FLOAT", 0.1)
     task_2.add_in("value_BOOLEAN", True)
     task_2.add_out("value_EMPTY")

     # Task 1 is the same as task 2

     # Other parameter types corresponding to DolphinScheduler
     task_3 = Shell(
         name="task_3",
         command="echo hello pydolphinscheduler",
         input_params={
             "value_LONG": ParameterType.LONG("1000000"),
             "value_DATE": ParameterType.DATE("2022-01-02"),
             "value_TIME": ParameterType.TIME("2022-01-01"),
             "value_TIMESTAMP": ParameterType.TIMESTAMP(123123124125),
             "value_LIST": ParameterType.LIST("123123"),
         },
         output_params={
             "output_INTEGER": ParameterType.INTEGER(100),
             "output_LIST": ParameterType.LIST(),
         },
     )

     workflow.submit()
 # [end workflow_declare]

image

Pull Request checklist

I confirm that the following checklist has been completed.

  • Add/Change test cases for the changes.
  • Add/Change the related documentation, should also change docs/source/config.rst when you change file default_config.yaml.
  • (Optional) Add your change to UPDATING.md when it is an incompatible change.

Copy link
Member

@zhongjiajie zhongjiajie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please remember change the example doc in the end

src/pydolphinscheduler/examples/parameter_example.py Outdated Show resolved Hide resolved
docs/source/tutorial.rst Outdated Show resolved Hide resolved
src/pydolphinscheduler/core/task.py Outdated Show resolved Hide resolved
src/pydolphinscheduler/examples/local_parameter_example.py Outdated Show resolved Hide resolved
src/pydolphinscheduler/examples/local_parameter_example.py Outdated Show resolved Hide resolved
src/pydolphinscheduler/examples/local_parameter_example.py Outdated Show resolved Hide resolved
tests/core/test_task.py Show resolved Hide resolved
src/pydolphinscheduler/core/task.py Show resolved Hide resolved
Copy link
Member

@zhongjiajie zhongjiajie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Others LGTM

src/pydolphinscheduler/core/task.py Outdated Show resolved Hide resolved
src/pydolphinscheduler/core/task.py Show resolved Hide resolved
src/pydolphinscheduler/core/task.py Outdated Show resolved Hide resolved
src/pydolphinscheduler/core/yaml_workflow.py Outdated Show resolved Hide resolved
docs/source/concept.rst Outdated Show resolved Hide resolved
Copy link
Member

@zhongjiajie zhongjiajie left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

.gitignore Show resolved Hide resolved
@jieguangzhou jieguangzhou merged commit 68d2670 into apache:main Dec 20, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[improve] More pythonic way for parameter local_params in class task
2 participants