diff --git a/docs/source/user_guide/pipeline/index.rst b/docs/source/user_guide/pipeline/index.rst index 62aa0774c..8fc200f8d 100644 --- a/docs/source/user_guide/pipeline/index.rst +++ b/docs/source/user_guide/pipeline/index.rst @@ -1,8 +1,8 @@ .. _pipeline: -######## -Pipeline -######## +######### +Pipelines +######### .. versionadded:: 2.8.0 diff --git a/docs/source/user_guide/pipeline/overview.rst b/docs/source/user_guide/pipeline/overview.rst index 823a3999e..6c2474911 100644 --- a/docs/source/user_guide/pipeline/overview.rst +++ b/docs/source/user_guide/pipeline/overview.rst @@ -1,20 +1,20 @@ Overview ******** -Oracle Cloud Infrastructure (OCI) Data Science Machine Learning (ML) Pipeline enables you to define and run an end-to-end machine learning orchestration covering all the steps of machine learning lifecycle that can be executed in a repeatable, continuous ML pipeline. +Oracle Cloud Infrastructure (OCI) Data Science Machine Learning (ML) Pipelines lets you define and run an end-to-end machine learning orchestration covering all the steps of machine learning lifecycle that can be executed in a repeatable, continuous ML pipeline. -Machine learning lifecycle is composed of data acquisition and extraction, data preparation, featurization, training model including algorithm selection and hyper-parameter tuning, model evaluation, deployment, and then monitoring the deployed model and retraining. +The machine learning lifecycle is composed of several steps: data acquisition and extraction, data preparation, featurization, model training including algorithm selection and hyper-parameter tuning, model evaluation, deployment, and finally monitoring the deployed model and possible retraining. Pipeline Step ============= -Pipeline step is a task in a pipeline. A pipeline step can be either a Data Science Job step or a Custom Script step. +A pipeline step is a task in a pipeline. A pipeline step can be either a Data Science Job step or a Custom Script step. Pipeline ======== -A pipeline is a workflow of tasks, called steps. Steps can run in sequence or in parallel, creating a Directed Acyclic Graph (DAG) of the steps. +A pipeline is a workflow of tasks, called steps. Steps can run in sequence or in parallel resulting in a Directed Acyclic Graph (DAG) of the steps. In a machine learning context, ML Pipelines provide a workflow of data import, data transformation, model training, and model evaluation. diff --git a/docs/source/user_guide/pipeline/pipeline.rst b/docs/source/user_guide/pipeline/pipeline.rst index e6e998029..6794d6fe8 100644 --- a/docs/source/user_guide/pipeline/pipeline.rst +++ b/docs/source/user_guide/pipeline/pipeline.rst @@ -196,21 +196,21 @@ You can call the ``run()`` method of the ``Pipeline`` instance to launch a new P It returns a ``PipelineRun`` instance. -The ``.run()`` method gives you the option to override the configurations in a pipeline run. It takes the following optional parameters: +The ``run()`` method gives you the option to override the configurations in a pipeline run. It takes the following optional parameters: -- ``display_name: str, optional``. Defaults to ``None``. The display name of the run. -- ``project_id: str, optional``. Defaults to ``None``. The project id to override the one defined previously. -- ``compartment_id: str, optional``. Defaults to ``None``. The compartment id to override the one defined previously. -- ``configuration_override_details: dict, optional``. Defaults to ``None``. The configuration details dictionary to override the one defined previously. The ``configuration_override_details`` contains the following keys: + - ``display_name: str, optional``. Defaults to ``None``. The display name of the run. + - ``project_id: str, optional``. Defaults to ``None``. The project id to override the one defined previously. + - ``compartment_id: str, optional``. Defaults to ``None``. The compartment id to override the one defined previously. + - ``configuration_override_details: dict, optional``. Defaults to ``None``. The configuration details dictionary to override the one defined previously. The ``configuration_override_details`` contains the following keys: ``type``: str, only ``DEFAULT`` is allowed; ``environment_variables``: dict, the environment variables; ``command_line_arguments``: str, the command line arguments; ``maximum_runtime_in_minutes``: int, the maximum runtime allowed in minutes. -- ``log_configuration_override_details: dict, optional``. Defaults to ``None``. The log configuration details dictionary to override the one defined previously. -- ``step_override_details: list[PipelineStepOverrideDetails], optional``. Defaults to ``None``. The step details list to override the one defined previously. -- ``free_form_tags: dict(str, str), optional``. Defaults to ``None``. The free from tags dictionary to override the one defined previously. -- ``defined_tags: dict(str, dict(str, object)), optional``. Defaults to ``None``. The defined tags dictionary to override the one defined previously. -- ``system_tags: dict(str, dict(str, object)), optional``. Defaults to ``None``. The system tags dictionary to override the one defined previously. + - ``log_configuration_override_details: dict, optional``. Defaults to ``None``. The log configuration details dictionary to override the one defined previously. + - ``step_override_details: list[PipelineStepOverrideDetails], optional``. Defaults to ``None``. The step details list to override the one defined previously. + - ``free_form_tags: dict(str, str), optional``. Defaults to ``None``. The free from tags dictionary to override the one defined previously. + - ``defined_tags: dict(str, dict(str, object)), optional``. Defaults to ``None``. The defined tags dictionary to override the one defined previously. + - ``system_tags: dict(str, dict(str, object)), optional``. Defaults to ``None``. The system tags dictionary to override the one defined previously. .. code-block:: python3 @@ -233,11 +233,11 @@ Use the ``from_ocid()`` method from the ``Pipeline`` class to load an existing p Visualize ========= -Use the ``.show()`` method on the ``Pipeline`` instance to visualize the pipeline in a graph. +Use the ``show()`` method on the ``Pipeline`` instance to visualize the pipeline in a graph. -The ``.show()`` method takes the following optional parameter: +The ``show()`` method takes the following optional parameter: -- ``rankdir: (str, optional)``. Defaults to ``TB``. The allowed values are ``TB`` or ``LR``. This parameter is applicable only for ``graph`` mode and it renders the direction of the graph as either top to bottom (TB) or left to right (LR). + - ``rankdir: (str, optional)``. Defaults to ``TB``. The allowed values are ``TB`` or ``LR``. This parameter is applicable only for ``graph`` mode and it renders the direction of the graph as either top to bottom (TB) or left to right (LR). .. code-block:: python3 @@ -252,11 +252,11 @@ Below is an example of the output. Delete ====== -Use the ``.delete()`` method on the ``Pipeline`` instance to delete a pipeline. It takes the following optional parameters: +Use the ``delete()`` method on the ``Pipeline`` instance to delete a pipeline. It takes the following optional parameters: -- ``delete_related_pipeline_runs: (bool, optional)``. Specify whether to delete related PipelineRuns or not. Defaults to ``True``. -- ``delete_related_job_runs: (bool, optional)``. Specify whether to delete related JobRuns or not. Defaults to ``True``. -- ``max_wait_seconds: (int, optional)``. The maximum time to wait, in seconds. Defaults to ``1800``. + - ``delete_related_pipeline_runs: (bool, optional)``. Specify whether to delete related PipelineRuns or not. Defaults to ``True``. + - ``delete_related_job_runs: (bool, optional)``. Specify whether to delete related JobRuns or not. Defaults to ``True``. + - ``max_wait_seconds: (int, optional)``. The maximum time to wait, in seconds. Defaults to ``1800``. A pipeline can only be deleted when its associated pipeline runs are all deleted, or alternatively, set the parameter ``delete_related_pipeline_runs`` to delete all associated runs in the same operation. diff --git a/docs/source/user_guide/pipeline/pipeline_run.rst b/docs/source/user_guide/pipeline/pipeline_run.rst index 712369a60..c4291a6d5 100644 --- a/docs/source/user_guide/pipeline/pipeline_run.rst +++ b/docs/source/user_guide/pipeline/pipeline_run.rst @@ -11,13 +11,13 @@ With a ``PipelineRun`` instance, you can watch the status of the run and stream Watch status ============ -Use the ``.show()`` method of the ``PipelineRun`` instance to watch the status of pipeline run. +Use the ``show()`` method of the ``PipelineRun`` instance to watch the status of pipeline run. -The ``.show()`` method takes the following optional parameter: +The ``show()`` method takes the following optional parameter: -- ``mode: (str, optional)``. Defaults to ``graph``. The allowed values are ``text`` or ``graph``. This parameter renders the current status of pipeline run as either ``text`` or ``graph``. -- ``wait: (bool, optional)``. Defaults to ``False`` and it only renders the current status of each step run in graph. If set to ``True``, it renders the current status of each step run until the entire pipeline is complete. -- ``rankdir: (str, optional)``. Defaults to ``TB``. The allowed values are ``TB`` or ``LR``. This parameter is applicable only for ``graph`` mode and it renders the direction of the graph as either top to bottom (TB) or left to right (LR). + - ``mode: (str, optional)``. Defaults to ``graph``. The allowed values are ``text`` or ``graph``. This parameter renders the current status of pipeline run as either ``text`` or ``graph``. + - ``wait: (bool, optional)``. Defaults to ``False`` and it only renders the current status of each step run in graph. If set to ``True``, it renders the current status of each step run until the entire pipeline is complete. + - ``rankdir: (str, optional)``. Defaults to ``TB``. The allowed values are ``TB`` or ``LR``. This parameter is applicable only for ``graph`` mode and it renders the direction of the graph as either top to bottom (TB) or left to right (LR). To watch the live update of each step run status in text until the entire pipeline is complete @@ -49,8 +49,8 @@ Below is an example of the output. Monitor Logs ============ -Use the ``.watch()`` method on the ``PipelineRun`` instance to stream the service, custom, or consolidated log of the pipeline run. -The ``.watch()`` method takes the following optional parameters: +Use the ``watch()`` method on the ``PipelineRun`` instance to stream the service, custom, or consolidated log of the pipeline run. +The ``watch()`` method takes the following optional parameters: - ``steps: (list, optional)``. Defaults to ``None`` and streams the log of the pipeline run. If a list of the step names is provided, the method streams the log of the specified pipeline step runs. - ``log_type: (str, optional)``. Defaults to ``None``. The allowed values are ``custom_log``, ``service_log``, or ``None``. If ``None`` is provided, the method streams both service and custom logs. @@ -122,7 +122,7 @@ Use the ``from_ocid()`` method from the ``PipelineRun`` class to load an existin Cancel ====== -Use the ``.cancel()`` method on the ``PipelineRun`` instance to cancel a pipeline run. +Use the ``cancel()`` method on the ``PipelineRun`` instance to cancel a pipeline run. Pipeline Runs can only be canceled when they are in the ACCEPTED or IN_PROGRESS state. @@ -135,10 +135,10 @@ Pipeline Runs can only be canceled when they are in the ACCEPTED or IN_PROGRESS Delete ====== -Use the ``.delete()`` method on the ``PipelineRun`` instance to delete a pipeline run. It takes the following optional parameter: +Use the ``delete()`` method on the ``PipelineRun`` instance to delete a pipeline run. It takes the following optional parameter: -- ``delete_related_job_runs: (bool, optional)``. Specify whether to delete related JobRuns or not. Defaults to ``True``. -- ``max_wait_seconds: (int, optional)``. The maximum time to wait in seconds. Defaults to ``1800``. + * ``delete_related_job_runs: (bool, optional)``. Specify whether to delete related JobRuns or not. Defaults to ``True``. + * ``max_wait_seconds: (int, optional)``. The maximum time to wait in seconds. Defaults to ``1800``. Pipeline runs can only be deleted when they are already in a SUCCEEDED, FAILED, or CANCELED state. diff --git a/docs/source/user_guide/pipeline/quick_start.rst b/docs/source/user_guide/pipeline/quick_start.rst index 35662621e..05ce53de2 100644 --- a/docs/source/user_guide/pipeline/quick_start.rst +++ b/docs/source/user_guide/pipeline/quick_start.rst @@ -5,9 +5,9 @@ Quick Start ADS Python SDK ============== -The following sections provide sample code to define, create and run a pipeline, and watch the pipeline run status. +The following sections provide sample code to define, create, and run a pipeline, including a visualization to track the pipeline run status. -The following example shows creating and runnning a pipeline with multiple steps. The steps, ``step_1`` and ``step_2`` run in parallel and +The following example shows creating and runnning a pipeline with multiple steps. The steps ``step_1`` and ``step_2`` run in parallel and ``step_3`` runs after ``step_1`` and ``step_2`` are complete. .. figure:: figures/quick_start_example.png @@ -353,7 +353,7 @@ Alternatively, to run an existing pipeline, provide the pipeline OCID with the ` Visualize --------- -To visualize a pipeline in a graph, provide the pipeline OCID +To visualize a pipeline in a graph, use the pipeline OCID .. code-block:: shell @@ -362,7 +362,7 @@ To visualize a pipeline in a graph, provide the pipeline OCID Watch status ------------ -To watch the status of pipeline run, provide the pipeline run OCID +To watch the status of pipeline run, use the pipeline run OCID .. admonition:: Tip @@ -383,7 +383,7 @@ Below is an example of watching the status of pipeline run in ``text`` mode Monitor logs ------------ -To monitor a pipeline run, provide the pipeline run OCID +To monitor a pipeline run, use the pipeline run OCID. .. admonition:: Tip @@ -404,7 +404,7 @@ Below is an example of viewing the last 10 consolidated logs with ``tail`` Cancel ------ -To cancel a pipeline run, provide the pipeline run OCID +To cancel a pipeline run, use the pipeline run OCID .. code-block:: shell @@ -419,7 +419,7 @@ Delete Get more information about deleting pipelines and pipeline runs by running ``%pipeline delete -h`` -To delete a pipeline run, provide the pipeline run OCID +To delete a pipeline run, use the pipeline run OCID .. code-block:: shell @@ -427,7 +427,7 @@ To delete a pipeline run, provide the pipeline run OCID Data Science Pipeline Runs can only be deleted when they are in the `SUCCEEDED`, `FAILED`, or `CANCELED` state. -To delete a pipeline, provide the pipeline OCID +To delete a pipeline, use the pipeline OCID .. code-block:: shell