From 0fc2ab0b76a5c5b916d30c06028ddf8d7ac3e773 Mon Sep 17 00:00:00 2001 From: Future-Outlier Date: Sat, 11 Nov 2023 15:29:36 +0800 Subject: [PATCH] [Docs] Sensor Agent Doc (#4195) * sensor agent doc Signed-off-by: Future Outlier * sensor agent index rst Signed-off-by: Future Outlier --------- Signed-off-by: Future Outlier Co-authored-by: Future Outlier Co-authored-by: Kevin Su --- rsts/deployment/agents/bigquery.rst | 51 ++++++---- rsts/deployment/agents/index.rst | 10 ++ rsts/deployment/agents/sensor.rst | 147 ++++++++++++++++++++++++++++ 3 files changed, 189 insertions(+), 19 deletions(-) create mode 100644 rsts/deployment/agents/sensor.rst diff --git a/rsts/deployment/agents/bigquery.rst b/rsts/deployment/agents/bigquery.rst index cf01e0b7bd..ac6ec896bb 100644 --- a/rsts/deployment/agents/bigquery.rst +++ b/rsts/deployment/agents/bigquery.rst @@ -20,30 +20,39 @@ Specify agent configuration .. tabs:: - .. group-tab:: Flyte binary + .. group-tab:: Flyte binary - Edit the relevant YAML file to specify the agent. + Edit the relevant YAML file to specify the agent. - .. code-block:: yaml - :emphasize-lines: 7,11 + .. code-block:: bash - tasks: - task-plugins: - enabled-plugins: - - container - - sidecar - - k8s-array - - bigquery - default-for-task-types: - - container: container - - container_array: k8s-array - - bigquery_query_job_task: agent-service + kubectl edit configmap flyte-sandbox-config -n flyte - .. group-tab:: Flyte core + .. code-block:: yaml + :emphasize-lines: 7,11,16 + + tasks: + task-plugins: + enabled-plugins: + - container + - sidecar + - k8s-array + - agent-service + default-for-task-types: + - container: container + - container_array: k8s-array + - bigquery_query_job_task: agent-service + + plugins: + agent-service: + supportedTaskTypes: + - bigquery_query_job_task + + .. group-tab:: Flyte core - Create a file named ``values-override.yaml`` and add the following configuration to it. + Create a file named ``values-override.yaml`` and add the following configuration to it. - .. code-block:: yaml + .. code-block:: yaml configmap: enabled_plugins: @@ -56,12 +65,16 @@ Specify agent configuration - container - sidecar - k8s-array - - bigquery + - agent-service default-for-task-types: container: container sidecar: sidecar container_array: k8s-array bigquery_query_job_task: agent-service + plugins: + agent-service: + supportedTaskTypes: + - bigquery_query_job_task Ensure that the propeller has the correct service account for BigQuery. diff --git a/rsts/deployment/agents/index.rst b/rsts/deployment/agents/index.rst index b3a196e1b6..af0bddaefd 100644 --- a/rsts/deployment/agents/index.rst +++ b/rsts/deployment/agents/index.rst @@ -27,6 +27,15 @@ Discover the process of setting up Agents for Flyte. ^^^^^^^^^^^^ Guide to setting up the MMCloud agent. + --- + + .. link-button:: deployment-agent-setup-sensor + :type: ref + :text: Sensor Agent + :classes: btn-block stretched-link + ^^^^^^^^^^^^ + Guide to setting up the Sensor agent. + .. toctree:: :maxdepth: 1 @@ -35,3 +44,4 @@ Discover the process of setting up Agents for Flyte. bigquery mmcloud + sensor diff --git a/rsts/deployment/agents/sensor.rst b/rsts/deployment/agents/sensor.rst new file mode 100644 index 0000000000..ecb45e426f --- /dev/null +++ b/rsts/deployment/agents/sensor.rst @@ -0,0 +1,147 @@ +.. _deployment-agent-setup-sensor: + +Sensor Agent +================= + +Sensor enables users to continuously check for a file or a condition to be met periodically. + +When the condition is met, the sensor will complete. + +This guide provides an overview of how to set up Sensor in your Flyte deployment. + +Spin up a cluster +----------------- + +.. tabs:: + + .. group-tab:: Flyte binary + + You can spin up a demo cluster using the following command: + + .. code-block:: bash + + flytectl demo start + + Or install Flyte using the :ref:`flyte-binary helm chart `. + + .. group-tab:: Flyte core + + If you've installed Flyte using the + `flyte-core helm chart `__, please ensure: + + * You have the correct kubeconfig and have selected the correct Kubernetes context. + * Confirm that you have the correct Flytectl configuration at ``~/.flyte/config.yaml``. + +.. note:: + + Add the Flyte chart repo to Helm if you're installing via the Helm charts. + + .. code-block:: bash + + helm repo add flyteorg https://flyteorg.github.io/flyte + +Specify agent configuration +---------------------------- + +Enable the Sensor agent by adding the following config to the relevant YAML file(s): + +.. tabs:: + + .. group-tab:: Flyte binary + + Edit the relevant YAML file to specify the agent. + + .. code-block:: bash + + kubectl edit configmap flyte-sandbox-config -n flyte + + .. code-block:: yaml + :emphasize-lines: 7,11,16 + + tasks: + task-plugins: + enabled-plugins: + - container + - sidecar + - k8s-array + - agent-service + default-for-task-types: + - container: container + - container_array: k8s-array + - sensor: agent-service + + plugins: + agent-service: + supportedTaskTypes: + - sensor + + .. group-tab:: Flyte core + + Create a file named ``values-override.yaml`` and add the following configuration to it. + + .. code-block:: yaml + + configmap: + enabled_plugins: + # -- Tasks specific configuration [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#GetConfig) + tasks: + # -- Plugins configuration, [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#TaskPluginConfig) + task-plugins: + # -- [Enabled Plugins](https://pkg.go.dev/github.com/flyteorg/flyteplugins/go/tasks/config#Config). Enable sagemaker*, athena if you install the backend + enabled-plugins: + - container + - sidecar + - k8s-array + - agent-service + default-for-task-types: + container: container + sidecar: sidecar + container_array: k8s-array + sensor: agent-service + plugins: + agent-service: + supportedTaskTypes: + - sensor + + +Upgrade the deployment +---------------------- + +.. tabs:: + + .. group-tab:: Flyte binary + + .. tabs:: + + .. group-tab:: Demo cluster + + .. code-block:: bash + + kubectl rollout restart deployment flyte-sandbox -n flyte + + .. group-tab:: Helm chart + + .. code-block:: bash + + helm upgrade flyteorg/flyte-binary -n --values + + Replace ```` with the name of your release (e.g., ``flyte-backend``), + ```` with the name of your namespace (e.g., ``flyte``), + and ```` with the name of your YAML file. + + .. group-tab:: Flyte core + + .. code-block:: + + helm upgrade flyte/flyte-core -n --values values-override.yaml + + Replace ```` with the name of your release (e.g., ``flyte``) + and ```` with the name of your namespace (e.g., ``flyte``). + +Wait for the upgrade to complete. + +You can check the status of the deployment pods by running the following command: + +.. code-block:: + + kubectl get pods -n flyte