Skip to content

Commit

Permalink
[Docs] Sensor Agent Doc (#4195)
Browse files Browse the repository at this point in the history
* sensor agent doc

Signed-off-by: Future Outlier <eric901201@gmai.com>

* sensor agent index rst

Signed-off-by: Future Outlier <eric901201@gmai.com>

---------

Signed-off-by: Future Outlier <eric901201@gmai.com>
Co-authored-by: Future Outlier <eric901201@gmai.com>
Co-authored-by: Kevin Su <pingsutw@apache.org>
  • Loading branch information
3 people committed Nov 11, 2023
1 parent 31748fe commit 0fc2ab0
Show file tree
Hide file tree
Showing 3 changed files with 189 additions and 19 deletions.
51 changes: 32 additions & 19 deletions rsts/deployment/agents/bigquery.rst
Original file line number Diff line number Diff line change
Expand Up @@ -20,30 +20,39 @@ Specify agent configuration

.. tabs::

.. group-tab:: Flyte binary
.. group-tab:: Flyte binary

Edit the relevant YAML file to specify the agent.
Edit the relevant YAML file to specify the agent.

.. code-block:: yaml
:emphasize-lines: 7,11
.. code-block:: bash
tasks:
task-plugins:
enabled-plugins:
- container
- sidecar
- k8s-array
- bigquery
default-for-task-types:
- container: container
- container_array: k8s-array
- bigquery_query_job_task: agent-service
kubectl edit configmap flyte-sandbox-config -n flyte
.. group-tab:: Flyte core
.. code-block:: yaml
:emphasize-lines: 7,11,16
tasks:
task-plugins:
enabled-plugins:
- container
- sidecar
- k8s-array
- agent-service
default-for-task-types:
- container: container
- container_array: k8s-array
- bigquery_query_job_task: agent-service
plugins:
agent-service:
supportedTaskTypes:
- bigquery_query_job_task
.. group-tab:: Flyte core

Create a file named ``values-override.yaml`` and add the following configuration to it.
Create a file named ``values-override.yaml`` and add the following configuration to it.

.. code-block:: yaml
.. code-block:: yaml
configmap:
enabled_plugins:
Expand All @@ -56,12 +65,16 @@ Specify agent configuration
- container
- sidecar
- k8s-array
- bigquery
- agent-service
default-for-task-types:
container: container
sidecar: sidecar
container_array: k8s-array
bigquery_query_job_task: agent-service
plugins:
agent-service:
supportedTaskTypes:
- bigquery_query_job_task
Ensure that the propeller has the correct service account for BigQuery.

Expand Down
10 changes: 10 additions & 0 deletions rsts/deployment/agents/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,15 @@ Discover the process of setting up Agents for Flyte.
^^^^^^^^^^^^
Guide to setting up the MMCloud agent.

---

.. link-button:: deployment-agent-setup-sensor
:type: ref
:text: Sensor Agent
:classes: btn-block stretched-link
^^^^^^^^^^^^
Guide to setting up the Sensor agent.


.. toctree::
:maxdepth: 1
Expand All @@ -35,3 +44,4 @@ Discover the process of setting up Agents for Flyte.

bigquery
mmcloud
sensor
147 changes: 147 additions & 0 deletions rsts/deployment/agents/sensor.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
.. _deployment-agent-setup-sensor:

Sensor Agent
=================

Sensor enables users to continuously check for a file or a condition to be met periodically.

When the condition is met, the sensor will complete.

This guide provides an overview of how to set up Sensor in your Flyte deployment.

Spin up a cluster
-----------------

.. tabs::

.. group-tab:: Flyte binary

You can spin up a demo cluster using the following command:

.. code-block:: bash
flytectl demo start
Or install Flyte using the :ref:`flyte-binary helm chart <deployment-deployment-cloud-simple>`.

.. group-tab:: Flyte core

If you've installed Flyte using the
`flyte-core helm chart <https://github.com/flyteorg/flyte/tree/master/charts/flyte-core>`__, please ensure:

* You have the correct kubeconfig and have selected the correct Kubernetes context.
* Confirm that you have the correct Flytectl configuration at ``~/.flyte/config.yaml``.

.. note::

Add the Flyte chart repo to Helm if you're installing via the Helm charts.

.. code-block:: bash
helm repo add flyteorg https://flyteorg.github.io/flyte
Specify agent configuration
----------------------------

Enable the Sensor agent by adding the following config to the relevant YAML file(s):

.. tabs::

.. group-tab:: Flyte binary

Edit the relevant YAML file to specify the agent.

.. code-block:: bash
kubectl edit configmap flyte-sandbox-config -n flyte
.. code-block:: yaml
:emphasize-lines: 7,11,16
tasks:
task-plugins:
enabled-plugins:
- container
- sidecar
- k8s-array
- agent-service
default-for-task-types:
- container: container
- container_array: k8s-array
- sensor: agent-service
plugins:
agent-service:
supportedTaskTypes:
- sensor
.. group-tab:: Flyte core

Create a file named ``values-override.yaml`` and add the following configuration to it.

.. code-block:: yaml
configmap:
enabled_plugins:
# -- Tasks specific configuration [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#GetConfig)
tasks:
# -- Plugins configuration, [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#TaskPluginConfig)
task-plugins:
# -- [Enabled Plugins](https://pkg.go.dev/github.com/flyteorg/flyteplugins/go/tasks/config#Config). Enable sagemaker*, athena if you install the backend
enabled-plugins:
- container
- sidecar
- k8s-array
- agent-service
default-for-task-types:
container: container
sidecar: sidecar
container_array: k8s-array
sensor: agent-service
plugins:
agent-service:
supportedTaskTypes:
- sensor
Upgrade the deployment
----------------------

.. tabs::

.. group-tab:: Flyte binary

.. tabs::

.. group-tab:: Demo cluster

.. code-block:: bash
kubectl rollout restart deployment flyte-sandbox -n flyte
.. group-tab:: Helm chart

.. code-block:: bash
helm upgrade <RELEASE_NAME> flyteorg/flyte-binary -n <YOUR_NAMESPACE> --values <YOUR_YAML_FILE>
Replace ``<RELEASE_NAME>`` with the name of your release (e.g., ``flyte-backend``),
``<YOUR_NAMESPACE>`` with the name of your namespace (e.g., ``flyte``),
and ``<YOUR_YAML_FILE>`` with the name of your YAML file.

.. group-tab:: Flyte core

.. code-block::
helm upgrade <RELEASE_NAME> flyte/flyte-core -n <YOUR_NAMESPACE> --values values-override.yaml
Replace ``<RELEASE_NAME>`` with the name of your release (e.g., ``flyte``)
and ``<YOUR_NAMESPACE>`` with the name of your namespace (e.g., ``flyte``).

Wait for the upgrade to complete.

You can check the status of the deployment pods by running the following command:

.. code-block::
kubectl get pods -n flyte

0 comments on commit 0fc2ab0

Please sign in to comment.