Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Docs] Sensor Agent Doc #4195

Merged
merged 3 commits into from
Nov 11, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
51 changes: 32 additions & 19 deletions rsts/deployment/agents/bigquery.rst
Original file line number Diff line number Diff line change
Expand Up @@ -20,30 +20,39 @@ Specify agent configuration

.. tabs::

.. group-tab:: Flyte binary
.. group-tab:: Flyte binary

Edit the relevant YAML file to specify the agent.
Edit the relevant YAML file to specify the agent.

.. code-block:: yaml
:emphasize-lines: 7,11
.. code-block:: bash

tasks:
task-plugins:
enabled-plugins:
- container
- sidecar
- k8s-array
- bigquery
default-for-task-types:
- container: container
- container_array: k8s-array
- bigquery_query_job_task: agent-service
kubectl edit configmap flyte-sandbox-config -n flyte

.. group-tab:: Flyte core
.. code-block:: yaml
:emphasize-lines: 7,11,16

tasks:
task-plugins:
enabled-plugins:
- container
- sidecar
- k8s-array
- agent-service
default-for-task-types:
- container: container
- container_array: k8s-array
- bigquery_query_job_task: agent-service

plugins:
agent-service:
supportedTaskTypes:
- bigquery_query_job_task

.. group-tab:: Flyte core

Create a file named ``values-override.yaml`` and add the following configuration to it.
Create a file named ``values-override.yaml`` and add the following configuration to it.

.. code-block:: yaml
.. code-block:: yaml

configmap:
enabled_plugins:
Expand All @@ -56,12 +65,16 @@ Specify agent configuration
- container
- sidecar
- k8s-array
- bigquery
- agent-service
default-for-task-types:
container: container
sidecar: sidecar
container_array: k8s-array
bigquery_query_job_task: agent-service
plugins:
agent-service:
supportedTaskTypes:
- bigquery_query_job_task

Ensure that the propeller has the correct service account for BigQuery.

Expand Down
10 changes: 10 additions & 0 deletions rsts/deployment/agents/index.rst
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,15 @@ Discover the process of setting up Agents for Flyte.
^^^^^^^^^^^^
Guide to setting up the MMCloud agent.

---

.. link-button:: deployment-agent-setup-sensor
:type: ref
:text: Sensor Agent
:classes: btn-block stretched-link
^^^^^^^^^^^^
Guide to setting up the Sensor agent.


.. toctree::
:maxdepth: 1
Expand All @@ -35,3 +44,4 @@ Discover the process of setting up Agents for Flyte.

bigquery
mmcloud
sensor
147 changes: 147 additions & 0 deletions rsts/deployment/agents/sensor.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,147 @@
.. _deployment-agent-setup-sensor:

Sensor Agent
=================

Sensor enables users to continuously check for a file or a condition to be met periodically.

When the condition is met, the sensor will complete.

This guide provides an overview of how to set up Sensor in your Flyte deployment.

Spin up a cluster
-----------------

.. tabs::

.. group-tab:: Flyte binary

You can spin up a demo cluster using the following command:

.. code-block:: bash

flytectl demo start

Or install Flyte using the :ref:`flyte-binary helm chart <deployment-deployment-cloud-simple>`.

.. group-tab:: Flyte core

If you've installed Flyte using the
`flyte-core helm chart <https://github.com/flyteorg/flyte/tree/master/charts/flyte-core>`__, please ensure:

* You have the correct kubeconfig and have selected the correct Kubernetes context.
* Confirm that you have the correct Flytectl configuration at ``~/.flyte/config.yaml``.

.. note::

Add the Flyte chart repo to Helm if you're installing via the Helm charts.

.. code-block:: bash

helm repo add flyteorg https://flyteorg.github.io/flyte

Specify agent configuration
----------------------------

Enable the Sensor agent by adding the following config to the relevant YAML file(s):

.. tabs::

.. group-tab:: Flyte binary

Edit the relevant YAML file to specify the agent.

.. code-block:: bash

kubectl edit configmap flyte-sandbox-config -n flyte

.. code-block:: yaml
:emphasize-lines: 7,11,16

tasks:
task-plugins:
enabled-plugins:
- container
- sidecar
- k8s-array
- agent-service
default-for-task-types:
- container: container
- container_array: k8s-array
- sensor: agent-service

plugins:
agent-service:
supportedTaskTypes:
- sensor

.. group-tab:: Flyte core

Create a file named ``values-override.yaml`` and add the following configuration to it.

.. code-block:: yaml

configmap:
enabled_plugins:
# -- Tasks specific configuration [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#GetConfig)
tasks:
# -- Plugins configuration, [structure](https://pkg.go.dev/github.com/flyteorg/flytepropeller/pkg/controller/nodes/task/config#TaskPluginConfig)
task-plugins:
# -- [Enabled Plugins](https://pkg.go.dev/github.com/flyteorg/flyteplugins/go/tasks/config#Config). Enable sagemaker*, athena if you install the backend
enabled-plugins:
- container
- sidecar
- k8s-array
- agent-service
default-for-task-types:
container: container
sidecar: sidecar
container_array: k8s-array
sensor: agent-service
plugins:
agent-service:
supportedTaskTypes:
- sensor


Upgrade the deployment
----------------------

.. tabs::

.. group-tab:: Flyte binary

.. tabs::

.. group-tab:: Demo cluster

.. code-block:: bash

kubectl rollout restart deployment flyte-sandbox -n flyte

.. group-tab:: Helm chart

.. code-block:: bash

helm upgrade <RELEASE_NAME> flyteorg/flyte-binary -n <YOUR_NAMESPACE> --values <YOUR_YAML_FILE>

Replace ``<RELEASE_NAME>`` with the name of your release (e.g., ``flyte-backend``),
``<YOUR_NAMESPACE>`` with the name of your namespace (e.g., ``flyte``),
and ``<YOUR_YAML_FILE>`` with the name of your YAML file.

.. group-tab:: Flyte core

.. code-block::

helm upgrade <RELEASE_NAME> flyte/flyte-core -n <YOUR_NAMESPACE> --values values-override.yaml

Replace ``<RELEASE_NAME>`` with the name of your release (e.g., ``flyte``)
and ``<YOUR_NAMESPACE>`` with the name of your namespace (e.g., ``flyte``).

Wait for the upgrade to complete.

You can check the status of the deployment pods by running the following command:

.. code-block::

kubectl get pods -n flyte