Sensor Agent Doc Integration Example (#1180)

* sensor agent doc Signed-off-by: Future Outlier <eric901201@gmai.com> * sensor example Signed-off-by: Future Outlier <eric901201@gmai.com> * new line Signed-off-by: Future Outlier <eric901201@gmai.com> * lint Signed-off-by: Future Outlier <eric901201@gmai.com> * Update examples/sensor/sensor/sensor.py Co-authored-by: Kevin Su <pingsutw@gmail.com> Signed-off-by: Future-Outlier <eric901201@gmail.com> * Update examples/sensor/sensor/sensor.py Co-authored-by: Kevin Su <pingsutw@gmail.com> Signed-off-by: Future-Outlier <eric901201@gmail.com> * Update examples/sensor/sensor/sensor.py Co-authored-by: Kevin Su <pingsutw@gmail.com> Signed-off-by: Future-Outlier <eric901201@gmail.com> * Update examples/sensor/sensor/sensor.py Co-authored-by: Kevin Su <pingsutw@gmail.com> Signed-off-by: Future-Outlier <eric901201@gmail.com> * Update examples/sensor/sensor/sensor.py Co-authored-by: Kevin Su <pingsutw@gmail.com> Signed-off-by: Future-Outlier <eric901201@gmail.com> * Update examples/sensor/sensor/sensor.py Co-authored-by: Kevin Su <pingsutw@gmail.com> Signed-off-by: Future-Outlier <eric901201@gmail.com> * Update examples/sensor/sensor/sensor.py Co-authored-by: Kevin Su <pingsutw@gmail.com> Signed-off-by: Future-Outlier <eric901201@gmail.com> * Update examples/sensor/sensor/sensor.py Co-authored-by: Kevin Su <pingsutw@gmail.com> Signed-off-by: Future-Outlier <eric901201@gmail.com> * Update examples/sensor/sensor/sensor.py Co-authored-by: Kevin Su <pingsutw@gmail.com> Signed-off-by: Future-Outlier <eric901201@gmail.com> * Update examples/sensor/README.md Co-authored-by: Kevin Su <pingsutw@gmail.com> Signed-off-by: Future-Outlier <eric901201@gmail.com> * Update examples/sensor/sensor/sensor.py Co-authored-by: Kevin Su <pingsutw@gmail.com> Signed-off-by: Future-Outlier <eric901201@gmail.com> * Update examples/sensor/sensor/sensor.py Co-authored-by: Kevin Su <pingsutw@gmail.com> Signed-off-by: Future-Outlier <eric901201@gmail.com> * add file_sensor example command line Signed-off-by: Future Outlier <eric901201@gmai.com> * use neverett's advice Signed-off-by: Future Outlier <eric901201@gmai.com> * fix doc error Signed-off-by: Future Outlier <eric901201@gmai.com> * Trigger CI Signed-off-by: Future Outlier <eric901201@gmai.com> * remove $ Signed-off-by: Future Outlier <eric901201@gmai.com> * fixed! Signed-off-by: Future Outlier <eric901201@gmai.com> --------- Signed-off-by: Future Outlier <eric901201@gmai.com> Signed-off-by: Future-Outlier <eric901201@gmail.com> Co-authored-by: Future Outlier <eric901201@gmai.com> Co-authored-by: Kevin Su <pingsutw@gmail.com> Co-authored-by: Kevin Su <pingsutw@apache.org>
flyteorg · Dec 6, 2023 · db817e8 · db817e8
1 parent ae775d5
commit db817e8
Show file tree

Hide file tree

Showing 5 changed files with 67 additions and 0 deletions.
diff --git a/docs/index.md b/docs/index.md
@@ -456,6 +456,7 @@ auto_examples/papermill_plugin/index
 auto_examples/pandera_plugin/index
 auto_examples/kfpytorch_plugin/index
 auto_examples/ray_plugin/index
+auto_examples/sensor/index
 auto_examples/snowflake_plugin/index
 auto_examples/k8s_spark_plugin/index
 auto_examples/sql_plugin/index

diff --git a/docs/integrations.md b/docs/integrations.md
@@ -106,6 +106,8 @@ the Flyte task that use the respective plugin.
   - Run Hive jobs in your workflows.
 * - {doc}`MMCloud <auto_examples/mmcloud_plugin/index>`
   - Execute tasks using MemVerge Memory Machine Cloud
+* - {doc}`Sensor <auto_examples/sensor/index>`
+  - Run Sensor jobs in your workflows.
 * - {doc}`Snowflake <auto_examples/snowflake_plugin/index>`
   - Run Snowflake jobs in your workflows.
 * - {doc}`Databricks <auto_examples/databricks_plugin/index>`

diff --git a/examples/sensor/README.md b/examples/sensor/README.md
@@ -0,0 +1,20 @@
+(sensor)=
+
+# Sensor
+
+```{eval-rst}
+.. tags:: Data, Basic
+```
+
+## Run the example on the Flyte cluster
+
+To run the provided example on the Flyte cluster, use the following command:
+
+```
+pyflyte run --remote \
+  https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/sensor/sensor/file_sensor_example.py wf
+```
+
+```{auto-examples-toc}
+file_sensor_example
+```
diff --git a/examples/sensor/sensor/__init__.py b/examples/sensor/sensor/__init__.py
diff --git a/examples/sensor/sensor/file_sensor_example.py b/examples/sensor/sensor/file_sensor_example.py
@@ -0,0 +1,44 @@
+# %% [markdown]
+# # File Sensor
+#
+# This example shows how to use the `FileSensor` to detect files appearing in your local or remote filesystem.
+#
+# First, import the required libraries.
+
+# %%
+from flytekit import task, workflow
+from flytekit.sensor.file_sensor import FileSensor
+
+# %% [markdown]
+# Next, create a FileSensor task.
+
+# %%
+sensor = FileSensor(name="test_file_sensor")
+
+# %% [markdown]
+# To use the FileSensor created in the previous step, you must specify the path parameter. In the sandbox, you can use the S3 path.
+
+
+# %%
+@task()
+def t1():
+    print("SUCCEEDED")
+
+
+@workflow()
+def wf():
+    sensor(path="s3://my-s3-bucket/file.txt") >> t1()
+
+
+if __name__ == "__main__":
+    wf()
+
+# %% [markdown]
+# You can also use the S3 or GCS file system.
+# We have already set the minio credentials in the agent by default. If you test the sandbox example locally, you will need to set the AWS credentials in your environment variables.
+#
+# ```{prompt} bash
+# export FLYTE_AWS_ENDPOINT="http://localhost:30002"
+# export FLYTE_AWS_ACCESS_KEY_ID="minio"
+# export FLYTE_AWS_SECRET_ACCESS_KEY="miniostorage"
+# ```