Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sensor Agent Doc Integration Example #1180

Merged
merged 32 commits into from
Dec 6, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
3303706
sensor agent doc
Oct 11, 2023
badf470
Merge branch 'master' into sensor-agent-doc-v2
Future-Outlier Oct 12, 2023
766c99b
sensor example
Oct 17, 2023
c4212cb
Merge branch 'sensor-agent-doc-v2' of https://github.com/Future-Outli…
Oct 17, 2023
e6662e9
new line
Oct 17, 2023
bfd74ef
Merge branch 'master' into sensor-agent-doc-v2
Future-Outlier Oct 18, 2023
52bb657
lint
Oct 18, 2023
3665a47
Merge branch 'master' into sensor-agent-doc-v2
Future-Outlier Oct 24, 2023
cc20402
Merge branch 'master' into sensor-agent-doc-v2
Future-Outlier Nov 7, 2023
41eedbe
Update examples/sensor/sensor/sensor.py
Future-Outlier Nov 8, 2023
edc829b
Update examples/sensor/sensor/sensor.py
Future-Outlier Nov 8, 2023
ba0cbd1
Update examples/sensor/sensor/sensor.py
Future-Outlier Nov 8, 2023
be664cb
Update examples/sensor/sensor/sensor.py
Future-Outlier Nov 8, 2023
60a89ca
Update examples/sensor/sensor/sensor.py
Future-Outlier Nov 8, 2023
bbb06a8
Update examples/sensor/sensor/sensor.py
Future-Outlier Nov 8, 2023
04d5f9e
Update examples/sensor/sensor/sensor.py
Future-Outlier Nov 8, 2023
dc15e86
Update examples/sensor/sensor/sensor.py
Future-Outlier Nov 8, 2023
704b4eb
Update examples/sensor/sensor/sensor.py
Future-Outlier Nov 8, 2023
ddf6d9b
Update examples/sensor/README.md
Future-Outlier Nov 8, 2023
6dffcc9
Update examples/sensor/sensor/sensor.py
Future-Outlier Nov 8, 2023
42b1ee6
Update examples/sensor/sensor/sensor.py
Future-Outlier Nov 8, 2023
9867fd3
add file_sensor example command line
Nov 8, 2023
6076178
use neverett's advice
Nov 15, 2023
ca50b8d
Merge branch 'master' into sensor-agent-doc-v2
pingsutw Nov 21, 2023
c81cf13
Merge branch 'master' into sensor-agent-doc-v2
pingsutw Nov 21, 2023
45bdfaf
Merge branch 'master' of https://github.com/Future-Outlier/flytesnack…
Nov 28, 2023
a1a6b9e
fix doc error
Nov 28, 2023
b53f2bd
Trigger CI
Nov 28, 2023
b9f6ea0
remove $
Nov 28, 2023
71f51a8
fixed!
Nov 28, 2023
3f751ed
Merge branch 'master' into sensor-agent-doc-v2
Future-Outlier Dec 4, 2023
e42c965
Merge branch 'master' into sensor-agent-doc-v2
eapolinario Dec 6, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions docs/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -456,6 +456,7 @@ auto_examples/papermill_plugin/index
auto_examples/pandera_plugin/index
auto_examples/kfpytorch_plugin/index
auto_examples/ray_plugin/index
auto_examples/sensor/index
auto_examples/snowflake_plugin/index
auto_examples/k8s_spark_plugin/index
auto_examples/sql_plugin/index
Expand Down
2 changes: 2 additions & 0 deletions docs/integrations.md
Original file line number Diff line number Diff line change
Expand Up @@ -106,6 +106,8 @@ the Flyte task that use the respective plugin.
- Run Hive jobs in your workflows.
* - {doc}`MMCloud <auto_examples/mmcloud_plugin/index>`
- Execute tasks using MemVerge Memory Machine Cloud
* - {doc}`Sensor <auto_examples/sensor/index>`
- Run Sensor jobs in your workflows.
* - {doc}`Snowflake <auto_examples/snowflake_plugin/index>`
- Run Snowflake jobs in your workflows.
* - {doc}`Databricks <auto_examples/databricks_plugin/index>`
Expand Down
20 changes: 20 additions & 0 deletions examples/sensor/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
(sensor)=

# Sensor

```{eval-rst}
.. tags:: Data, Basic
```

## Run the example on the Flyte cluster

To run the provided example on the Flyte cluster, use the following command:

```
pyflyte run --remote \
https://raw.githubusercontent.com/flyteorg/flytesnacks/master/examples/sensor/sensor/file_sensor_example.py wf
```

```{auto-examples-toc}
file_sensor_example
```
Empty file.
44 changes: 44 additions & 0 deletions examples/sensor/sensor/file_sensor_example.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# %% [markdown]
# # File Sensor
#
# This example shows how to use the `FileSensor` to detect files appearing in your local or remote filesystem.
#
# First, import the required libraries.

# %%
from flytekit import task, workflow
from flytekit.sensor.file_sensor import FileSensor

# %% [markdown]
# Next, create a FileSensor task.

# %%
sensor = FileSensor(name="test_file_sensor")

# %% [markdown]
# To use the FileSensor created in the previous step, you must specify the path parameter. In the sandbox, you can use the S3 path.


# %%
@task()
def t1():
print("SUCCEEDED")


@workflow()
def wf():
sensor(path="s3://my-s3-bucket/file.txt") >> t1()


if __name__ == "__main__":
wf()

# %% [markdown]
# You can also use the S3 or GCS file system.
# We have already set the minio credentials in the agent by default. If you test the sandbox example locally, you will need to set the AWS credentials in your environment variables.
#
# ```{prompt} bash
# export FLYTE_AWS_ENDPOINT="http://localhost:30002"
# export FLYTE_AWS_ACCESS_KEY_ID="minio"
# export FLYTE_AWS_SECRET_ACCESS_KEY="miniostorage"
# ```
Loading