Skip to content
Permalink
Browse files
Add sample dag and doc for S3ListOperator (#23449)
* Add sample dag and doc for S3ListOperator

* Fix doc
  • Loading branch information
vincbeck committed May 9, 2022
1 parent ce81a29 commit 7354d2eac990208d5a631c05649c482560bd827b
Show file tree
Hide file tree
Showing 3 changed files with 29 additions and 0 deletions.
@@ -30,6 +30,7 @@
S3DeleteObjectsOperator,
S3FileTransformOperator,
S3GetBucketTaggingOperator,
S3ListOperator,
S3PutBucketTaggingOperator,
)
from airflow.providers.amazon.aws.sensors.s3 import S3KeySensor, S3KeysUnchangedSensor
@@ -115,6 +116,14 @@ def check_fn(files: List) -> bool:
)
# [END howto_operator_s3_create_object]

# [START howto_operator_s3_list]
list_keys = S3ListOperator(
task_id="s3_list_operator",
bucket=BUCKET_NAME,
prefix=PREFIX,
)
# [END howto_operator_s3_list]

# [START howto_sensor_s3_key_single_key]
# Check if a file exists
s3_sensor_one_key = S3KeySensor(
@@ -193,6 +202,7 @@ def check_fn(files: List) -> bool:
get_tagging,
delete_tagging,
s3_create_object,
list_keys,
[s3_sensor_one_key, s3_sensor_two_keys, s3_sensor_key_function],
s3_copy_object,
s3_sensor_keys_unchanged,
@@ -618,6 +618,10 @@ class S3ListOperator(BaseOperator):
This operator returns a python list with the name of objects which can be
used by `xcom` in the downstream task.
.. seealso::
For more information on how to use this operator, take a look at the guide:
:ref:`howto/operator:S3ListOperator`
:param bucket: The S3 bucket where to find the objects. (templated)
:param prefix: Prefix string to filters the objects whose name begin with
such prefix. (templated)
@@ -196,6 +196,21 @@ To create a new (or replace) Amazon S3 object you can use
:start-after: [START howto_operator_s3_create_object]
:end-before: [END howto_operator_s3_create_object]

.. _howto/operator:S3ListOperator:

List Amazon S3 objects
----------------------

To list all Amazon S3 objects within an Amazon S3 bucket you can use
:class:`~airflow.providers.amazon.aws.operators.s3.S3ListOperator`.
You can specify a ``prefix`` to filter the objects whose name begins with such prefix.

.. exampleinclude:: /../../airflow/providers/amazon/aws/example_dags/example_s3.py
:language: python
:dedent: 4
:start-after: [START howto_operator_s3_list]
:end-before: [END howto_operator_s3_list]

.. _howto/operator:S3CopyObjectOperator:

Copy an Amazon S3 object

0 comments on commit 7354d2e

Please sign in to comment.