Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement Async S3KeySensor #14

Closed
8 tasks
kaxil opened this issue Sep 20, 2021 · 4 comments
Closed
8 tasks

Implement Async S3KeySensor #14

kaxil opened this issue Sep 20, 2021 · 4 comments
Assignees
Labels
area/async Deferrable/async operators

Comments

@kaxil
Copy link
Collaborator

kaxil commented Sep 20, 2021

Based on the research in #32 - Async RedshiftOperator

Acceptance Criteria:

@kaxil kaxil added the area/async Deferrable/async operators label Sep 21, 2021
@kaxil kaxil assigned wolfier and unassigned wolfier and jedcunningham Sep 23, 2021
@kaxil kaxil added the research Requires research or investigation label Sep 23, 2021
@kaxil kaxil assigned dstandish and unassigned wolfier Nov 9, 2021
@phanikumv phanikumv changed the title Async S3KeySensor Implement Async S3KeySensor Dec 30, 2021
@phanikumv phanikumv removed the research Requires research or investigation label Dec 30, 2021
@phanikumv
Copy link
Collaborator

This implementation needs to include the Async version of S3Hook

https://github.com/apache/airflow/blob/206cce971da6941e8c1b0d3c4dbf4fa8afe0fba4/airflow/providers/amazon/aws/hooks/s3.py

@sunank200
Copy link
Collaborator

Based on the existing sync implementation, S3KeySensor uses S3Hook. S3KeySensorAsync needs the Async implementation for S3HookAsync. aiobotocore can be used for the hook connection. S3KeySensorAsync can defer itself with a trigger tied to the event that should resume it. This frees up the worker to run something else. The trigger is run until it fires, at which point its source task is re-scheduled. The scheduler queues the task to resume on a worker node. Working on this implementation.

@sunank200
Copy link
Collaborator

sunank200 commented Jan 20, 2022

Draft #45

@sunank200
Copy link
Collaborator

Done with changes recommended by Daniel. Working on test.

@kaxil kaxil closed this as completed in 570dc1e Feb 15, 2022
@kaxil kaxil mentioned this issue Feb 22, 2022
11 tasks
OlympuJupiter added a commit to OlympuJupiter/astronomer-providers that referenced this issue Nov 14, 2022
- Implement the `S3KeySensorAsync`.
- Implement `S3Trigger`.
- Implement `AwsBaseHookAsync` from `AwsBaseHook`.
- Implement `S3HookAsync` from `S3Hook`.
- Tests for implementation.

closes astronomer/astronomer-providers#14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/async Deferrable/async operators
Projects
None yet
Development

No branches or pull requests

7 participants