Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Spike] Async GCSObjectExistenceSensor #13

Closed
kaxil opened this issue Sep 20, 2021 · 2 comments
Closed

[Spike] Async GCSObjectExistenceSensor #13

kaxil opened this issue Sep 20, 2021 · 2 comments
Assignees
Labels
research Requires research or investigation

Comments

@kaxil
Copy link
Collaborator

kaxil commented Sep 20, 2021

Async version of https://github.com/apache/airflow/blob/1008d8bf8acf459dbc692691a589c27fa4567123/airflow/providers/google/cloud/sensors/gcs.py#L30 using one of the following libraries:

Official Python client does not support it yet: googleapis/google-cloud-python#3103

Acceptance Criteria:

  • Document possible options and selection reason for a particular library in this GitHub issue via a Summary comment
@kaxil kaxil self-assigned this Sep 21, 2021
@kaxil kaxil added the area/async Deferrable/async operators label Sep 21, 2021
@kaxil kaxil assigned wolfier and unassigned kaxil Nov 9, 2021
@kaxil kaxil changed the title Async GCSObjectExistenceSensor [Spike] Async GCSObjectExistenceSensor Dec 29, 2021
@kaxil kaxil added the research Requires research or investigation label Dec 29, 2021
@astronomer astronomer deleted a comment from phanikumv Dec 29, 2021
@phanikumv phanikumv removed the area/async Deferrable/async operators label Dec 30, 2021
@rajaths010494
Copy link
Contributor

rajaths010494 commented Jan 4, 2022

Both gcloud-aio and aiogoogle has a aiohttp layer for all API calls to googleAPIs.
gcloud-aio has support to cloud storage,, BigQuery, pubsubs are there so we could use one library and also the library is very similar to the one used in airflow sync version which is google-cloud-storage.
So choosing gcloud-aio for having to ease of changes because similarity between sync library to async library

gcloud-aio Pros Cons
similar to sync library google cloud stroage rewrite method support isnt there for huge files above 5 TB doesnt have
supports other bigquery and pubusbs services
aiogoogle Pros Cons
has more apis for google services Certain methods implemenatations are not present (blob chunk size while copying not present)
Doesn't have support for bigquery and pubsubs

@phanikumv
Copy link
Collaborator

1.Please correct typos in your comments
2.aiogoogle - which method implementations are absent. Please quote few examples

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
research Requires research or investigation
Projects
None yet
Development

No branches or pull requests

5 participants