Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Migrate Google example DAGs to new design #22447

Closed
71 of 89 tasks
potiuk opened this issue Mar 22, 2022 · 6 comments
Closed
71 of 89 tasks

Migrate Google example DAGs to new design #22447

potiuk opened this issue Mar 22, 2022 · 6 comments
Labels
AIP-47 AIP-47 New Design of System Tests kind:meta High-level information important to the community
Projects
AIP-47 New Design of System Tests

Comments

@potiuk
Copy link
Member

potiuk commented Mar 22, 2022 •
edited
Loading

There is a new design of system tests that was introduced by the AIP-47.

All current example dags need to be migrated and converted into system tests, so they can be run in the CI process automatically before releases.

This is an aggregated issue for all example DAGs related to Google provider. It is created to track progress of their migration.

List of paths to example DAGs:

  • airflow/providers/google/leveldb/example_dags/example_leveldb.py
  • airflow/providers/google/firebase/example_dags/example_firestore.py
  • airflow/providers/google/ads/example_dags/example_ads.py
  • airflow/providers/google/suite/example_dags/example_sql_to_sheets.py
  • airflow/providers/google/suite/example_dags/example_gcs_to_sheets.py
  • airflow/providers/google/suite/example_dags/example_gcs_to_gdrive.py
  • airflow/providers/google/suite/example_dags/example_sheets.py
  • airflow/providers/google/marketing_platform/example_dags/example_analytics.py
  • airflow/providers/google/marketing_platform/example_dags/example_display_video.py
  • airflow/providers/google/marketing_platform/example_dags/example_campaign_manager.py
  • airflow/providers/google/marketing_platform/example_dags/example_search_ads.py
  • airflow/providers/google/cloud/example_dags/example_bigquery_queries.py
  • airflow/providers/google/cloud/example_dags/example_gcs_to_sftp.py
  • airflow/providers/google/cloud/example_dags/example_oracle_to_gcs.py
  • airflow/providers/google/cloud/example_dags/example_compute_igm.py
  • airflow/providers/google/cloud/example_dags/example_automl_vision_classification.py
  • airflow/providers/google/cloud/example_dags/example_datacatalog.py
  • airflow/providers/google/cloud/example_dags/example_bigquery_to_mssql.py
  • airflow/providers/google/cloud/example_dags/example_mssql_to_gcs.py
  • airflow/providers/google/cloud/example_dags/example_mysql_to_gcs.py
  • airflow/providers/google/cloud/example_dags/example_bigquery_to_bigquery.py
  • airflow/providers/google/cloud/example_dags/example_natural_language.py
  • airflow/providers/google/cloud/example_dags/example_life_sciences.py
  • airflow/providers/google/cloud/example_dags/example_automl_nl_text_extraction.py
  • airflow/providers/google/cloud/example_dags/example_automl_video_intelligence_classification.py
  • airflow/providers/google/cloud/example_dags/example_trino_to_gcs.py
  • airflow/providers/google/cloud/example_dags/example_bigquery_transfer.py
  • airflow/providers/google/cloud/example_dags/example_cloud_storage_transfer_service_gcp.py
  • airflow/providers/google/cloud/example_dags/example_translate.py
  • airflow/providers/google/cloud/example_dags/example_local_to_gcs.py
  • airflow/providers/google/cloud/example_dags/example_compute_ssh.py
  • airflow/providers/google/cloud/example_dags/example_dataflow_sql.py
  • airflow/providers/google/cloud/example_dags/example_text_to_speech.py
  • airflow/providers/google/cloud/example_dags/example_presto_to_gcs.py
  • airflow/providers/google/cloud/example_dags/example_looker.py
  • airflow/providers/google/cloud/example_dags/example_sftp_to_gcs.py
  • airflow/providers/google/cloud/example_dags/example_facebook_ads_to_gcs.py
  • airflow/providers/google/cloud/example_dags/example_gcs_to_bigquery.py
  • airflow/providers/google/cloud/example_dags/example_postgres_to_gcs.py
  • airflow/providers/google/cloud/example_dags/example_spanner.py
  • airflow/providers/google/cloud/example_dags/example_gdrive_to_gcs.py
  • airflow/providers/google/cloud/example_dags/example_dlp.py
  • airflow/providers/google/cloud/example_dags/example_bigquery_dts.py
  • airflow/providers/google/cloud/example_dags/example_datastore.py
  • airflow/providers/google/cloud/example_dags/example_bigquery_operations.py
  • airflow/providers/google/cloud/example_dags/example_automl_nl_text_classification.py
  • airflow/providers/google/cloud/example_dags/example_bigquery_to_gcs.py
  • airflow/providers/google/cloud/example_dags/example_automl_vision_object_detection.py
  • airflow/providers/google/cloud/example_dags/example_bigtable.py
  • airflow/providers/google/cloud/example_dags/example_gcs_to_local.py
  • airflow/providers/google/cloud/example_dags/example_vertex_ai.py
  • airflow/providers/google/cloud/example_dags/example_automl_video_intelligence_tracking.py
  • airflow/providers/google/cloud/example_dags/example_gcs_timespan_file_transform.py
  • airflow/providers/google/cloud/example_dags/example_dataproc_metastore.py
  • airflow/providers/google/cloud/example_dags/example_salesforce_to_gcs.py
  • airflow/providers/google/cloud/example_dags/example_functions.py
  • airflow/providers/google/cloud/example_dags/example_compute.py
  • airflow/providers/google/cloud/example_dags/example_sheets_to_gcs.py
  • airflow/providers/google/cloud/example_dags/example_dataflow.py
  • airflow/providers/google/cloud/example_dags/example_datafusion.py
  • airflow/providers/google/cloud/example_dags/example_cloud_sql_query.py
  • airflow/providers/google/cloud/example_dags/example_translate_speech.py
  • airflow/providers/google/cloud/example_dags/example_automl_tables.py
  • airflow/providers/google/cloud/example_dags/example_cloud_memorystore.py
  • airflow/providers/google/cloud/example_dags/example_bigquery_sensors.py
  • airflow/providers/google/cloud/example_dags/example_gcs.py
  • airflow/providers/google/cloud/example_dags/example_mlengine.py
  • airflow/providers/google/cloud/example_dags/example_dataprep.py
  • airflow/providers/google/cloud/example_dags/example_cloud_storage_transfer_service_aws.py
  • airflow/providers/google/cloud/example_dags/example_pubsub.py
  • airflow/providers/google/cloud/example_dags/example_gcs_to_gcs.py
  • airflow/providers/google/cloud/example_dags/example_s3_to_gcs.py
  • airflow/providers/google/cloud/example_dags/example_gdrive_to_local.py
  • airflow/providers/google/cloud/example_dags/example_azure_fileshare_to_gcs.py
  • airflow/providers/google/cloud/example_dags/example_workflows.py
  • airflow/providers/google/cloud/example_dags/example_kubernetes_engine.py
  • airflow/providers/google/cloud/example_dags/example_dataproc.py
  • airflow/providers/google/cloud/example_dags/example_speech_to_text.py
  • airflow/providers/google/cloud/example_dags/example_cloud_composer.py
  • airflow/providers/google/cloud/example_dags/example_automl_translation.py
  • airflow/providers/google/cloud/example_dags/example_stackdriver.py
  • airflow/providers/google/cloud/example_dags/example_tasks.py
  • airflow/providers/google/cloud/example_dags/example_calendar_to_gcs.py
  • airflow/providers/google/cloud/example_dags/example_video_intelligence.py
  • airflow/providers/google/cloud/example_dags/example_cloud_build.py
  • airflow/providers/google/cloud/example_dags/example_dataflow_flex_template.py
  • airflow/providers/google/cloud/example_dags/example_automl_nl_text_sentiment.py
  • airflow/providers/google/cloud/example_dags/example_cloud_sql.py
  • airflow/providers/google/cloud/example_dags/example_vision.py

In progress:

  • airflow/providers/google/cloud/example_dags/example_bigquery_to_bigquery.py
  • airflow/providers/google/cloud/example_dags/example_dataflow.py
  • airflow/providers/google/cloud/example_dags/example_datafusion.py
  • airflow/providers/google/cloud/example_dags/example_dataproc.py
  • airflow/providers/google/cloud/example_dags/example_dataproc_metastore.py
  • airflow/providers/google/cloud/example_dags/example_datastore.py
  • airflow/providers/google/cloud/example_dags/example_gcs.py
  • airflow/providers/google/cloud/example_dags/example_gcs_timespan_file_transform.py
  • airflow/providers/google/cloud/example_dags/example_gcs_to_gcs.py
  • airflow/providers/google/cloud/example_dags/example_gcs_to_local.py
  • airflow/providers/google/cloud/example_dags/example_local_to_gcs.py
@potiuk potiuk added the AIP-47 AIP-47 New Design of System Tests label Mar 22, 2022
@potiuk potiuk added the kind:meta High-level information important to the community label Mar 22, 2022
@mnojek
Copy link
Contributor

mnojek commented Mar 24, 2022

Hey, please update the checklist.

These example DAGs are being removed by PR #22311:

  • airflow/providers/google/cloud/example_dags/example_bigquery_operations.py
  • airflow/providers/google/cloud/example_dags/example_bigquery_queries.py
  • airflow/providers/google/cloud/example_dags/example_bigquery_sensors.py

And these are WIP (Work In Progress) (not sure how we are going to mark them, but it would be good to notify others so that we will avoid work duplication):

  • airflow/providers/google/cloud/example_dags/example_bigquery_to_bigquery.py
  • airflow/providers/google/cloud/example_dags/example_dataflow.py
  • airflow/providers/google/cloud/example_dags/example_datafusion.py
  • airflow/providers/google/cloud/example_dags/example_dataproc.py
  • airflow/providers/google/cloud/example_dags/example_dataproc_metastore.py
  • airflow/providers/google/cloud/example_dags/example_datastore.py
  • airflow/providers/google/cloud/example_dags/example_gcs.py
  • airflow/providers/google/cloud/example_dags/example_gcs_timespan_file_transform.py
  • airflow/providers/google/cloud/example_dags/example_gcs_to_gcs.py
  • airflow/providers/google/cloud/example_dags/example_gcs_to_local.py
  • airflow/providers/google/cloud/example_dags/example_local_to_gcs.py

@potiuk
Copy link
Member Author

potiuk commented Mar 28, 2022

Updated

This was referenced Apr 5, 2022
Closed
Merged
This was referenced Apr 6, 2022
Merged
Merged
This was referenced Apr 20, 2022
Merged
Merged
This was referenced Apr 27, 2022
Merged
Merged
@potiuk
Copy link
Member Author

potiuk commented Apr 28, 2022

BTW. I will automate refreshing the list shortly :) @bhirsz :)

@chethanuk
Copy link
Contributor

Hi @potiuk Can we update the checklist if and when possible [So Can know which needs to be done, which are WIP]. Thanks :)

@potiuk
Copy link
Member Author

potiuk commented Jun 3, 2022

Sure. I merge all the green ones and re-run the script to update the checklists

@potiuk potiuk mentioned this issue Jun 3, 2022
Open
chethanuk added a commit to chethanuk/airflow-1 that referenced this issue Jun 5, 2022
@chethanuk
AIP-47 - Migrate google leveldb DAGs to new design #apache#22447
efe78db
chethanuk added a commit to chethanuk/airflow-1 that referenced this issue Jun 5, 2022
@chethanuk
AIP-47 - Migrate google marketing DAGs to new design apache#22447
945b20c
This was referenced Jun 6, 2022
Closed
Merged
potiuk pushed a commit that referenced this issue Jun 6, 2022
@chenglongyan
Migrate Google leveldb system tests to new design AIP-47 (#24255)
203fe71 
related: #22447, #22430
potiuk pushed a commit that referenced this issue Jun 7, 2022
@chethanuk
AIP-47 - Migrate google leveldb DAGs to new design ##22447 (#24233)
c3a9ef1
This was referenced Jun 7, 2022
Merged
Merged
@chenglongyan chenglongyan mentioned this issue Jul 19, 2022
Merged
potiuk pushed a commit that referenced this issue Jul 19, 2022
@chenglongyan
Migrate Google example automl_vision to new design AIP-47 (#25152)
3a80b36 
related: #22447, #22430
potiuk pushed a commit that referenced this issue Jul 19, 2022
@chenglongyan
Migrate Google example compute_igm to new design AIP-47 (#25132)
9644451 
related: #22447, #22430
@chenglongyan chenglongyan mentioned this issue Jul 20, 2022
Merged
potiuk pushed a commit that referenced this issue Jul 21, 2022
@chenglongyan
Migrate Google example bigquery_to_mssql to new design AIP-47 (#25174)
66b3ca1 
related: #22447, #22430
This was referenced Jul 24, 2022
Merged
Merged
Merged
potiuk pushed a commit that referenced this issue Jul 26, 2022
@chenglongyan
Delete redundant system test bigquery_to_bigquery (#25261)
b288bf4 
related: #22447, #22430
potiuk pushed a commit that referenced this issue Jul 26, 2022
@chenglongyan
Migrate Google example natural_language to new design AIP-47 (#25262)
10c9a36 
related: #22447, #22430
potiuk pushed a commit that referenced this issue Jul 26, 2022
@chenglongyan
Migrate Google example life_sciences to new design AIP-47 (#25264)
28db8c1 
related: #22447, #22430
This was referenced Jul 30, 2022
Merged
Merged
potiuk pushed a commit that referenced this issue Aug 2, 2022
@chenglongyan
Migrate Google example automl_nl_text_extraction to new design AIP-47 (…
d290002 
…#25418)

related: #22447, #22430
potiuk pushed a commit that referenced this issue Aug 2, 2022
@chenglongyan
Migrate Google example trino_to_gcs to new design AIP-47 (#25420)
c8cacad 
related: #22447, #22430
This was referenced Aug 12, 2022
Closed
Closed
This was referenced Aug 21, 2022
Closed
Closed
This was referenced Aug 29, 2022
Closed
Merged
@VladaZakharova VladaZakharova mentioned this issue Jul 10, 2023
Closed
81 tasks
This was referenced Jul 17, 2023
Closed
Merged
Merged
Merged
@moiseenkov moiseenkov mentioned this issue Jul 24, 2023
Merged
@eladkal eladkal mentioned this issue Jan 24, 2024
Merged
@eladkal
Copy link
Contributor

eladkal commented Jun 2, 2024

Do we still need this tracking issue?

@potiuk potiuk closed this as completed Jun 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
AIP-47 AIP-47 New Design of System Tests kind:meta High-level information important to the community
Projects
No open projects
AIP-47 New Design of System Tests
To do
Milestone
No milestone
Development

No branches or pull requests
4 participants
@potiuk @mnojek @chethanuk @eladkal