Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[INTEGRATION][AIRFLOW] Add new extractor for GCSToGCSOperator #1495

Merged
merged 1 commit into from Jan 31, 2023

Conversation

sekikn
Copy link
Contributor

@sekikn sekikn commented Jan 14, 2023

Signed-off-by: Kengo Seki sekikn@apache.org

Problem

Currently, there's no extractor corresponding to GCSToGCSOperator.

Solution

This PR adds a new extractor for GCSToGCSOperator.

Checklist

  • You've signed-off your work
  • Your pull request title follows our guidelines
  • Your changes are accompanied by tests (if relevant)
  • Your change contains a small diff and is self-contained
  • You've updated any relevant documentation (if relevant)
  • You've updated the CHANGELOG.md with details about your change under the "Unreleased" section (if relevant, depending on the change, this may not be necessary)
  • You've versioned the core OpenLineage model or facets according to SchemaVer (if relevant)
  • You've added a header to source files (if relevant)

SPDX-License-Identifier: Apache-2.0
Copyright 2018-2023 contributors to the OpenLineage project

Signed-off-by: Kengo Seki <sekikn@apache.org>
@boring-cyborg boring-cyborg bot added documentation Improvements or additions to documentation extractor integration/airflow labels Jan 14, 2023
@sekikn
Copy link
Contributor Author

sekikn commented Jan 14, 2023

I ran the following command locally after putting my own gcloud-service-key.json in integration/airflow/tests/integration/gcloud and confirmed the gcs_dag integration test succeeded.

$ env AIRFLOW_IMAGE=apache/airflow:2.4.2-python3.7 \
  GOOGLE_CLOUD_STORAGE_SOURCE_URI=gs://gcp-public-data-landsat/LC08/01/044/034/LC08_L1GT_044034_20130330_20170310_01_T2/LC08_L1GT_044034_20130330_20170310_01_T2_ANG.txt \
  GOOGLE_CLOUD_STORAGE_DESTINATION_URI=gs://sekikn/output \
  tests/integration/docker/up.sh

@mobuchowski mobuchowski merged commit f7bfd30 into OpenLineage:main Jan 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
documentation Improvements or additions to documentation extractor integration/airflow
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants