Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[AIRFLOW-4964] Add BigQuery Data Transfer Hook and Operator #5769

Merged
merged 3 commits into from
Sep 6, 2019
Merged

[AIRFLOW-4964] Add BigQuery Data Transfer Hook and Operator #5769

merged 3 commits into from
Sep 6, 2019

Conversation

turbaszek
Copy link
Member

Make sure you have checked all steps below.

Jira

Description

  • Here are some details about my PR, including screenshots of any UI changes:

Tests

  • My PR adds unit and system tests

Commits

  • My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
    1. Subject is separated from body by a blank line
    2. Subject is limited to 50 characters (not including Jira issue reference)
    3. Subject does not end with a period
    4. Subject uses the imperative mood ("add", not "adding")
    5. Body wraps at 72 characters
    6. Body explains "what" and "why", not "how"

Documentation

  • In case of new functionality, my PR adds documentation that describes how to use it.
    • All the public functions and the classes in the PR contain docstrings that explain what it does
    • If you implement backwards incompatible changes, please leave a note in the Updating.md so we can assign it to a appropriate release

Code Quality

  • Passes flake8

@mik-laj mik-laj added the provider:google Google (including GCP) related issues label Aug 9, 2019
mik-laj
mik-laj reviewed Aug 12, 2019
View reviewed changes
airflow/contrib/example_dags/example_gcp_bigquery_dts.py Outdated
# In the case of Airflow, the customer needs to create a transfer
# config with the automatic scheduling disabled and then trigger
# a transfer run using a specialized Airflow operator
schedule_options = {"disable_auto_scheduling": True}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this option turned off by default? We are talking about such feature.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it's done in hook's method _disable_auto_scheduling.

mik-laj
mik-laj reviewed Aug 12, 2019
View reviewed changes
setup.py Outdated Show resolved Hide resolved
@mik-laj mik-laj changed the title [AIRFLOW-4964] Add BigQuery Data Transfer Hook and Operator [AIRFLOW-4964][WIP-DONT-MERGE] Add BigQuery Data Transfer Hook and Operator Aug 13, 2019
@turbaszek turbaszek force-pushed the feature/big-query-dts-operators branch from 071e68f to 498a7d9 Compare August 13, 2019 14:50
@turbaszek
Copy link
Member Author

Depends on #5791.

@turbaszek turbaszek force-pushed the feature/big-query-dts-operators branch from 498a7d9 to ee0a6de Compare August 15, 2019 06:28
@turbaszek turbaszek force-pushed the feature/big-query-dts-operators branch 3 times, most recently from 35d9a4c to a53b0cc Compare August 26, 2019 14:02
@coufon
Copy link
Contributor

coufon commented Sep 4, 2019

LGTM

@turbaszek turbaszek changed the title [AIRFLOW-4964][WIP-DONT-MERGE] Add BigQuery Data Transfer Hook and Operator [AIRFLOW-4964] Add BigQuery Data Transfer Hook and Operator Sep 5, 2019
@mik-laj mik-laj force-pushed the feature/big-query-dts-operators branch from a53b0cc to 7db44cc Compare September 5, 2019 08:50
@potiuk potiuk merged commit 1b1d79c into apache:master Sep 6, 2019
potiuk added a commit to PolideaInternal/airflow that referenced this pull request Sep 6, 2019
@potiuk
fixup! [AIRFLOW-4964] Add BigQuery Data Transfer Hook and Operator (a…
f482330 
…pache#5769)
potiuk added a commit that referenced this pull request Sep 6, 2019
@potiuk
fixup! [AIRFLOW-4964] Add BigQuery Data Transfer Hook and Operator (#…
a39ace9 
…5769)
@turbaszek turbaszek deleted the feature/big-query-dts-operators branch September 19, 2019 11:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mik-laj mik-laj mik-laj approved these changes

@potiuk potiuk potiuk approved these changes

Assignees
No one assigned
Labels
provider:google Google (including GCP) related issues
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@turbaszek @coufon @potiuk @mik-laj