Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[AIRFLOW-2795] Oracle to Oracle Transfer Operator #3639

Merged
merged 1 commit into from Jul 31, 2018

Conversation

marcusrehm
Copy link
Contributor

Make sure you have checked all steps below.

JIRA

Description

  • This PR adds a OracleToOracleTransfer operator in contrib operators.

Tests

  • My PR adds the following unit tests OR does not need testing for this extremely good reason:
    • tests/contrib/operators/test_oracle_to_oracle_transfer.py

Commits

  • My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
    1. Subject is separated from body by a blank line
    2. Subject is limited to 50 characters
    3. Subject does not end with a period
    4. Subject uses the imperative mood ("add", not "adding")
    5. Body wraps at 72 characters
    6. Body explains "what" and "why", not "how"

Documentation

  • In case of new functionality, my PR adds documentation that describes how to use it.
    • When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.

Code Quality

  • Passes git diff upstream/master -u -- "*.py" | flake8 --diff

@codecov-io
Copy link

codecov-io commented Jul 25, 2018

Codecov Report

Merging #3639 into master will decrease coverage by 0.34%.
The diff coverage is n/a.

Impacted file tree graph

@@            Coverage Diff             @@
##           master    #3639      +/-   ##
==========================================
- Coverage   77.51%   77.17%   -0.35%     
==========================================
  Files         205      206       +1     
  Lines       15751    15769      +18     
==========================================
- Hits        12210    12169      -41     
- Misses       3541     3600      +59
Impacted Files Coverage Δ
airflow/hooks/pig_hook.py 0% <0%> (-100%) ⬇️
airflow/utils/dag_processing.py 89.45% <0%> (-0.43%) ⬇️
airflow/models.py 88.56% <0%> (-0.02%) ⬇️
airflow/www_rbac/views.py 72.85% <0%> (ø) ⬆️
airflow/www/views.py 68.88% <0%> (ø) ⬆️
airflow/minihivecluster.py 0% <0%> (ø)
airflow/www/app.py 100% <0%> (+0.98%) ⬆️
airflow/utils/sqlalchemy.py 89.13% <0%> (+15.21%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dfa7b26...6c0720e. Read the comment docs.

Copy link
Contributor

@Fokko Fokko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please revise the docstring

@apply_defaults
def __init__(
self,
oracle_destination_conn_id,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The source and destination differ in order in the docstring/arguments

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done!

oracle_destination_conn_id,
destination_table,
oracle_source_conn_id,
source_sql,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is called sql in the docstring

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done too.

cursor = src_conn.cursor()
self.log.info("Querying data from source: {0}".format(
self.oracle_source_conn_id))
cursor.execute(self.source_sql, self.source_sql_params)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ideally we would like to template the sql using the ninja2 templating engine. Why would we need the extra params? Note: I'm not an Oracle expert :-)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @Fokko , both source_sql and source_sql_params are templated fields. The reason to use extra params is because in some DBAPI, special characters are only considered when passed using parameters.

@marcusrehm marcusrehm force-pushed the oracle_to_oracle_transfer branch 3 times, most recently from 81999b0 to 7971332 Compare July 25, 2018 13:49
@marcusrehm
Copy link
Contributor Author

Just bumping up

@Fokko Fokko merged commit 9983466 into apache:master Jul 31, 2018
lxneng pushed a commit to lxneng/incubator-airflow that referenced this pull request Aug 10, 2018
aliceabe pushed a commit to aliceabe/incubator-airflow that referenced this pull request Jan 3, 2019
ashb pushed a commit to ashb/airflow that referenced this pull request Jan 10, 2019
cfei18 pushed a commit to cfei18/incubator-airflow that referenced this pull request Jan 23, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants