[AIRFLOW-1338] Fix incompatible GCP dataflow hook #2388
Conversation
Codecov Report
@@ Coverage Diff @@
## master #2388 +/- ##
==========================================
+ Coverage 69.23% 69.26% +0.02%
==========================================
Files 146 146
Lines 11227 11227
==========================================
+ Hits 7773 7776 +3
+ Misses 3454 3451 -3
Continue to review full report at Codecov.
|
|
@alexvanboxel want to have a look? |
|
+1 merged.. I think @alexvanboxel is out, and changes LGTM |
|
Thanks @criccomini, I'll sync with @yk5 to fix the updating.md. |
| @@ -142,6 +142,7 @@ def check_previous(): | |||
| 'google-api-python-client>=1.5.0, <1.6.0', | |||
| 'oauth2client>=2.0.2, <2.1.0', | |||
| 'PyOpenSSL', | |||
| 'google-cloud-dataflow', | |||
There was a problem hiding this comment.
The reason I am asking this is due to the issues with dependencies as stated in this 2 Jira issues:
There was a problem hiding this comment.
I see, fine with removing this for now. We'll fix it on the google-data-flow side.
There was a problem hiding this comment.
Thanks @fenglu-g . I shall submit a PR removing 'google-cloud-dataflow' and will mention you so that you can approve it.
GCP dataflow hook is incompatible with recent google-cloud-dataflow
/apache-beam release (>=2.0.0). The DataflowPipelineRunner has
renamed to DataflowRunner, which breaks the existing
gcp_dataflow_hook. This PR updates the runner name.
FYI- this is the apache beam commit (https://goo.gl/9PhGQ6) that
introduces the incompatible change.