[AIRFLOW-2715] Pick up the region setting while launching Dataflow templates

[janhicken]

Make sure you have checked all steps below.

Jira

• My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"
  • https://issues.apache.org/jira/browse/AIRFLOW-XXX
  • In case you are fixing a typo in the documentation you can prepend your commit with [AIRFLOW-XXX], code changes always need a Jira issue.

Description

• Here are some details about my PR, including screenshots of any UI changes:
  To launch an instance of a Dataflow template in the configured region,
  the API service.projects().locations().teplates() instead of
  service.projects().templates() has to be used. Otherwise, all jobs will
  always be started in us-central1.

In case there is no region configured, the default region us-central1 will get picked up.

To make it even worse, the polling for the job status already honors the
region parameter and will search for the job in the wrong region in the
current implementation. Because the job's status is not found, the
corresponding Airflow task will hang.

This PR is a second approach and follow-up of #4125

Tests

• My PR adds the following unit tests OR does not need testing for this extremely good reason:
  tests.contrib.hooks.test_gcp_dataflow_hook.DataFlowTemplateHookTest has been modified

Commits

• My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters (not including Jira issue reference)
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

Documentation

• In case of new functionality, my PR adds documentation that describes how to use it.
  • When adding new operators/hooks/sensors, the autoclass documentation generation needs to be added.

Code Quality

• Passes flake8

[kaxil]

Can you add this to DataflowTemplateOperator?

[janhicken]

Do you mean some documentation?

[kaxil]

Can you add test for the following scenarios?

• When users doesn't set the region parameter?

[janhicken]

This is already done in tests.contrib.hooks.test_gcp_dataflow_hook.DataFlowTemplateHookTest

[Fokko]

@janhicken Restarting Travis, can you rebase onto master? It might be that master was failing at that time.

[codecov-io]

Codecov Report

Merging #4139 into master will decrease coverage by 4.53%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #4139      +/-   ##
==========================================
- Coverage   77.66%   73.12%   -4.54%     
==========================================
  Files         199      199              
  Lines       16290    17807    +1517     
==========================================
+ Hits        12652    13022     +370     
- Misses       3638     4785    +1147

Impacted Files	Coverage Δ
airflow/www_rbac/security.py	63.88% <0%> (-28.73%)	⬇️
airflow/operators/docker_operator.py	75% <0%> (-22.68%)	⬇️
airflow/operators/bash_operator.py	70% <0%> (-21.38%)	⬇️
airflow/models.py	70.95% <0%> (-21.29%)	⬇️
airflow/api/common/experimental/trigger_dag.py	80.39% <0%> (-19.61%)	⬇️
airflow/www_rbac/views.py	65.24% <0%> (-7.08%)	⬇️
airflow/jobs.py	77.36% <0%> (+0.27%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e6291e8...75f8141. Read the comment docs.

[janhicken]

Rebase has been done

[Fokko]

Rerunning the failed tests.