[AIRFLOW-5682] Allow labels in gcs_to_bq operator

[lyallcooper]

Make sure you have checked all steps below.

Jira

• My PR addresses the following Airflow Jira issues and references them in the PR title.

Description

• Adds support for labels when using the GoogleCloudStorageToBigQuery
  operator.

Tests

• My PR adds the following unit test: test_bigquery.TestLabelsInRunLoad

Commits

• My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters (not including Jira issue reference)
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

Documentation

• In case of new functionality, my PR adds documentation that describes how to use it.
  • All the public functions and the classes in the PR contain docstrings that explain what it does

[mik-laj]

According to the integration guide, you should also add default labels with the Airflow version
https://docs.google.com/document/d/1_rTdJSLCt0eyrAylmmgYc3yZr-_h51fVlnvMmWqhCkY/edit?ts=5bb72dfd#

[mik-laj]

Sample PR: https://github.com/apache/airflow/pull/6296/files

[lyallcooper]

@mik-laj sorry, I don't seem to have access to that google doc.

[lyallcooper]

Also it seems the other BQ-related operators that support labels don't pass in the Airflow version as a default label.

[mik-laj]

Currently, this integration is being analyzed and will be refactored later to make it comply with the guide. We want to ensure similar behavior for all GCP integrations, but BigQuery just isn't done yet.

@TobKed Do you want to add something? I have the impression that you are responsible for this integration.

[TobKed]

1. @mik-laj is right, it will be good idea to set default label to something like:

from airflow.version import version as airflow_version
_AIRFLOW_VERSION = 'v' + airflow_version.replace('.', '-').replace('+', '-')

2. We have plan to refactor BQ hooks and operators.
   Discussion is here: Review an improvement plan for Bigquery operators PolideaInternal/airflow#231
   One of the idea is to pass both the google.cloud.bigquery.job.QueryJobConfig class as well as dict representation. (see also: https://googleapis.dev/python/bigquery/latest/generated/google.cloud.bigquery.job.QueryJobConfig.html#google.cloud.bigquery.job.QueryJobConfig)

[codecov-io]

Codecov Report

Merging #6351 into master will decrease coverage by 0.37%.
The diff coverage is 100%.

@@            Coverage Diff             @@
##           master    #6351      +/-   ##
==========================================
- Coverage   80.07%   79.69%   -0.38%     
==========================================
  Files         616      616              
  Lines       35794    35800       +6     
==========================================
- Hits        28662    28532     -130     
- Misses       7132     7268     +136

Impacted Files	Coverage Δ
airflow/operators/gcs_to_bq.py	71.01% <100%> (+0.42%)	⬆️
airflow/gcp/hooks/bigquery.py	70.27% <100%> (+0.08%)	⬆️
airflow/operators/mysql_operator.py	0% <0%> (-100%)	⬇️
airflow/operators/mysql_to_hive.py	0% <0%> (-100%)	⬇️
airflow/executors/sequential_executor.py	47.61% <0%> (-52.39%)	⬇️
airflow/utils/log/colored_log.py	72.72% <0%> (-20.46%)	⬇️
airflow/utils/sqlalchemy.py	77.96% <0%> (-15.26%)	⬇️
airflow/executors/__init__.py	63.26% <0%> (-4.09%)	⬇️
airflow/utils/dag_processing.py	56.23% <0%> (-2.67%)	⬇️
airflow/hooks/hive_hooks.py	75.82% <0%> (-1.79%)	⬇️
... and 3 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update f2b7f5a...9b2565f. Read the comment docs.

[lyallcooper]

Alright, updated to add in the airflow-version label automatically

[mik-laj]

Can you use assert_called_once_with instead creating special function? This will make the code clearer.

[mik-laj]

   Summary of failed tests
tests.gcp.hooks.test_bigquery.TestLabelsInRunJob.test_run_query_with_arg Failure:builtins.AssertionError
tests.gcp.hooks.test_bigquery.TestLabelsInRunLoad.test_run_query_with_arg Failure:builtins.AssertionError

Travis is sad. Can you fix it?

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[TobKed]

Hi @lyallcooper could rebase on the newest master please? There are some changes which cause travis to be sad

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.