[AIRFLOW-1688] Automatically add load.time_partitioning to bigquery_hook when table name includes $

[albertocalderari]

Allow automatic creation of a new partitioned tables when the '$' symbol is in the table name adding the option:

load.timePartitioning: {type: 'DAY'}

i.e. my_dataset.my_table$20170101 would be automatically loaded as a new partitioned table without the need of creating it manually.

Dear Airflow maintainers,

Please accept this PR. I understand that it will not be reviewed until I have checked off all the steps below!

JIRA

• My PR addresses the following Airflow JIRA issues and references them in the PR title. For example, "[AIRFLOW-1688] Automatically add load.time_partitioning to bigquery_hook when table name includes $"
  • https://issues.apache.org/jira/browse/AIRFLOW-1688

Description

• Here are some details about my PR, including screenshots of any UI changes:
  The gcs_to_bq operator throws and exception when trying to auto-create a new date partitioned table.
  To allow the table creation the load configuration needs the api option:
  load.timePartitioning:
  {type: 'DAY'}
  I will add a fix to identify date partitioned table from the presence of a $ in the table name and add the option.

Tests

• My PR adds the following unit tests OR does not need testing for this extremely good reason:
  I can't unit test this change, but I performed multiple integration tests and the code is now running in our prod environment for the last few months without issue.

Commits

• My commits all reference JIRA issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

[codecov-io]

Codecov Report

Merging #2669 into master will increase coverage by <.01%.
The diff coverage is n/a.

@@            Coverage Diff            @@
##           master   #2669      +/-   ##
=========================================
+ Coverage   71.69%   71.7%   +<.01%     
=========================================
  Files         154     154              
  Lines       11807   11807              
=========================================
+ Hits         8465    8466       +1     
+ Misses       3342    3341       -1

Impacted Files	Coverage Δ
airflow/models.py	87.14% <0%> (+0.04%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update cfc2f73...643af3c. Read the comment docs.

[wileeam]

I'd consider a regex here rather than just a check on the $ character

[albertocalderari]

I used $ as it is the partition decorator separator in BQ and won't change.
Also BQ doe not allow special characters in the table or dataset name (except the partition separator).

[wileeam]

Maybe you would want to add support for the other two nested options of timePartitioning: expirationMs and field?

[albertocalderari]

I agree with you on this, will try to add it sometime this week

[wileeam]

I would also consider some tests (based on the review comments as well)

[albertocalderari]

@wileeam I changed the review comments. I did perform integration testing on this change.

[albertocalderari]

@wileeam so... I managed to add the extra options and the unit tests. Unfortunately I have also managed to corrupt my fork. I have therefore created a new PR #2820 - #2820

Sorry about that.