[AIRFLOW-4462] MSSQL backend broken

[deodeveloper]

• Airflow manual trigger/scheduler…trigger doesn't always work on mssql(azure mssql, mssql 2017, other versions)

• fix mssql server as airflow backend. Applies only to sql server 2005 later versions and azure sql db. Issue is with execution_date used as primary key in task_instance and a equality comparison column in dag_run. When comparing equality python microsecond precision to sql server, it returns false when the millisecond is non-zero,e.g, 2019-04-20T18:51:35.033. This change will make the execution_date from datetime to datetime2(precision=6) matches with mysql precision. The date equality comparision works as desired. Microsoft also recommends to use datetime2.

Without this fix, airflow is not usable in sql server and azure sql server db.

Make sure you have checked all steps below.

Jira

• My PR addresses the following Airflow Jira issues and references them in the PR title. For example, "[AIRFLOW-XXX] My Airflow PR"
  • https://issues.apache.org/jira/browse/AIRFLOW-XXX
  • In case you are fixing a typo in the documentation you can prepend your commit with [AIRFLOW-XXX], code changes always need a Jira issue.
  • In case you are proposing a fundamental code change, you need to create an Airflow Improvement Proposal (AIP).
  • In case you are adding a dependency, check if the license complies with the ASF 3rd Party License Policy.

Description

• Here are some details about my PR, including screenshots of any UI changes:

Tests

• My PR adds the following unit tests OR does not need testing for this extremely good reason:

Commits

• My commits all reference Jira issues in their subject lines, and I have squashed multiple commits if they address the same issue. In addition, my commits follow the guidelines from "How to write a good git commit message":
  1. Subject is separated from body by a blank line
  2. Subject is limited to 50 characters (not including Jira issue reference)
  3. Subject does not end with a period
  4. Subject uses the imperative mood ("add", not "adding")
  5. Body wraps at 72 characters
  6. Body explains "what" and "why", not "how"

Documentation

• In case of new functionality, my PR adds documentation that describes how to use it.
  • All the public functions and the classes in the PR contain docstrings that explain what it does
  • If you implement backwards incompatible changes, please leave a note in the Updating.md so we can assign it to a appropriate release

Code Quality

• Passes flake8

[OmerJog]

@deodeveloper You have some flake8 errors:

./airflow/migrations/versions/2a2738575802_change_execution_date_type_from_.py:9:2: W291 trailing whitespace
./airflow/migrations/versions/2a2738575802_change_execution_date_type_from_.py:11:2: W291 trailing whitespace
./airflow/migrations/versions/2a2738575802_change_execution_date_type_from_.py:42:77: W291 trailing whitespace
./airflow/migrations/versions/2a2738575802_change_execution_date_type_from_.py:43:96: W291 trailing whitespace
./airflow/migrations/versions/2a2738575802_change_execution_date_type_from_.py:122:77: W291 trailing whitespace
./airflow/migrations/versions/2a2738575802_change_execution_date_type_from_.py:123:96: W291 trailing whitespace
./airflow/migrations/versions/2a2738575802_change_execution_date_type_from_.py:213:104: W291 trailing whitespace
./airflow/migrations/versions/2a2738575802_change_execution_date_type_from_.py:214:111: E501 line too long (115 > 110 characters)

Does this also fixes https://issues.apache.org/jira/browse/AIRFLOW-1987 ?

[deodeveloper]

@OmerJog
updated the pull request to address flake8 warnings.
AIRFLOW-1987 is related version 1.9.x; I don't see that issue with 1.10.x. I haven't tested with 1.9.x, so I removed that tag form Jira AIRFLOW-4462
With this merge, people using mssql can run airflow.

Not sure what causing the travis ci build failure, it looks like one of the docker run is failing without any response. My changes are not related to the build failure. Can you please suggest how to deal with travis docker build failure?

[OmerJog]

I think you should create your PR against Master and not against v1-10-stable
But wait to hear from commiters before you take action.

[potiuk]

Yep. It should be base on master @OmerJog @deodeveloper

[deodeveloper]

Yep. It should be base on master @OmerJog @deodeveloper

I created a pull request against the master branch. pull request #5255 (#5255)
But this fix also applied 10.x stable to make airflow work with sql server, not sure how we going to merge the changes to 10.x stable.