Fix various BigQuery to MSSQL/MySQL transfer issues

[jon-fearer]

closes: BigQueryToMsSqlOperator uses deprecated method and doesn't use keyword arguments
related: Fix BigQuery data extraction in BigQueryToMySqlOperator

This fixes an issue with the BigQueryToMsSqlOperator that is similar to what was encountered with the BigQueryToMySqlOperator. Specifically, this PR:

• Consolidates "_bq_get_data" from both operators into a single function with the updated logic from the BigQueryToMySqlOperator
• Updates the deprecated "get_tabledata" function to call "list_rows" using key word arguments instead of positional
• Adds a few missing type hints

Thanks in advance to maintainers. Please let me know if you have any thoughts or feedback.

[boring-cyborg]

Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:

• Pay attention to the quality of your code (flake8, mypy and type annotations). Our pre-commits will help you with that.
• In case of a new feature add useful documentation (in docstrings or in docs/ directory). Adding a new operator? Check this short guide Consider adding an example DAG that shows how users should use it.
• Consider using Breeze environment for testing locally, it’s a heavy docker but it ships with a working Airflow and a lot of integrations.
• Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
• Please follow ASF Code of Conduct for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
• Be sure to read the Airflow Coding style.
  Apache Airflow is a community-driven project and together we are making it better 🚀.
  In case of doubts contact the developers at:
  Mailing List: dev@airflow.apache.org
  Slack: https://s.apache.org/airflow-slack

[potiuk]

The tests are failing and likely related to your chanves - examples are not importable

[jon-fearer]

@potiuk Thanks for the heads up. I had a circular import between the new utils file I created and the two operators. This has been fixed.

[github-actions]

The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.

[potiuk]

Some tests are failing for the changes.

[jon-fearer]

@potiuk Those tests should be fixed now

[jon-fearer]

@potiuk It looks like the MSSQL tests stalled(?) because these are the only logs I can get. Are you able to trigger a re-run?

2021-12-08T07:31:40.6610060Z Found online and idle hosted runner in the current repository's organization account that matches the required labels: 'ubuntu-20.04'
2021-12-08T07:31:40.7176800Z Waiting for a Hosted runner in the 'organization' to pick this job...
2021-12-08T07:31:41.1324041Z Job is waiting for a hosted runner to come online.
2021-12-08T07:31:45.8070939Z Job is about to start running on the hosted runner: GitHub Actions 24 (hosted)

Edit - this seems to have sorted itself out

[potiuk]

did

[jon-fearer]

@potiuk It looks like the tests are failing during environment setup. Is that what you see too?

  ERROR: for mssqlsetup  Container "de99c5129d49" is unhealthy.
  Encountered errors while bringing up the project.
  CONTAINER ID   IMAGE                                                       COMMAND                  CREATED          STATUS                                  PORTS                                                                                                                               NAMES
  de99c5129d49   mcr.microsoft.com/mssql/server:2017-latest                  "/opt/mssql/bin/nonr…"   32 seconds ago   Restarting (1) Less than a second ago                                                                                                                                       airflow-integration-mssql_mssql_1

Edit - this seems to have sorted itself out

[jon-fearer]

The docs spelling check wasn't recognizing BigQueryHook as a valid word, so I added it here

[eladkal]

@jon-fearer can you fix conflicts?

[jon-fearer]

@eladkal Conflicts resolved. I used merge, let me know if rebase is preferred here and I can do that instead

[potiuk]

@eladkal Conflicts resolved. I used merge, let me know if rebase is preferred here and I can do that instead

It's good - we squash&merge so this makes no big difference.