Airflow upgrade_check gives false positives on DbApi functions implementations check

[amaraliou]

Apache Airflow version:
1.10.14

Kubernetes version (if you are using kubernetes) (use kubectl version):
N/A

Environment:

• Cloud provider or hardware configuration: GCP
• OS (e.g. from /etc/os-release): Ubuntu 18.04.4 LTS
• Kernel (e.g. uname -a): N/A
• Install tools: pip

What happened:

Running airflow upgrade_check threw the following warnings:

  1.  Class <class 'airflow.hooks.http_hook.HttpHook'> incorrectly implements the function run while inheriting from BaseHook. Please make this class inherit from airflow.hooks.db_api_hook.DbApiHook instead
  2.  Class <class 'airflow.contrib.hooks.bigquery_hook.BigQueryHook'> incorrectly implements the function get_pandas_df while inheriting from BaseHook. Please make this class inherit from airflow.hooks.db_api_hook.DbApiHook instead
  3.  Class <class 'airflow.contrib.hooks.bigquery_hook.BigQueryHook'> incorrectly implements the function run while inheriting from BaseHook. Please make this class inherit from airflow.hooks.db_api_hook.DbApiHook instead
  4.  Class <class 'airflow.contrib.hooks.bigquery_hook.BigQueryHook'> incorrectly implements the function get_records while inheriting from BaseHook. Please make this class inherit from airflow.hooks.db_api_hook.DbApiHook instead
  5.  Class <class '_hooks.gc_bigquery_hook.BigQueryStandardHook'> incorrectly implements the function get_pandas_df while inheriting from BaseHook. Please make this class inherit from airflow.hooks.db_api_hook.DbApiHook instead
  6.  Class <class '_hooks.gc_bigquery_hook.BigQueryStandardHook'> incorrectly implements the function run while inheriting from BaseHook. Please make this class inherit from airflow.hooks.db_api_hook.DbApiHook instead
  7.  Class <class '_hooks.gc_bigquery_hook.BigQueryStandardHook'> incorrectly implements the function get_records while inheriting from BaseHook. Please make this class inherit from airflow.hooks.db_api_hook.DbApiHook instead
  8.  Class <class 'unusual_prefix_9358643817187292af016429303a09afd4152fc5_gc_bigquery_hook.BigQueryStandardHook'> incorrectly implements the function get_pandas_df while inheriting from BaseHook. Please make this class inherit from airflow.hooks.db_api_hook.DbApiHook instead
  9.  Class <class 'unusual_prefix_9358643817187292af016429303a09afd4152fc5_gc_bigquery_hook.BigQueryStandardHook'> incorrectly implements the function run while inheriting from BaseHook. Please make this class inherit from airflow.hooks.db_api_hook.DbApiHook instead
 10.  Class <class 'unusual_prefix_9358643817187292af016429303a09afd4152fc5_gc_bigquery_hook.BigQueryStandardHook'> incorrectly implements the function get_records while inheriting from BaseHook. Please make this class inherit from airflow.hooks.db_api_hook.DbApiHook instead
 11.  Class <class '_hooks.gc_bigquery_hook.GCBigQueryHook'> incorrectly implements the function get_pandas_df while inheriting from BaseHook. Please make this class inherit from airflow.hooks.db_api_hook.DbApiHook instead
 12.  Class <class '_hooks.gc_bigquery_hook.GCBigQueryHook'> incorrectly implements the function run while inheriting from BaseHook. Please make this class inherit from airflow.hooks.db_api_hook.DbApiHook instead
 13.  Class <class '_hooks.gc_bigquery_hook.GCBigQueryHook'> incorrectly implements the function get_records while inheriting from BaseHook. Please make this class inherit from airflow.hooks.db_api_hook.DbApiHook instead
 14.  Class <class 'unusual_prefix_9358643817187292af016429303a09afd4152fc5_gc_bigquery_hook.GCBigQueryHook'> incorrectly implements the function get_pandas_df while inheriting from BaseHook. Please make this class inherit from airflow.hooks.db_api_hook.DbApiHook instead
 15.  Class <class 'unusual_prefix_9358643817187292af016429303a09afd4152fc5_gc_bigquery_hook.GCBigQueryHook'> incorrectly implements the function run while inheriting from BaseHook. Please make this class inherit from airflow.hooks.db_api_hook.DbApiHook instead
 16.  Class <class 'unusual_prefix_9358643817187292af016429303a09afd4152fc5_gc_bigquery_hook.GCBigQueryHook'> incorrectly implements the function get_records while inheriting from BaseHook. Please make this class inherit from airflow.hooks.db_api_hook.DbApiHook instead

The custom hooks all inherit from BigQueryHook. Looking at the code, it inherits both DbApiHook and GoogleCloudBaseHook which inherits BaseHook. I also noticed that the same BigQueryHook implementation appears in the 2.0-stable branch so this is probably a false positive. The warning (1) is already fixed by #14344.

What you expected to happen:

No warnings

How to reproduce it: N/A

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[omar-binary]

any luck with this? 👆

[potiuk]

@kaxil - maybe we should add this one as well to the last upgrade_check release? Sounds like pretty annoying warning for popular operators?

[ephraimbuddy]

any luck with this? 👆

What version of the upgrade-check are you using because I believe this was resolved in #14344

[omar-binary]

I can still see it in V1.3.0

Apache Airflow version:
1.10.15

airflow@d9b97fe4d7cc:/opt/airflow$ airflow upgrade_check --version
1.3.0
airflow@d9b97fe4d7cc:/opt/airflow$ airflow upgrade_check --config /opt/airflow/plugins/upgrade.yml
Using config file: /opt/airflow/plugins/upgrade.yml

====================================================================================== STATUS ======================================================================================

Check for latest versions of apache-airflow and checker...................................................................................................................SUCCESS
Remove airflow.AirflowMacroPlugin class...................................................................................................................................SUCCESS
Ensure users are not using custom metaclasses in custom operators.........................................................................................................SUCCESS
Chain between DAG and operator not allowed................................................................................................................................SUCCESS
Connection.conn_type is not nullable......................................................................................................................................SUCCESS
Custom Executors now require full path....................................................................................................................................SUCCESS
/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/google/cloud/hooks/bigquery.py:119: DeprecationWarning: This method will be deprecated. Please use `BigQueryHook.get_client` method
  warnings.warn(
Hooks that run DB functions must inherit from DBApiHook...................................................................................................................FAIL
Fernet is enabled by default..............................................................................................................................................SUCCESS
GCP service account key deprecation.......................................................................................................................................SUCCESS
Unify hostname_callable option in core section............................................................................................................................SUCCESS
Changes in import paths of hooks, operators, sensors and others...........................................................................................................SUCCESS
Legacy UI is deprecated by default........................................................................................................................................SUCCESS
Logging configuration has been moved to new section.......................................................................................................................SUCCESS
Removal of Mesos Executor.................................................................................................................................................SUCCESS
No additional argument allowed in BaseOperator............................................................................................................................SUCCESS
SendGrid email uses old airflow.contrib module............................................................................................................................SUCCESS
Check Spark JDBC Operator default connection name.........................................................................................................................SUCCESS
Changes in import path of remote task handlers............................................................................................................................SUCCESS
Connection.conn_id is not unique..........................................................................................................................................SUCCESS
Use CustomSQLAInterface instead of SQLAInterface for custom data models...................................................................................................SUCCESS
Found 1 problem.

================================================================================= RECOMMENDATIONS ==================================================================================

Hooks that run DB functions must inherit from DBApiHook
-------------------------------------------------------
Hooks that run DB functions must inherit from DBApiHook instead of BaseHook

Problems:

  1.  Class <class 'airflow.providers.google.cloud.hooks.bigquery.BigQueryHook'> incorrectly implements the function get_pandas_df while inheriting from BaseHook. Please make this class inherit from airflow.hooks.db_api_hook.DbApiHook instead

airflow@d9b97fe4d7cc:/opt/airflow$ 

[kaxil]

PR: #16543

[kaxil]

Closed by #16543 .

Will cut 1.4.0rc2 today and start the vote to release it

[eladkal]

fixed by #16543 and released in 1.4.0