Incompatible snowflake-connector-python and pyarrow dependencies

[cbuffett]

Apache Airflow version

2.6.1

What happened

The 2.6.1-python3.7 constraints specify pyarrow==9.0.0 and snowflake-connector-python=3.0.3. However, version 3.0.0 of snowflake-connector-python bumped the pyarrow dependency from >=8.0.0,<8.1.0 to >=10.0.1,<10.1.0. This results in a dependency resolution error when building a custom Airflow docker image. This only appears to happen when using the pandas compatible version of snowflake-connector-python

#0 151.6 There are incompatible versions in the resolved dependencies:
#0 151.6   pyarrow==9.0.0 (from -c https://raw.githubusercontent.com/apache/airflow/constraints-2.6.1/constraints-3.7.txt (line 470))
#0 151.6   pyarrow>=3.0.0 (from db-dtypes==1.1.1->-c https://raw.githubusercontent.com/apache/airflow/constraints-2.6.1/constraints-3.7.txt (line 229))
#0 151.6   pyarrow<10.1.0,>=10.0.1 (from snowflake-connector-python[pandas]==3.0.3->-c https://raw.githubusercontent.com/apache/airflow/constraints-2.6.1/constraints-3.7.txt (line 557))

What you think should happen instead

No response

How to reproduce

When building the 2.6.1 Airflow image, if the dependencies include snowflake-connector-python[pandas], a dependency incompatibility error is generated due to a mismatch required between pyarrow versions.

Operating System

Ubuntu 20.04 LTS

Versions of Apache Airflow Providers

No response

Deployment

Docker-Compose

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.

[potiuk]

Surely. Constraints of airflow do not include pandas extra. As explained in our documentation, when you want to install/upgrede providers independently from the airflow core, you should not use constraints

https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html#installing-upgrading-downgrading-providers-separately-from-airflow-core

Constraints are only to install standard airflow + standard providers in reproducible way and then you can install (and then test) additional extras and dependencies as separate installation step without constraints.

There is no guarantee for that - sometimes conflicting dependencies will make it impossible to have a working set of dependencies in which case you should follow https://airflow.apache.org/docs/apache-airflow/stable/best-practices.html#handling-conflicting-complex-python-dependencies

In the upcoming versiom of the documentation more examples and explanation has been added so that you can be better informed about it - see #31882 - this new docs will be published in the next few days together with releasing of 2.6.2.