use-feature=2020-resolver breaks airflow installation setup (lots of dependency error)

[Alibm73-zz]

Apache Airflow version: 1.10.12
Environment: VMware VM
Cloud provider or hardware configuration: 8 G RAM, 50G HDD, 4Cores
OS:
NAME="CentOS Linux"
VERSION="7 (Core)"
ID="centos"
ID_LIKE="rhel fedora"
VERSION_ID="7"
PRETTY_NAME="CentOS Linux 7 (Core)"
ANSI_COLOR="0;31"
CPE_NAME="cpe:/o:centos:centos:7"
HOME_URL="https://www.centos.org/"
BUG_REPORT_URL="https://bugs.centos.org/"

CENTOS_MANTISBT_PROJECT="CentOS-7"
CENTOS_MANTISBT_PROJECT_VERSION="7"
REDHAT_SUPPORT_PRODUCT="centos"
REDHAT_SUPPORT_PRODUCT_VERSION="7"
Kernel:
Linux mycentosvm 3.10.0-1160.6.1.el7.x86_64 #1 SMP Tue Nov 17 13:59:11 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
Install tools:
yum groupinstall "Development tools"
yum install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel python3-devel wget gcc gcc-c++ libffi-devel cyrus-sasl-devel
yum install mysql-server [version8]
[set global_explicit, create airflow database and user then give permissions]
alternative python 3.7 [pip 20.2.4 setuptools 47.1.0 ]

What happened:
All kinds of errors:
I copied https://raw.githubusercontent.com/apache/airflow/constraints-1.10.12/constraints-3.7.txt to a file called req.txt to be able to change the dependencies

1. Error1:

ERROR: Could not satisfy constraints for 'pyhive': installation from path or url cannot be constrained to a version

My solution1:
Install specified pyhive manually

pip install PyHive==0.6.3

2. Error2:

ERROR: Cannot install apache-airflow, apache-airflow and apache-airflow because these package versions have conflicting dependencies.

The conflict is caused by:
    email-validator 1.1.1 depends on idna>=2.0.0
    requests 2.24.0 depends on idna<3 and >=2.5
    moto 1.3.14 depends on idna<2.9 and >=2.5

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies

My solution2:

pip install idna==2.9

3. Error3:

ERROR: Cannot install apache-airflow[async,celery,devel,devel-hadoop,druid,hdfs,hive,jdbc,mysql,rabbitmq,ssh] 1.10.12, apache-airflow and apache-airflow because these package versions have conflicting dependencies.

The conflict is caused by:
    apache-airflow[async,celery,devel,devel-hadoop,druid,hdfs,hive,jdbc,mysql,rabbitmq,ssh] 1.10.12 depends on pytest<6.0.0; extra == "devel_hadoop"
    pytest-rerunfailures 9.0 depends on pytest>=5.0
    pytest-xdist 2.0.0 depends on pytest>=6.0.0

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies

My solution3:

pip install pytest==6.0.0

4. Error4:

  Downloading eventlet-0.14.0.tar.gz (279 kB)
     |████████████████████████████████| 279 kB 404 kB/s
    ERROR: Command errored out with exit status 1:
     command: /usr/local/bin/python3.7 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-ozxuwjov/eventlet_711991891ddc448ba8dd97630910301c/setup.py'"'"'; __file__='"'"'/tmp/pip-install-ozxuwjov/eventlet_711991891ddc448ba8dd97630910301c/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-pip-egg-info-aw4qpa8h
         cwd: /tmp/pip-install-ozxuwjov/eventlet_711991891ddc448ba8dd97630910301c/
    Complete output (8 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/tmp/pip-install-ozxuwjov/eventlet_711991891ddc448ba8dd97630910301c/setup.py", line 3, in <module>
        from eventlet import __version__
      File "/tmp/pip-install-ozxuwjov/eventlet_711991891ddc448ba8dd97630910301c/eventlet/__init__.py", line 42
        except ImportError, e:
                          ^
    SyntaxError: invalid syntax
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

My solution4:

 pip install eventlet==0.29.1 

5. Error5:

bitmq,ssh]==1.10.12) (0.4.17)
ERROR: Could not find a version that satisfies the requirement pytest<6.0.0; extra == "devel_hadoop" (from apache-airflow[async,celery,devel,devel-hadoop,druid,hdfs,hive,jdbc,mysql,rabbitmq,ssh])
ERROR: No matching distribution found for pytest<6.0.0; extra == "devel_hadoop"

then install older version and i get this error:

  Downloading paramiko-2.7.1-py2.py3-none-any.whl (206 kB)
     |████████████████████████████████| 206 kB 377 kB/s
Collecting pytest-xdist
  Using cached pytest_xdist-2.0.0-py2.py3-none-any.whl (36 kB)
ERROR: Cannot install apache-airflow[async,celery,devel,devel-hadoop,druid,hdfs,hive,jdbc,mysql,rabbitmq,ssh] 1.10.12 and apache-airflow because these package versions have conflicting dependencies.

The conflict is caused by:
    apache-airflow[async,celery,devel,devel-hadoop,druid,hdfs,hive,jdbc,mysql,rabbitmq,ssh] 1.10.12 depends on pytest<6.0.0; extra == "devel_hadoop"
    pytest-xdist 2.0.0 depends on pytest>=6.0.0

To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies

I even don't know any of my solutions even work!

How to reproduce it:
Run this command

pip install \
 apache-airflow[async,celery,devel,devel_hadoop,druid,hdfs,hive,jdbc,mysql,rabbitmq,ssh]==1.10.12 \
 --constraint "https://raw.githubusercontent.com/apache/airflow/constraints-1.10.12/constraints-3.7.txt" \
 --use-feature=2020-resolver

Anything else we need to know:
According to the announcement linked below, pip will introduce a new dependency resolver in October 2020, which will be more robust but might break some existing setups. Therefore they are suggesting users to try running their pip install scripts at least once (in dev mode) with this option: --use-feature=2020-resolver to anticipate any potential issue before the new resolver becomes the default in October 2020 with pip version 20.3.

How often does this problem occur? Once? Every time etc?
Everytime, the dependency problem is a nightmare, especially with adding hadoop extras

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!

[Alibm73-zz]

forgot this issue.
6.

Collecting hdfs[avro,dataframe,kerberos]>=2.0.4
  Downloading hdfs-2.5.8.tar.gz (41 kB)
     |████████████████████████████████| 41 kB 390 kB/s
ERROR: Could not satisfy constraints for 'hdfs': installation from path or url cannot be constrained to a version

[turbaszek]

@Alibm73 this seems to be related to already ongoing effort of solving dependencies issues #12508 #10854

[Alibm73-zz]

@Alibm73 this seems to be related to already ongoing effort of solving dependencies issues #12508 #10854

So what's the solution. will installing earlier versions help? I need some guidance
thnx

[potiuk]

Don't use --use-feature=2020-resolver

[potiuk]

Closing as duplicate of #12508 #10854