pip auto remove dependency of other package while installing one package

[Firkraag]

Description

I have ipython installed already, while requires prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0.
Then I install aws-shell, which requires prompt-toolkit-1.0.18, contradicting to the requirement of ipython.
Then pip automatically removes prompt-toolkit required by ipython, making ipython completely useless.

Expected behavior

When the dependencies of the new package contradicts to those of installed packages, pip will give a warning message or abort instead of removing the dependencies of installed packages automatically.

pip version

pip 21.0.1

Python version

Python 3.9.2

OS

macos mojave version 10.14.6

How to Reproduce

1. run python3 -m venv venv to create an virtualenv and activate by venv/bin/activate
2. run pip install ipython
3. run pip install aws-shell
4. run ipython
5. ipython cannot start

Output

$ pip install ipython
Looking in indexes: https://pypi.douban.com/simple
Collecting ipython
  Downloading https://pypi.doubanio.com/packages/c9/b1/82cbe2b856386f44f37fdae54d9b425813bd86fe33385c9d658d64826098/ipython-7.22.0-py3-none-any.whl (785 kB)
     |████████████████████████████████| 785 kB 702 kB/s
Requirement already satisfied: pickleshare in ./lib/python3.9/site-packages (from ipython) (0.7.5)
Requirement already satisfied: traitlets>=4.2 in ./lib/python3.9/site-packages (from ipython) (5.0.5)
Requirement already satisfied: backcall in ./lib/python3.9/site-packages (from ipython) (0.2.0)
Requirement already satisfied: setuptools>=18.5 in ./lib/python3.9/site-packages (from ipython) (54.1.2)
Requirement already satisfied: pygments in ./lib/python3.9/site-packages (from ipython) (2.8.1)
Requirement already satisfied: appnope in ./lib/python3.9/site-packages (from ipython) (0.1.2)
Collecting prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0
  Downloading https://pypi.doubanio.com/packages/eb/e6/4b4ca4fa94462d4560ba2f4e62e62108ab07be2e16a92e594e43b12d3300/prompt_toolkit-3.0.18-py3-none-any.whl (367 kB)
     |████████████████████████████████| 367 kB 941 kB/s
Requirement already satisfied: pexpect>4.3 in ./lib/python3.9/site-packages (from ipython) (4.8.0)
Requirement already satisfied: jedi>=0.16 in ./lib/python3.9/site-packages (from ipython) (0.18.0)
Requirement already satisfied: decorator in ./lib/python3.9/site-packages (from ipython) (5.0.6)
Requirement already satisfied: parso<0.9.0,>=0.8.0 in ./lib/python3.9/site-packages (from jedi>=0.16->ipython) (0.8.2)
Requirement already satisfied: ptyprocess>=0.5 in ./lib/python3.9/site-packages (from pexpect>4.3->ipython) (0.7.0)
Requirement already satisfied: wcwidth in ./lib/python3.9/site-packages (from prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0->ipython) (0.2.5)
Requirement already satisfied: ipython-genutils in ./lib/python3.9/site-packages (from traitlets>=4.2->ipython) (0.2.0)
Installing collected packages: prompt-toolkit, ipython
  Attempting uninstall: prompt-toolkit
    Found existing installation: prompt-toolkit 1.0.18
    Uninstalling prompt-toolkit-1.0.18:
      Successfully uninstalled prompt-toolkit-1.0.18
Successfully installed ipython-7.22.0 prompt-toolkit-3.0.18
$ pip install aws-shell
Looking in indexes: https://pypi.douban.com/simple
Collecting aws-shell
  Using cached https://pypi.doubanio.com/packages/cb/01/9e5d72ebf69dc670ceaab4cfe4d4db92256ae2c332fc5b16c90301065ed6/aws_shell-0.2.2-py2.py3-none-any.whl (50 kB)
Collecting prompt-toolkit<1.1.0,>=1.0.0
  Downloading https://pypi.doubanio.com/packages/64/27/5fd61a451d086ad4aa806dc72fe1383d2bc0e74323668672287f616d5d51/prompt_toolkit-1.0.18-py3-none-any.whl (245 kB)
     |████████████████████████████████| 245 kB 548 kB/s
Requirement already satisfied: awscli<2.0.0,>=1.16.10 in ./lib/python3.9/site-packages (from aws-shell) (1.19.48)
Requirement already satisfied: boto3<2.0.0,>=1.9.0 in ./lib/python3.9/site-packages (from aws-shell) (1.17.48)
Requirement already satisfied: configobj<6.0.0,>=5.0.6 in ./lib/python3.9/site-packages (from aws-shell) (5.0.6)
Requirement already satisfied: Pygments<3.0.0,>=2.1.3 in ./lib/python3.9/site-packages (from aws-shell) (2.8.1)
Requirement already satisfied: PyYAML<5.5,>=3.10 in ./lib/python3.9/site-packages (from awscli<2.0.0,>=1.16.10->aws-shell) (5.4.1)
Requirement already satisfied: colorama<0.4.4,>=0.2.5 in ./lib/python3.9/site-packages (from awscli<2.0.0,>=1.16.10->aws-shell) (0.4.3)
Requirement already satisfied: rsa<4.8,>=3.1.2 in ./lib/python3.9/site-packages (from awscli<2.0.0,>=1.16.10->aws-shell) (4.7.2)
Requirement already satisfied: s3transfer<0.4.0,>=0.3.0 in ./lib/python3.9/site-packages (from awscli<2.0.0,>=1.16.10->aws-shell) (0.3.6)
Requirement already satisfied: botocore==1.20.48 in ./lib/python3.9/site-packages (from awscli<2.0.0,>=1.16.10->aws-shell) (1.20.48)
Requirement already satisfied: docutils<0.16,>=0.10 in ./lib/python3.9/site-packages (from awscli<2.0.0,>=1.16.10->aws-shell) (0.15.2)
Requirement already satisfied: urllib3<1.27,>=1.25.4 in ./lib/python3.9/site-packages (from botocore==1.20.48->awscli<2.0.0,>=1.16.10->aws-shell) (1.26.4)
Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in ./lib/python3.9/site-packages (from botocore==1.20.48->awscli<2.0.0,>=1.16.10->aws-shell) (0.10.0)
Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in ./lib/python3.9/site-packages (from botocore==1.20.48->awscli<2.0.0,>=1.16.10->aws-shell) (2.8.1)
Requirement already satisfied: six in ./lib/python3.9/site-packages (from configobj<6.0.0,>=5.0.6->aws-shell) (1.15.0)
Requirement already satisfied: wcwidth in ./lib/python3.9/site-packages (from prompt-toolkit<1.1.0,>=1.0.0->aws-shell) (0.2.5)
Requirement already satisfied: pyasn1>=0.1.3 in ./lib/python3.9/site-packages (from rsa<4.8,>=3.1.2->awscli<2.0.0,>=1.16.10->aws-shell) (0.4.8)
Installing collected packages: prompt-toolkit, aws-shell
  Attempting uninstall: prompt-toolkit
    Found existing installation: prompt-toolkit 3.0.18
    Uninstalling prompt-toolkit-3.0.18:
      Successfully uninstalled prompt-toolkit-3.0.18
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
ipython 7.22.0 requires prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0, but you have prompt-toolkit 1.0.18 which is incompatible.
Successfully installed aws-shell-0.2.2 prompt-toolkit-1.0.18
$ ipython
Traceback (most recent call last):
  File "/private/tmp/venv/bin/ipython", line 5, in <module>
    from IPython import start_ipython
  File "/private/tmp/venv/lib/python3.9/site-packages/IPython/__init__.py", line 56, in <module>
    from .terminal.embed import embed
  File "/private/tmp/venv/lib/python3.9/site-packages/IPython/terminal/embed.py", line 16, in <module>
    from IPython.terminal.interactiveshell import TerminalInteractiveShell
  File "/private/tmp/venv/lib/python3.9/site-packages/IPython/terminal/interactiveshell.py", line 21, in <module>
    from prompt_toolkit.formatted_text import PygmentsTokens
ModuleNotFoundError: No module named 'prompt_toolkit.formatted_text'

Code of Conduct

I agree to follow the PSF Code of Conduct.

[pradyunsg]

Hi! Thanks for filing this issue!

pip will give a warning message or abort instead of removing the dependencies of installed packages automatically.

pip does provide an error message. Copying from your post:

ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
ipython 7.22.0 requires prompt-toolkit!=3.0.0,!=3.0.1,<3.1.0,>=2.0.0, but you have prompt-toolkit 1.0.18 which is incompatible.

This is all the information that pip has about the conflict, which it presents to the user (original discussion: #7744). The discussion about changing this behavior to abort is happening at #9094.

[pradyunsg]

Consolidating into #9094.