CPU package depends on non-CPU torch package

[Korijn]

🐛 Bug

The torch-vision CPU wheel depends on the non-CPU torch wheel

Evidence: torchvision-0.6.0+cpu-cp36-cp36m-linux_x86_64.whl\torchvision-0.6.0+cpu.dist-info\METADATA:

Metadata-Version: 2.1
Name: torchvision
Version: 0.6.0+cpu
Summary: image and video datasets and models for torch deep learning
Home-page: https://github.com/pytorch/vision
Author: PyTorch Core Team
Author-email: soumith@pytorch.org
License: BSD
Platform: UNKNOWN
Requires-Dist: numpy
Requires-Dist: torch (==1.5.0)           <<<==== PROBLEM ====
Requires-Dist: pillow (>=4.1.1)
Provides-Extra: scipy
Requires-Dist: scipy ; extra == 'scipy'

To Reproduce

I've run into this with the poetry package manager for version 1.4.0 and 0.5.0, but the issue is also present in newer versions as can be seen in the previous snippet.

λ poetry lock -vvv                                                                         
Using virtualenv: C:\Users\korij_000\AppData\Local\pypoetry\Cache\virtualenvs\bla-8bWMJ8mE-
py3.6                                                                                      
Updating dependencies                                                                      
Resolving dependencies...                                                                  
   1: fact: bla is 0.1.0                                                                   
   1: derived: bla                                                                         
   1: fact: bla depends on torch (1.4.0+cpu)                                               
   1: fact: bla depends on torchvision (0.5.0+cpu)                                         
   1: selecting bla (0.1.0)                                                                
   1: derived: torchvision (0.5.0+cpu)                                                     
   1: derived: torch (1.4.0+cpu)                                                           
torch: 1 packages found for torchvision 0.5.0+cpu                                          
torch: 1 packages found for torch 1.4.0+cpu                                                
torch: Downloading wheel: torchvision-0.5.0%2Bcpu-cp27-cp27m-linux_x86_64.whl              
   1: fact: torchvision (0.5.0+cpu) depends on numpy (*)                                   
   1: fact: torchvision (0.5.0+cpu) depends on six (*)                                     
   1: fact: torchvision (0.5.0+cpu) depends on torch (1.4.0)                               
   1: fact: torchvision (0.5.0+cpu) depends on pillow (>=4.1.1)                            
   1: derived: pillow (>=4.1.1)                                                            
   1: conflict: torchvision (0.5.0+cpu) depends on torch (1.4.0)      <<<==== PROBLEM ====
   1: ! not torch (1.4.0) is satisfied by torch (1.4.0+cpu)                                
   1: ! which is caused by "bla depends on torch (1.4.0+cpu)"                              
   1: ! thus: torchvision is forbidden                                                     
   1: ! torchvision (0.5.0+cpu) is satisfied by torchvision (0.5.0+cpu)                    
   1: ! which is caused by "bla depends on torchvision (0.5.0+cpu)"                        
   1: ! thus: version solving failed                                                       
   1: Version solving took 1.964 seconds.                                                  
   1: Tried 1 solutions.                                                                   
                                                                                           
[SolverProblemError]                                                                       
Because torchvision (0.5.0+cpu) depends on torch (1.4.0)                                   
 and bla depends on torch (1.4.0+cpu), torchvision is forbidden.                           
So, because bla depends on torchvision (0.5.0+cpu), version solving failed.                

Note that I also had to work around another issue just to get poetry to find these packages since they are not on PyPI. See pytorch/pytorch#39779

Expected behavior

I would expect the CPU wheel for torchvision to depend on the CPU version for torch.

[fmassa]

Thanks for opening the issue!

cc @seemethere @peterjc123 could you have a look at this?

[seemethere]

The way the dependency works is that it states that the 1.5.0 version is valid for this specific package.

That can refer to 1.5.0 alone or, in the case of cpu packages it can refer to 1.5.0+cpu.

With conda this is a bit easier since there is a dependency workaround we use to designate different CUDA versions / CPU, but unfortunately that workaround does not work with python wheels.

If you do note, our official documentation for installation asks you to specify that you explicitly want the CPU version when installing both torch and torchvision:

pip install torch==1.5.0+cpu torchvision==0.6.0+cpu -f https://download.pytorch.org/whl/torch_stable.html

Technically with the way that pip handles dependencies 1.5.0+cpu should be a valid replacement for 1.5.0, but this may be an issue with poetry having different dependency resolutions than pip.

[Korijn]

Thanks for the reply. I'll investigate and try opening an issue at poetry.

[seemethere]

This may also be related to pytorch/pytorch#39779

[Korijn]

This may also be related to pytorch/pytorch#39779

Related yes but a separate problem. I opened pytorch/pytorch#39779 for that

[seemethere]

This may also be related to pytorch/pytorch#39779

Related yes but a separate problem. I opened pytorch/pytorch#39779 for that

oh haha,I guess I should've noticed that you also filed that originally!

[Korijn]

From PEP 440 under Version matching

If the specified version identifier is a public version identifier (no local version label), then the local version label of any candidate versions MUST be ignored when matching versions.

Following Poetry's debug output, it first adds the constraints from pyproject.toml, then reads the METADATA file in the torchvision wheel, and adds the constraint torch==1.4.0 to the solver.

Given the above line from the PEP, that constraint can only be solved by ignoring the local version identifier.

How "ignoring" is to be interpreted is not entirely clear to me (is any local identifier acceptable or should it be absent?), but I guess poetry takes it as "there can't be a local version identifier", because the constraint in the METADATA file requires an exact match (==).

Having said that, I also wanted to respond to something you said earlier:

this may be an issue with poetry having different dependency resolutions than pip.

As far as I am aware, pip doesn't have a (constraint-based) solver. It just installs dependencies one by one.

Anyway, I'll open an issue at poetry and see what they have to say.

[Korijn]

Created python-poetry/poetry#2543, let's see what their thoughts are.

[Jefffish09]

I met the same problem.

Any solution now?

[Korijn]

No response from maintainers yet on the poetry side, just other people chiming in with the same issue/duplicates. No change in pytorch/torch vision side yet either.

[pawamoy]

I've added a workaround in python-poetry/poetry#2543 (comment)

[Korijn]

I've reached the conclusion that the problem lies with poetry's dependency resolver which appears not to be fully compliant with PEP440. Check out python-poetry/poetry#2543