Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

rez-pip install of black does not add tomli as a requirement. #1341

Open
jopollack opened this issue Jun 28, 2022 · 11 comments
Open

rez-pip install of black does not add tomli as a requirement. #1341

jopollack opened this issue Jun 28, 2022 · 11 comments
Labels
bug rez-pip ingesting py pkgs into rez (pip, wheels, etc)

Comments

@jopollack
Copy link

Rez Version 2.106.0

I'm not sure what's going on here, but when running rez-pip to install black, it leaves out one of the requirements, namely tomli. It installs tomli, but it doesn't add it to the package.py, so the resulting package is broken and I have to manually call tomli in the rez-env or add it into the package.py as a requirement.

rez-pip black --install -p /work/rez --python-version 3.7
Collecting black
  Downloading https://files.pythonhosted.org/packages/2b/70/1d0e33a4df4ed73e9f02f698a29b5d94ff58e39f029c939ecf96a10fb1f3/black-22.6.0-py3-none-any.whl (156kB)
     |████████████████████████████████| 163kB 25.6MB/s 
Collecting typing-extensions>=3.10.0.0; python_version < "3.10" (from black)
  Downloading https://files.pythonhosted.org/packages/75/e1/932e06004039dd670c9d5e1df0cd606bf46e29a28e65d5bb28e894ea29c9/typing_extensions-4.2.0-py3-none-any.whl
Collecting typed-ast>=1.4.2; python_version < "3.8" and implementation_name == "cpython" (from black)
  Downloading https://files.pythonhosted.org/packages/04/93/482d12fd3334b53ec4087e658ab161ab23affcf8b052166b4cf972ca673b/typed_ast-1.5.4-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (843kB)
     |████████████████████████████████| 849kB 34.4MB/s 
Collecting mypy-extensions>=0.4.3 (from black)
  Downloading https://files.pythonhosted.org/packages/5c/eb/975c7c080f3223a5cdaff09612f3a5221e4ba534f7039db34c35d95fa6a5/mypy_extensions-0.4.3-py2.py3-none-any.whl
Collecting pathspec>=0.9.0 (from black)
  Downloading https://files.pythonhosted.org/packages/42/ba/a9d64c7bcbc7e3e8e5f93a52721b377e994c22d16196e2b0f1236774353a/pathspec-0.9.0-py2.py3-none-any.whl
Collecting tomli>=1.1.0; python_full_version < "3.11.0a7" (from black)
  Downloading https://files.pythonhosted.org/packages/97/75/10a9ebee3fd790d20926a90a2547f0bf78f371b2f13aa822c759680ca7b9/tomli-2.0.1-py3-none-any.whl
Collecting platformdirs>=2 (from black)
  Downloading https://files.pythonhosted.org/packages/ed/22/967181c94c3a4063fe64e15331b4cb366bdd7dfbf46fcb8ad89650026fec/platformdirs-2.5.2-py3-none-any.whl
Collecting click>=8.0.0 (from black)
  Downloading https://files.pythonhosted.org/packages/c2/f1/df59e28c642d583f7dacffb1e0965d0e00b218e0186d7858ac5233dce840/click-8.1.3-py3-none-any.whl (96kB)
     |████████████████████████████████| 102kB 10.4MB/s 
Collecting importlib-metadata; python_version < "3.8" (from click>=8.0.0->black)
  Downloading https://files.pythonhosted.org/packages/d2/a2/8c239dc898138f208dd14b441b196e7b3032b94d3137d9d8453e186967fc/importlib_metadata-4.12.0-py3-none-any.whl
Collecting zipp>=0.5 (from importlib-metadata; python_version < "3.8"->click>=8.0.0->black)
  Using cached https://files.pythonhosted.org/packages/80/0e/16a7ee38617aab6a624e95948d314097cc2669edae9b02ded53309941cfc/zipp-3.8.0-py3-none-any.whl
Installing collected packages: typing-extensions, typed-ast, mypy-extensions, pathspec, tomli, platformdirs, zipp, importlib-metadata, click, black
Successfully installed black-22.6.0 click-8.1.3 importlib-metadata-4.12.0 mypy-extensions-0.4.3 pathspec-0.9.0 platformdirs-2.5.2 tomli-2.0.1 typed-ast-1.5.4 typing-extensions-4.2.0 zipp-3.8.0
WARNING: You are using pip version 19.2.3, however version 22.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
15:16:52 WARNING  Metadata: missing: ['Home-page', 'Author'], warnings: []
15:16:52 WARNING  Metadata: missing: ['Home-page', 'Author'], warnings: []
15:16:52 WARNING  Metadata: missing: ['Home-page', 'Author'], warnings: []
15:16:52 INFO     Installed [black-22.6.0] /work/rez/black/22.6.0/package.py (6e0a42ad67bd5da79eb94cd760314c7f48292a49)
15:16:52 INFO     Installed [click-8.1.3] /work/rez/click/8.1.3/package.py (f98bb122ce29a25ef03cd5ba3ebdfa16f3a2d0c5)
15:16:52 INFO     Installed [importlib_metadata-4.12.0] /work/rez/importlib_metadata/4.12.0/package.py (1ea170f09dad84a21d7e5918bcf56066e867f747)
15:16:52 INFO     Installed [mypy_extensions-0.4.3] /work/rez/mypy_extensions/0.4.3/package.py (88a70aca30cb79a278872594adf043dc6c40af99)
15:16:52 INFO     Installed [pathspec-0.9.0] /work/rez/pathspec/0.9.0/package.py (88a70aca30cb79a278872594adf043dc6c40af99)
15:16:53 INFO     Installed [platformdirs-2.5.2] /work/rez/platformdirs/2.5.2/package.py (88a70aca30cb79a278872594adf043dc6c40af99)
15:16:53 INFO     Installed [tomli-2.0.1] /work/rez/tomli/2.0.1/package.py (88a70aca30cb79a278872594adf043dc6c40af99)
15:16:53 INFO     Installed [typed_ast-1.5.4] /work/rez/typed_ast/1.5.4/package.py (d9e9d80193dcd9578844ec4c2c22c9366ef0b88a)
15:16:53 INFO     Installed [typing_extensions-4.2.0] /work/rez/typing_extensions/4.2.0/package.py (88a70aca30cb79a278872594adf043dc6c40af99)
15:16:53 INFO     Installed [zipp-3.8.0] /work/rez/zipp/3.8.0/package.py (88a70aca30cb79a278872594adf043dc6c40af99)
15:16:53 INFO     10 packages were installed.

# -*- coding: utf-8 -*-

name = 'black'

version = '22.6.0'

description = 'The uncompromising code formatter.'

authors = ['Łukasz Langa lukasz@langa.pl']

tools = [
    'black',
    'blackd'
]

requires = [
    'click-8.0.0+',
    'platformdirs-2+',
    'mypy_extensions-0.4.3+',
    'pathspec-0.9.0+'
]

variants = [['python-3.7', 'typing_extensions-3.10.0.0+', 'typed_ast-1.4.2+']]

def commands():
    env.PYTHONPATH.append('{root}/python')
    env.PATH.append('{root}/bin')

help = [['Home Page', 'https://github.com/psf/black']]

timestamp = 1656454612

hashed_variants = True

is_pure_python = True

from_pip = True

pip_name = 'black (22.6.0)'

format_version = 2

Traceback (most recent call last):
  File "/work/rez/black/22.6.0/6e0a42ad67bd5da79eb94cd760314c7f48292a49/bin/black", line 6, in <module>
    from black import patched_main
  File "/work/rez/black/22.6.0/6e0a42ad67bd5da79eb94cd760314c7f48292a49/python/black/__init__.py", line 53, in <module>
    from black.files import (
  File "/work/rez/black/22.6.0/6e0a42ad67bd5da79eb94cd760314c7f48292a49/python/black/files.py", line 31, in <module>
    import tomli as tomllib
ModuleNotFoundError: No module named 'tomli'

As you can see, it installs tomli, but it doesn't end up in the package.py, so attempting to run black causes an import error.
@nerdvegas
Copy link
Contributor

nerdvegas commented Jul 1, 2022 via email

@instinct-vfx
Copy link
Contributor

My guess is that's because it is a conditional requirement and something likely breaks:

Requires-Dist: tomli (>=1.1.0) ; python_full_version < "3.11.0a7"

@instinct-vfx
Copy link
Contributor

(the difference to the other requirements besides the alpha version is python_full_version instead of python_version

@instinct-vfx
Copy link
Contributor

Also: Works fine on windows. It ends up in the variant as it is version specific:

name = 'black'

version = '22.6.0'

description = 'The uncompromising code formatter.'

authors = ['Łukasz Langa lukasz@langa.pl']

tools = [
    'black.exe',
    'blackd.exe'
]

requires = [
    'platformdirs-2+',
    'pathspec-0.9.0+',
    'click-8.0.0+',
    'mypy_extensions-0.4.3+'
]

variants = \
    [['platform-windows',
      'arch-AMD64',
      'python-3.7',
      'typed_ast-1.4.2+',
      'typing_extensions-3.10.0.0+',
      'tomli-1.1.0+']]

@JeanChristopheMorinPerso
Copy link
Member

We do support python_full_version, see https://github.com/AcademySoftwareFoundation/rez/blob/master/src/rez/utils/pip.py#L391. And as you can see, we use the packaging library to evaluate the markers (which is the library also used by pip and plenty of other tools).

So we would have to debug that issue and see what's going on as it's supposed to work.

@JeanChristopheMorinPerso JeanChristopheMorinPerso added rez-pip ingesting py pkgs into rez (pip, wheels, etc) bug labels Jul 1, 2022
@instinct-vfx
Copy link
Contributor

What distro are you on? As i can not repro it on windows

@ColinKennedy
Copy link

FWIW I reproduced this on Windows 3 days ago when the ticket was made but tried again last night and couldn't reproduce it. The only difference between then and now is that I added a python 3 Rez package via @instinct-vfx 's rezpy. Before that, pip was getting found from Rez's own python. That may be a contributing factor.

@ColinKennedy
Copy link

Yep, just confirmed again. I removed the python 3 packages that I'd installed via rezpy, Rez used its own python (rez-python --version == "3.7.9"), and then this happened.

variants = \
    [['platform-windows',
      'arch-AMD64',
      'python-3.7',
      'typing_extensions-3.10.0.0+',
      'typed_ast-1.4.2+']]

Note the missing tomli.

I then reinstalled a python Rez package (3.7.9) using rezpy

rezpy --python_version=3.7.9

Then re-ran rez-pip and got these variants

variants = \
    [['platform-windows',
      'arch-AMD64',
      'python-3.7',
      'typed_ast-1.4.2+',
      'tomli-1.1.0+',
      'typing_extensions-3.10.0.0+']]

I suspect @instinct-vfx uses a python package via rezpy, which is why he cannot reproduce it at the moment. But if you remove the python package and use Rez's own python, it should happen.

@jopollack
Copy link
Author

In my case I discovered the bug on Linux using rez-pip in a default environment with rez 2.106.0, so the existing pip or python packages rez comes with may be a factor.

@ColinKennedy
Copy link

Your original reproduction doesn't show which python / pip was used (I'd thought rez-pip always outputs that at the start. Did you omit that information or was it truly not shown in your output?) but yes, likely Rez's own rez-python was chosen or a python package that may not be set up properly. In my personal experience, in Windows, rez-bind fails to build a proper Python package and then rez-pip detects that and falls back to rez-python.

@jopollack
Copy link
Author

Yes, I omitted it. It found pip-19.2.3 in the python-3.7 I had installed.

rez-pip black --install -p /work/rez --python-version 3.7
15:16:46 INFO     Trying to use pip from python package
15:16:48 INFO     Found pip-19.2.3 inside /work/rez/python/3.7.7/package.py. Will use it via /work/rez/python/3.7.7/bin/python3.7
15:16:48 INFO     Installing 'black' with pip taken from '/work/rez/python/3.7.7/bin/python3.7'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug rez-pip ingesting py pkgs into rez (pip, wheels, etc)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@instinct-vfx @nerdvegas @ColinKennedy @jopollack @JeanChristopheMorinPerso