Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracking issue: PyPI dependencies pixi can't manage yet. #771

Open
ruben-arts opened this issue Feb 2, 2024 · 31 comments
Open

Tracking issue: PyPI dependencies pixi can't manage yet. #771

ruben-arts opened this issue Feb 2, 2024 · 31 comments
Labels
pypi Issue related to PyPI dependencies

Comments

@ruben-arts
Copy link
Contributor

There are a few PyPI packages pixi can't install yet, where pip can.

Please paste your examples in this issue so we have a list of known packages we can track, test and benchmark with along the way.

Information we would like:

  1. What did you run and what was the outcome?
    e.g. pixi add --pypi packagex
    If it doesn't recreate in an empty environment please share your pixi.toml that recreates the issue.

  2. What error did pixi return?
    e.g.

    × RECORD file doesn't match wheel contents: missing hash for mediapipe/version.txt (expected sha256=-fE2KU)
    
  3. Can pip install the package?
    Does pip install packagex work?

  4. What platform are you on?
    e.g. linux-64

  5. Did you find a workaround, if so please explain.
    e.g. build it into a conda package, using a custom fork, etc.

Your input would greatly help us improve the pixi's experience!
Thanks in advance! ❤️

@pablovela5620
Copy link
Contributor

Mediapipe

  1. pixi add --pypi mediapipe
  2. × RECORD file doesn't match wheel contents: missing hash for mediapipe/version.txt (expected sha256=-fE2KU)
  3. python -m pip install mediapipe does work
  4. platform osx-arm64
  5. work around was just to use pip to install using pixi tasks
[project]
name = "ipdscan"
version = "0.1.0"
description = "Add a short description here"
authors = ["pablovela5620 <pablovela5620@gmail.com>"]
channels = ["conda-forge"]
platforms = ["osx-arm64"]

[tasks]
mp-install = "python -m pip install mediapipe"

[dependencies]
python = "3.11.*"
pip = ">=23.3.2,<23.4"
rerun-sdk = ">=0.12.0,<0.13"
requests = ">=2.31.0,<2.32"
tqdm = ">=4.66.1,<4.67"

[pypi-dependencies]
imutils = "*"

@tylerjw
Copy link

tylerjw commented Feb 4, 2024

I wanted to use this library that is packaged in pypi: https://github.com/spirali/elsie

Here is the error I was seeing when trying to use pixi add:

 WARN rattler_installs_packages::index::package_database: errors while processing source distributions:
  × failed to resolve `pypi-dependencies`, due to underlying error
  ╰─▶ No metadata could be extracted for the following available artifacts:
      	- lxml-4.6.5.tar.gz

Error:   × error while processing source distribution 'lxml-4.6.5.tar.gz':
  │  could not build wheel: <string>:67: DeprecationWarning: pkg_resources is deprecated as an API. See https://
  │ setuptools.pypa.io/en/latest/pkg_resources.html
  │ 
  help: Probably an error during processing of source distributions. Please check the error message above.

Posting in the discord channel got it working with this fix:

pixi add python lxml
pixi add --pypi elsie

The reason for this is:

there seems to be an issue with one of the lxml source distributions. Meaning you have to locally build the package but that is not working because of the error it is giving you. pixi reports back on errorous packages where pip probably continues trying other versions.

@liquidcarbon
Copy link
Contributor

liquidcarbon commented Feb 12, 2024

pixi add duckdbpixi add --pypi duckdb ❌ vs pixi run pip install duckdb ✅ on Windows

PS C:\code> pixi init duckdb-pip
✔ Initialized project in C:\code\duckdb-pip
PS C:\code> cd .\duckdb-pip\
PS C:\code\duckdb-pip> pixi add python=3.11
✔ Added python=3.11
PS C:\code\duckdb-pip> pixi add duckdb     
  × could not determine any available versions for duckdb on win-64. Either the package could not be found or version constraints on other
  │ dependencies result in a conflict.
  ╰─▶ Cannot solve the request because of: No candidates were found for duckdb *.


PS C:\code\duckdb-pip> pixi add --pypi duckdb
  × could not build wheel: warning: no files found matching '*.h' under directory 'duckdb'
  │ warning: no files found matching '*.hpp' under directory 'duckdb'
  │ warning: no files found matching '*.cpp' under directory 'duckdb'
  │ warning: no files found matching '*.h' under directory 'src'
  │ warning: manifest_maker: MANIFEST.in, line 6: 'recursive-include' expects <dir> <pattern1> <pattern2> ...
  │
  │ C:\Users\AKISLU~1\AppData\Local\Temp\.tmpkvrxrc\venv\Lib\site-packages\setuptools\command\build_py.py:207: _Warning: Package 'duckdb-      
  │ stubs.value' is absent from the `packages` configuration.
  │ !!
  │
  │         ********************************************************************************
  │         ############################
  │         # Package would be ignored #
  │         ############################
  │         Python recognizes 'duckdb-stubs.value' as an importable package[^1],
  │         but it is absent from setuptools' `packages` configuration.
  │
  │         This leads to an ambiguous overall configuration. If you want to distribute this
  │         package, please make sure that 'duckdb-stubs.value' is explicitly added
  │         to the `packages` configuration field.
  │
  │         Alternatively, you can also rely on setuptools' discovery methods
  │         (for example by using `find_namespace_packages(...)`/`find_namespace:`
  │         instead of `find_packages(...)`/`find:`).
  │
  │         You can read more about "package discovery" on setuptools documentation page:
  │
  │         - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
  │
  │         If you don't want 'duckdb-stubs.value' to be distributed and are
  │         already explicitly excluding 'duckdb-stubs.value' via
  │         `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
  │         you can try to use `exclude_package_data`, or `include-package-data=False` in
  │         combination with a more fine grained `package-data` configuration.
  │
  │         You can read more about "package data files" on setuptools documentation page:
  │
  │         - https://setuptools.pypa.io/en/latest/userguide/datafiles.html
  │
  │
  │         [^1]: For Python, any directory (with suitable naming) can be imported,
  │               even if it does not contain any `.py` files.
  │               On the other hand, currently there is no concept of package data
  │               directory, all directories are treated like packages.
  │         ********************************************************************************
  │
  │ !!
  │   check.warn(importable)
  │ C:\Users\AKISLU~1\AppData\Local\Temp\.tmpkvrxrc\venv\Lib\site-packages\setuptools\command\build_py.py:207: _Warning: Package 'duckdb-      
  │ stubs.value.constant' is absent from the `packages` configuration.
  │ !!
  │
  │         ********************************************************************************
  │         ############################
  │         # Package would be ignored #
  │         ############################
  │         Python recognizes 'duckdb-stubs.value.constant' as an importable package[^1],
  │         but it is absent from setuptools' `packages` configuration.
  │
  │         This leads to an ambiguous overall configuration. If you want to distribute this
  │         package, please make sure that 'duckdb-stubs.value.constant' is explicitly added
  │         to the `packages` configuration field.
  │
  │         Alternatively, you can also rely on setuptools' discovery methods
  │         (for example by using `find_namespace_packages(...)`/`find_namespace:`
  │         instead of `find_packages(...)`/`find:`).
  │
  │         You can read more about "package discovery" on setuptools documentation page:
  │
  │         - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
  │
  │         If you don't want 'duckdb-stubs.value.constant' to be distributed and are
  │         already explicitly excluding 'duckdb-stubs.value.constant' via
  │         `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
  │         you can try to use `exclude_package_data`, or `include-package-data=False` in
  │         combination with a more fine grained `package-data` configuration.
  │
  │         You can read more about "package data files" on setuptools documentation page:
  │
  │         - https://setuptools.pypa.io/en/latest/userguide/datafiles.html
  │
  │
  │         [^1]: For Python, any directory (with suitable naming) can be imported,
  │               even if it does not contain any `.py` files.
  │               On the other hand, currently there is no concept of package data
  │               directory, all directories are treated like packages.
  │         ********************************************************************************
  │
  │ !!
  │   check.warn(importable)
  │ C:\Users\AKISLU~1\AppData\Local\Temp\.tmpkvrxrc\venv\Lib\site-packages\setuptools\command\build_py.py:207: _Warning: Package
  │ 'duckdb.experimental' is absent from the `packages` configuration.
  │ !!
  │
  │         ********************************************************************************
  │         ############################
  │         # Package would be ignored #
  │         ############################
  │         Python recognizes 'duckdb.experimental' as an importable package[^1],
  │         but it is absent from setuptools' `packages` configuration.
  │
  │         This leads to an ambiguous overall configuration. If you want to distribute this
  │         package, please make sure that 'duckdb.experimental' is explicitly added
  │         to the `packages` configuration field.
  │
  │         Alternatively, you can also rely on setuptools' discovery methods
  │         (for example by using `find_namespace_packages(...)`/`find_namespace:`
  │         instead of `find_packages(...)`/`find:`).
  │
  │         You can read more about "package discovery" on setuptools documentation page:
  │
  │         - https://setuptools.pypa.io/en/latest/userguide/package_discovery.html
  │
  │         If you don't want 'duckdb.experimental' to be distributed and are
  │         already explicitly excluding 'duckdb.experimental' via
  │         `find_namespace_packages(...)/find_namespace` or `find_packages(...)/find`,
  │         you can try to use `exclude_package_data`, or `include-package-data=False` in
  │         combination with a more fine grained `package-data` configuration.
  │
  │         You can read more about "package data files" on setuptools documentation page:
  │
  │         - https://setuptools.pypa.io/en/latest/userguide/datafiles.html
  │
  │
  │         [^1]: For Python, any directory (with suitable naming) can be imported,
  │               even if it does not contain any `.py` files.
  │               On the other hand, currently there is no concept of package data
  │               directory, all directories are treated like packages.
  │         ********************************************************************************
  │
  │ !!
  │   check.warn(importable)
  │ error: Microsoft Visual C++ 14.0 or greater is required. Get it with "Microsoft C++ Build Tools": https://visualstudio.microsoft.com/      
  │ visual-cpp-build-tools/
  │

PS C:\code\duckdb-pip> pixi add pip          
✔ Added pip
PS C:\code\duckdb-pip> pixi run pip install duckdb
Collecting duckdb
  Downloading duckdb-0.9.2-cp311-cp311-win_amd64.whl.metadata (798 bytes)
Downloading duckdb-0.9.2-cp311-cp311-win_amd64.whl (10.3 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 10.3/10.3 MB 21.8 MB/s eta 0:00:00
Installing collected packages: duckdb
Successfully installed duckdb-0.9.2

@liquidcarbon
Copy link
Contributor

liquidcarbon commented Feb 12, 2024

  1. work around was just to use pip to install using pixi tasks

@pablovela5620 I am also finding this necessary quite often, probably more often than @ruben-arts would like :)

In this pattern the environment definition is fragmented between pixi.toml tasks and pixi.lock. It could work, esp with pinned pip installs in tasks, but is it the intention?

@ruben-arts
Copy link
Contributor Author

In this pattern the environment definition is fragmented between pixi.toml tasks and pixi.lock. It could work, esp with pinned pip installs in tasks, but is it the intention?

@liquidcarbon This is indeed not what we want as UX but we simply need to develop more to support all PyPI packages. Its a weird bunch of requirements we have to support to be equivalent to pip. So please keep posting non working packages!

@ruben-arts
Copy link
Contributor Author

@liquidcarbon the duckdb on conda-forge has been continued on python-duckdb which should be available on windows. We'll keep the example for pypi to test more!

@tdejager
Copy link
Contributor

tdejager commented Feb 15, 2024

@liquidcarbon I cannot reproduce this, neither on OSX or on windows. It's strange it does not select the .whl for some reason.

@tdejager
Copy link
Contributor

@tylerjw actually it also fails for the same version in pip on apple silicon, like pip we error out when we cannot build the first source distribution.

@tdejager
Copy link
Contributor

tdejager commented Feb 15, 2024

@pablovela5620 so it seems mediapipe 10.9 is a package with an invalid RECORD file, I manually checked it and it's incorrect.

This is mentioned in the PyPa

During extraction, wheel installers verify all the hashes in RECORD against the file contents. Apart from RECORD and its signatures, installation will fail if any file in the archive is not both mentioned and correctly hashed in RECORD.

Mediapipe has a version.txt that is not mentioned in the RECORD for the 10.9 release.

Which in this case triggers the error, I'm unsure why pip does no do this. But I feel its good to adhere to the standard here.

In any case, mediapipe 10.8 does seem to work, you could use that instead.

Also see: google-ai-edge/mediapipe#5025

tdejager pushed a commit to prefix-dev/rip that referenced this issue Feb 15, 2024
These PR address 2 major issues and 1 small issue
* zip file stamps issue
alexcrichton/tar-rs#349 which we encountered
when installing tomli-2.1.0
* changed from_filename signature so it can also produce Stree, which
it's needed from pixi side
* lowercase Root-Is-Purelib so we can install elsie library (
prefix-dev/pixi#771 (comment))
@liquidcarbon
Copy link
Contributor

liquidcarbon commented Feb 15, 2024

@liquidcarbon I cannot reproduce this, neither on OSX or on windows. It's strange it does not select the .whl for some reason.

@tdejager Just tried on Win10, original comment was on Win11
same thing:

PS C:\Users\a\Desktop\code\duckdb-pip> pixi add python=3.11
✔ Added python 3.11.*
PS C:\Users\a\Desktop\code\duckdb-pip> pixi add duckdb
× could not determine any available versions for duckdb on win-64. Either the package could not be found or version
│ constraints on other dependencies result in a conflict.
╰─▶ Cannot solve the request because of: No candidates were found for duckdb *.

I was on Pixi 0.9.1, upped to 0.13.0 - same thing

But pixi add --pypi duckdb works. 🤷‍♂️

@ruben-arts
Copy link
Contributor Author

The conda package you add with pixi add duckdb should be pixi add python-duckdb

https://prefix.dev/channels/conda-forge/packages/python-duckdb

@liquidcarbon
Copy link
Contributor

@ruben-arts noted -- I just wasn't sure which part @tdejager was trying to reproduce

@awray3
Copy link

awray3 commented Feb 25, 2024

Tensorflow metal on Apple silicon MacOS 14.2.1:

pixi init tf-metal && cd tf-metal
pixi add "python>=3.11" "tensorflow>=2.13" pip
pixi add --pypi tensorflow-metal
> × failed to resolve `pypi-dependencies`, due to underlying error
  ╰─▶ The following packages are incompatible
      └─ tensorflow-metal * cannot be installed because there are no viable options:
         └─ tensorflow-metal 0.1.0 | 0.1.1 | 0.1.2 | 0.2.0 | 0.3.0 | 0.4.0 | 0.5.0 | 0.5.1 | 0.6.0 | 0.7.0 | 0.7.1
      | 0.8.0 | 1.0.0 | 1.0.1 | 1.1.0 is excluded because none of the artifacts are compatible with the Python
      interpreter or glibc version and there are no supported sdists

pixi run pip install tensorflow-metal
> Successfully installed tensorflow-metal-1.1.0

@baszalmstra
Copy link
Contributor

Most likely you are missing a system requirement: https://pixi.sh/latest/configuration/#the-system-requirements-table

Most likely macos=12.0

@awray3
Copy link

awray3 commented Feb 25, 2024

Nice, that fixed it. But would the generated pixi.toml work on a linux machine? I guess I would have to mark tensorflow-metal as platform dependent somehow. My goal is to have a pixi.toml that installs tensorflow metal on a mac and tensorflow with gpu support on linux.

EDIT: Nevermind, I found the example demonstrating how to do this. Thank you!

@ruben-arts ruben-arts added the pypi Issue related to PyPI dependencies label Feb 29, 2024
@roaldarbol
Copy link

Installing opencv-python-headless with conda works (through python -m pip install opencv-python-headless), but not with pixi. I'm on osx-64.

(base) ➜  idtracker pixi add --pypi opencv-python-headless
  × failed to resolve `pypi-dependencies`, due to underlying error
  ╰─▶ No metadata could be extracted for the following available artifacts:
        - opencv-python-headless-4.9.0.80.tar.gz

Error:   × error while processing source distribution 'opencv-python-headless-4.9.0.80.tar.gz':
  │  could not build wheel: Traceback (most recent call last):
  │   File "/var/folders/rh/y9ws4bgj7x50twypcwtk2ypr0000gp/T/.tmpvjLm8l/build_frontend.py", line 124, in <module>
  │     get_requires_for_build_wheel(backend, work_dir)
  │   File "/var/folders/rh/y9ws4bgj7x50twypcwtk2ypr0000gp/T/.tmpvjLm8l/build_frontend.py", line 58, in get_requires_for_build_wheel
  │     result = f()
  │              ^^^
  │   File "/var/folders/rh/y9ws4bgj7x50twypcwtk2ypr0000gp/T/.tmpvjLm8l/venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 325, in
  │ get_requires_for_build_wheel
  │     return self._get_build_requires(config_settings, requirements=['wheel'])
  │            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  │   File "/var/folders/rh/y9ws4bgj7x50twypcwtk2ypr0000gp/T/.tmpvjLm8l/venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
  │     self.run_setup()
  │   File "/var/folders/rh/y9ws4bgj7x50twypcwtk2ypr0000gp/T/.tmpvjLm8l/venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 487, in run_setup
  │     super().run_setup(setup_script=setup_script)
  │   File "/var/folders/rh/y9ws4bgj7x50twypcwtk2ypr0000gp/T/.tmpvjLm8l/venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 311, in run_setup
  │     exec(code, locals())
  │   File "<string>", line 10, in <module>
  │ ModuleNotFoundError: No module named 'skbuild'
  │ 
  help: Probably an error during processing of source distributions. Please check the error message above.

@baszalmstra
Copy link
Contributor

@roaldarbol The error message is absolutely terrible but if you add:

[system-requirements]
macos = "11.0"

It should work.

@roaldarbol
Copy link

It does indeed! Thanks!

@jacobbieker
Copy link

Trying to install torch-geometric-temporal with pixi add. It fails as it cannot find torch, which had already been install previously

  1. pixi add --pypi torch-geometric-temporal
  2. python -m pip install torch-geomtric-temporal does work, after the other dependencies are install with pixi
  3. platform linux-64
  4. work around was just to use pip to install using pixi tasks
    Configuration:
[project]
name = "graph_weather"
version = "0.1.0"
description = "Add a short description here"
authors = ["Jacob Bieker <jacob@bieker.tech>"]
channels = ["pyg", "nvidia", "conda-forge", "pytorch"]
platforms = ["linux-64"]

[tasks]
tinstall = "python -m pip install torch-geometric-temporal"

[dependencies]
python = "3.11.*"
torchvision = ">=0.16.1,<0.17"
pytorch-cluster = ">=1.6.3,<1.7"
pytorch-scatter = ">=2.1.2,<2.2"
pytorch-cuda = "12.1.*"
xarray = ">=2024.2.0,<2024.3"
pytorch-spline-conv = ">=1.2.2,<1.3"
pytorch = ">=2.1"
pandas = ">=2.2.1,<2.3"
h3-py = ">=3.7.6,<3.8"
numcodecs = ">=0.12.1,<0.13"
scipy = ">=1.12.0,<1.13"
zarr = ">=2.17.0,<2.18"
pyg = ">=2.5.0,<2.6"
tqdm = ">=4.66.2,<4.67"
pytorch-sparse = ">=0.6.18,<0.7"
lightning = ">=2.2.0.post0,<2.2.1"
einops = ">=0.7.0,<0.8"
fsspec = ">=2024.2.0,<2024.3"
datasets = ">=2.18.0,<2.19"
pip = ">=24.0,<25"


[pypi-dependencies]
pytest = "*"  # This means any version (this `*` is custom in pixi)
pre-commit = "*"
pysolar = "*"

@ruben-arts
Copy link
Contributor Author

Trying to install torch-geometric-temporal with pixi add. It fails as it cannot find torch, which had already been install previously

It does indeed not work. Testing the repro gives me this error:

  × failed to solve the pypi requirements of 'default' 'linux-64'
  ├─▶ failed to resolve pypi dependencies
  ├─▶ Failed to download and build: torch-scatter==2.1.2
  ├─▶ Failed to build: torch-scatter==2.1.2
  ╰─▶ Build backend failed to determine extra requires with `build_wheel()`:
      --- stdout:
      
      --- stderr:
      Traceback (most recent call last):
        File "<string>", line 14, in <module>
        File "/home/rarts/.cache/rattler/cache/uv-cache/.tmpvVNUWW/.venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 325, in
      get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/rarts/.cache/rattler/cache/uv-cache/.tmpvVNUWW/.venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 295, in
      _get_build_requires
          self.run_setup()
        File "/home/rarts/.cache/rattler/cache/uv-cache/.tmpvVNUWW/.venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 487, in
      run_setup
          super().run_setup(setup_script=setup_script)
        File "/home/rarts/.cache/rattler/cache/uv-cache/.tmpvVNUWW/.venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 311, in
      run_setup
          exec(code, locals())
        File "<string>", line 8, in <module>
      ModuleNotFoundError: No module named 'torch'
      ---

In both the uv and rip workflow although import torch in the pixi run python does work for this pixi.toml.

Thanks for the info @jacobbieker

@tdejager
Copy link
Contributor

tdejager commented Mar 4, 2024

Trying to install torch-geometric-temporal with pixi add. It fails as it cannot find torch, which had already been install previously

It does indeed not work. Testing the repro gives me this error:

  × failed to solve the pypi requirements of 'default' 'linux-64'
  ├─▶ failed to resolve pypi dependencies
  ├─▶ Failed to download and build: torch-scatter==2.1.2
  ├─▶ Failed to build: torch-scatter==2.1.2
  ╰─▶ Build backend failed to determine extra requires with `build_wheel()`:
      --- stdout:
      
      --- stderr:
      Traceback (most recent call last):
        File "<string>", line 14, in <module>
        File "/home/rarts/.cache/rattler/cache/uv-cache/.tmpvVNUWW/.venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 325, in
      get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/rarts/.cache/rattler/cache/uv-cache/.tmpvVNUWW/.venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 295, in
      _get_build_requires
          self.run_setup()
        File "/home/rarts/.cache/rattler/cache/uv-cache/.tmpvVNUWW/.venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 487, in
      run_setup
          super().run_setup(setup_script=setup_script)
        File "/home/rarts/.cache/rattler/cache/uv-cache/.tmpvVNUWW/.venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 311, in
      run_setup
          exec(code, locals())
        File "<string>", line 8, in <module>
      ModuleNotFoundError: No module named 'torch'
      ---

In both the uv and rip workflow although import torch in the pixi run python does work for this pixi.toml.

Thanks for the info @jacobbieker

Would be kind-of useful if we can keep the build environments for uv as well, so it's easier to debug these things.

@liblaf
Copy link

liblaf commented Mar 12, 2024

What did you run and what was the outcome?

pixi add --pypi -vv "trimesh[all]"

What error did pixi return?

Log

 INFO pixi::lock_file::outdated: the pypi dependencies of environment 'default' for platform linux-64 are out of date because the requirement 'trimesh[all]' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::update: updated conda packages in the 'default' prefix in 5ms 378us 132ns
 INFO resolve_pypi{group=default platform=linux-64}: pixi::lock_file::resolve: the following python packages are assumed to be installed by conda: libexpat 2.6.1, xz 5.2.6, readline 8.2, openssl 3.2.1, ld-impl-linux-64 2.40, libsqlite 3.45.1, libuuid 2.38.1, libffi 3.4.2, ncurses 6.4, bzip2 1.0.8, libzlib 1.2.13, tk 8.6.13, tzdata 2024a0, libxcrypt 4.4.36, libnsl 2.0.1, libgcc-ng 13.2.0, python 3.11.8, ca-certificates 2024.2.2, libgomp 13.2.0
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::partial_solution: add_decision: root @ 0a0.dev0    
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::partial_solution: add_decision: trimesh[all] @ 4.2.0    
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::partial_solution: add_decision: trimesh @ 4.2.0    
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::partial_solution: add_decision: trimesh[test] @ 4.2.0    
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::partial_solution: add_decision: trimesh[easy] @ 4.2.0    
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::partial_solution: add_decision: trimesh[recommend] @ 4.2.0    
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::partial_solution: add_decision: numpy @ 1.26.4    
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::partial_solution: add_decision: pytest-cov @ 4.1.0    
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::partial_solution: add_decision: coveralls @ 3.3.1    
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::partial_solution: add_decision: mypy @ 1.9.0    
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::partial_solution: add_decision: ezdxf @ 1.2.0    
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::partial_solution: add_decision: pytest @ 8.1.1    
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::partial_solution: add_decision: pymeshlab @ 2022.2.post3    
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::partial_solution: add_decision: pyinstrument @ 4.6.2    
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::partial_solution: add_decision: matplotlib @ 3.8.3    
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::partial_solution: add_decision: ruff @ 0.3.2    
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::partial_solution: add_decision: typeguard @ 4.1.5    
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::partial_solution: add_decision: colorlog @ 6.8.2    
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::partial_solution: add_decision: mapbox-earcut @ 1.0.1    
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::partial_solution: add_decision: chardet @ 5.2.0    
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::partial_solution: add_decision: lxml @ 5.1.0    
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::partial_solution: add_decision: jsonschema @ 4.21.1    
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::partial_solution: add_decision: networkx @ 3.2.1    
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::partial_solution: add_decision: svg-path @ 6.3    
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::partial_solution: add_decision: pycollada @ 0.8    
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::partial_solution: add_decision: setuptools @ 69.1.1    
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::partial_solution: add_decision: shapely @ 2.0.3    
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::partial_solution: add_decision: xxhash @ 3.4.1    
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::partial_solution: add_decision: rtree @ 1.2.0    
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::partial_solution: add_decision: httpx @ 0.27.0    
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::partial_solution: add_decision: scipy @ 1.12.0    
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::core: Start conflict resolution because incompat satisfied:
   embreex <2.17.7 | >2.17.7, <2.17.7.post1 | >2.17.7.post1, <2.17.7.post2 | >2.17.7.post2, <2.17.7.post3 | >2.17.7.post3, <2.17.7.post4 | >2.17.7.post4 is forbidden    
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::core: prior cause: embreex <2.17.7.post1 | >2.17.7.post1, <2.17.7.post2 | >2.17.7.post2, <2.17.7.post3 | >2.17.7.post3, <2.17.7.post4 | >2.17.7.post4 is forbidden    
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::core: prior cause: embreex <2.17.7.post2 | >2.17.7.post2, <2.17.7.post3 | >2.17.7.post3, <2.17.7.post4 | >2.17.7.post4 is forbidden    
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::core: prior cause: embreex <2.17.7.post3 | >2.17.7.post3, <2.17.7.post4 | >2.17.7.post4 is forbidden    
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::core: prior cause: embreex <2.17.7.post4 | >2.17.7.post4 is forbidden    
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::core: backtrack to DecisionLevel(5)    
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::core: Start conflict resolution because incompat satisfied:
   embreex ==2.17.7.post4 is forbidden    
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::core: prior cause: embreex * is forbidden    
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::core: prior cause: trimesh[easy] ==4.2.0 is forbidden    
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::core: backtrack to DecisionLevel(2)    
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::partial_solution: not adding trimesh[easy] @ 4.1.8 because of its dependencies    
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::partial_solution: not adding trimesh[easy] @ 4.1.7 because of its dependencies    
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::partial_solution: not adding trimesh[easy] @ 4.1.6 because of its dependencies    
...
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::partial_solution: not adding trimesh[easy] @ 1.9.15 because of its dependencies    
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::partial_solution: not adding trimesh[easy] @ 1.9.14 because of its dependencies    
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::partial_solution: not adding trimesh[easy] @ 1.9.13 because of its dependencies    
 INFO resolve_pypi{group=default platform=linux-64}:process_request{request=Metadata trimesh==1.9.12}:get_or_build_wheel_metadata{dist=trimesh==1.9.12}:build_source_dist_metadata:setup_build{package_id="trimesh==1.9.12" subdirectory=None}:solve: pubgrub::internal::partial_solution: add_decision: root @ 0a0.dev0    
 INFO resolve_pypi{group=default platform=linux-64}:process_request{request=Metadata trimesh==1.9.12}:get_or_build_wheel_metadata{dist=trimesh==1.9.12}:build_source_dist_metadata:setup_build{package_id="trimesh==1.9.12" subdirectory=None}:solve: pubgrub::internal::partial_solution: add_decision: setuptools @ 69.1.1    
 INFO resolve_pypi{group=default platform=linux-64}:process_request{request=Metadata trimesh==1.9.12}:get_or_build_wheel_metadata{dist=trimesh==1.9.12}:build_source_dist_metadata:setup_build{package_id="trimesh==1.9.12" subdirectory=None}:solve: pubgrub::internal::partial_solution: add_decision: wheel @ 0.43.0    
  × failed to solve the pypi requirements of 'default' 'linux-64'
  ├─▶ failed to resolve pypi dependencies
  ├─▶ Failed to download and build: trimesh==1.9.12
  ├─▶ Failed to build: trimesh==1.9.12
  ╰─▶ Build backend failed to determine extra requires with `build_wheel()` with exit status: 1
      --- stdout:
      
      --- stderr:
      Traceback (most recent call last):
        File "<string>", line 9, in <module>
      ModuleNotFoundError: No module named 'pypandoc'
      
      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
        File "<string>", line 14, in <module>
        File "/home/liblaf/.cache/rattler/cache/uv-cache/.tmpc8Jzlx/.venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 325, in
      get_requires_for_build_wheel
          return self._get_build_requires(config_settings, requirements=['wheel'])
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/home/liblaf/.cache/rattler/cache/uv-cache/.tmpc8Jzlx/.venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 295, in _get_build_requires
          self.run_setup()
        File "/home/liblaf/.cache/rattler/cache/uv-cache/.tmpc8Jzlx/.venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 487, in run_setup
          super().run_setup(setup_script=setup_script)
        File "/home/liblaf/.cache/rattler/cache/uv-cache/.tmpc8Jzlx/.venv/lib/python3.11/site-packages/setuptools/build_meta.py", line 311, in run_setup
          exec(code, locals())
        File "<string>", line 12, in <module>
      FileNotFoundError: [Errno 2] No such file or directory: 'README.md'
      ---

Can pip install the package? Does pip install "trimesh[all]" work?

Yes.

What platform are you on?

linux-64

The above log shows that the problem seems to be with embreex, so I tried adding it directly, and got the following error:

$ pixi add --pypi -vv embreex
 INFO pixi::lock_file::outdated: the pypi dependencies of environment 'default' for platform linux-64 are out of date because the requirement 'embreex' could not be satisfied (required by '<environment>')
 INFO pixi::lock_file::update: updated conda packages in the 'default' prefix in 6ms 587us 9ns
 INFO resolve_pypi{group=default platform=linux-64}: pixi::lock_file::resolve: the following python packages are assumed to be installed by conda: libxcrypt 4.4.36, readline 8.2, libgomp 13.2.0, setuptools 69.1.1, xz 5.2.6, libnsl 2.0.1, tzdata 2024a0, libuuid 2.38.1, wheel 0.42.0, libgcc-ng 13.2.0, bzip2 1.0.8, ca-certificates 2024.2.2, libsqlite 3.45.1, ncurses 6.4, pip 24.0, openssl 3.2.1, ld-impl-linux-64 2.40, libzlib 1.2.13, libffi 3.4.2, libexpat 2.6.1, python 3.11.8, tk 8.6.13
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::partial_solution: add_decision: root @ 0a0.dev0
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::core: Start conflict resolution because incompat satisfied:
   embreex <2.17.7 | >2.17.7, <2.17.7.post1 | >2.17.7.post1, <2.17.7.post2 | >2.17.7.post2, <2.17.7.post3 | >2.17.7.post3, <2.17.7.post4 | >2.17.7.post4 is forbidden
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::core: prior cause: embreex <2.17.7.post1 | >2.17.7.post1, <2.17.7.post2 | >2.17.7.post2, <2.17.7.post3 | >2.17.7.post3, <2.17.7.post4 | >2.17.7.post4 is forbidden
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::core: prior cause: embreex <2.17.7.post2 | >2.17.7.post2, <2.17.7.post3 | >2.17.7.post3, <2.17.7.post4 | >2.17.7.post4 is forbidden
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::core: prior cause: embreex <2.17.7.post3 | >2.17.7.post3, <2.17.7.post4 | >2.17.7.post4 is forbidden
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::core: prior cause: embreex <2.17.7.post4 | >2.17.7.post4 is forbidden
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::core: prior cause: embreex * is forbidden
 INFO resolve_pypi{group=default platform=linux-64}:solve: pubgrub::internal::core: prior cause: root ==0a0.dev0 is forbidden
  × failed to solve the pypi requirements of 'default' 'linux-64'
  ├─▶ failed to resolve pypi dependencies
  ╰─▶ Because only the following versions of embreex are available:
          embreex==2.17.7
          embreex==2.17.7.post1
          embreex==2.17.7.post2
          embreex==2.17.7.post3
          embreex==2.17.7.post4
      and embreex==2.17.7 is unusable because no wheels are available with a matching platform, we can conclude that embreex<2.17.7.post1 cannot be used.
      And because embreex==2.17.7.post1 is unusable because no wheels are available with a matching platform, we can conclude that embreex<2.17.7.post2 cannot be used.
      And because embreex==2.17.7.post2 is unusable because no wheels are available with a matching platform and embreex==2.17.7.post3 is unusable because no wheels are
      available with a matching platform, we can conclude that embreex<2.17.7.post4 cannot be used.
      And because embreex==2.17.7.post4 is unusable because no wheels are available with a matching platform and you require embreex, we can conclude that the
      requirements are unsatisfiable.

@evetion
Copy link

evetion commented Mar 15, 2024

What did you run and what was the outcome?

pixi add --platform osx-arm64 --pypi "meshkernel==4.1"

What error did pixi return?

  × failed to resolve `pypi-dependencies`, due to underlying error
  ╰─▶ The following packages are incompatible
      └─ meshkernel ==4.1 cannot be installed because there are no viable options:
         └─ meshkernel 4.1.0 is excluded because none of the artifacts are compatible with the Python interpreter or glibc version and there are no supported sdists

Can pip install the package?

pixi run pip install "meshkernel==4.1" works on the osx-arm64 platform.

What platform are you on?

This happens on the osx-64 and osx-arm64 platforms (it works on win-64 and linux-64). I expect this to be some mismatching in the name/metadata of/in the .whl files for the macos specific ones.

Did you find a workaround, if so please explain.

Not yet, it would help if the resolver can be more verbose (-vvv has no effect on it) how it matches the platform/interpreter/glibc version and what's incompatible. Possibly renaming the wheels on the meshkernel side would be enough to fix it.

Encountered in Deltares/Ribasim#1137

@wolfv
Copy link
Member

wolfv commented Mar 15, 2024

@evetion that error often means that the system-requirements are not set high enough. I can see that wheels are only available for macos = 14.0 (for arm64). We should do a better job at explaining this.

Unfortunately, setting that in my pixi.toml didn't work just now, so we might also still have to do some more debugging of the uv integration! I'll take a look :)

@wolfv
Copy link
Member

wolfv commented Mar 15, 2024

Sorry, my bad. meshkernel actually does work fine when you add the following:

[system-requirements]
macos = "14.0"

We should make the error more actionable and figure out a better default behavior.

@wolfv
Copy link
Member

wolfv commented Mar 15, 2024

@liblaf I wonder if for you it's a similar issue and adding glibc 2.28 or higher would fix it.

e.g.

[system-requirements]
glibc = "2.28"

@liblaf
Copy link

liblaf commented Mar 15, 2024

@liblaf I wonder if for you it's a similar issue and adding glibc 2.28 or higher would fix it.

e.g.

[system-requirements]
glibc = "2.28"

@wolfv thx! pixi add --pypi "trimesh[all]" now works for me with the following config:

[system-requirements]
libc = "2.39"

@wolfv
Copy link
Member

wolfv commented Mar 15, 2024

Good to hear, thanks @liblaf! We should make this easier for users, and maybe use a more recent glibc / macos version as default.

@evetion
Copy link

evetion commented Mar 15, 2024

Sorry, my bad. meshkernel actually does work fine when you add the following:

[system-requirements]
macos = "14.0"

We should make the error more actionable and figure out a better default behavior.

Thanks for the quick fix! This does indeed work. I'm looking forward to further improvements to pixi.

@ruben-arts
Copy link
Contributor Author

The pypi support seems to be much better now. I'm closing this issue, feel free to open when you have a related issue.

@pablovela5620
Copy link
Contributor

The pypi support seems to be much better now. I'm closing this issue, feel free to open when you have a related issue.

@ruben-arts I'm still having some issues with a few packages. In particular pymeshlab when considering nerfstudio.

Here is a minimal example

[project]
name = "pymeshlab-test"
version = "0.1.0"
description = "Add a short description here"
authors = ["pablovela5620 <pablovela5620@gmail.com>"]
channels = ["conda-forge"]
platforms = ["linux-64"]

[tasks]

[dependencies]
python = "3.11.*"

[pypi-dependencies]
pymeshlab = "==2022.2.post4"

You can see below that the latest version is 2023.12.post1 but anything above ==2022.2.post4 fails
pymeshlab

along with this, when using a bit more complicated of a pyproject.toml like I have for the recent dn-splatter, even ==2022.2post2 fails (which is required for the nerfstudio install)

[project]
name = "dn-splatter"
description = "Depth and normal priors for 3D Gaussian splatting and meshing"
version = "0.0.1"

dependencies = [
    "pymeshlab==2022.2.post2"
                ]

[tool.setuptools.packages.find]
include = ["dn_splatter*"]

[project.entry-points.'nerfstudio.method_configs']
dn_splatter = 'dn_splatter.dn_config:dn_splatter'
#g-nerfacto = 'dn_splatter.eval.eval_configs:gnerfacto'
#g-depthfacto = 'dn_splatter.eval.eval_configs:gdepthfacto'
#g-neusfacto = 'dn_splatter.eval.eval_configs:gneusfacto'

[project.entry-points.'nerfstudio.dataparser_configs']
mushroom = 'dn_splatter:MushroomDataParserSpecification'
replica = 'dn_splatter:ReplicaDataParserSpecification'
nrgbd = 'dn_splatter:NRGBDDataParserSpecification'
gsdf = 'dn_splatter:GSDFStudioDataParserSpecification'
scannetpp = 'dn_splatter:ScanNetppDataParserSpecification'
coolermap = 'dn_splatter:CoolerMapDataParserSpecification'
normal-nerfstudio = 'dn_splatter:NormalNerfstudioSpecification'

[project.scripts]
# export mesh scripts
gs-mesh = "dn_splatter.export_mesh:entrypoint"

[tool.pixi.project]
name = "dn-splatter"
version = "0.1.0"
description = "Depth and normal priors for 3D Gaussian splatting and meshing"
channels = ["nvidia/label/cuda-11.8.0", "nvidia", "conda-forge", "pytorch"]
platforms = ["linux-64"]

[tool.pixi.dependencies]
python = "3.10.*"
cuda = {version = "*", channel="nvidia/label/cuda-11.8.0"}
pytorch-cuda = {version = "11.8.*", channel="pytorch"}
pytorch = {version = ">=2.2.0,<2.3", channel="pytorch"}
torchvision = {version = ">=0.17.0,<0.18", channel="pytorch"}

[tool.pixi.pypi-dependencies]
dn-splatter = { path = ".", editable = true}

error

(default) pablo@pablo-ubuntu:~/0Dev/personal/forked-repos/dn-splatter$ pixi install
  × failed to solve the pypi requirements of 'default' 'linux-64'
  ├─▶ failed to resolve pypi dependencies
  ╰─▶ Because pymeshlab==2022.2.post2 is unusable because no wheels are available with a matching Python ABI and you require pymeshlab==2022.2.post2, we can
      conclude that the requirements are unsatisfiable.

@ruben-arts ruben-arts reopened this Apr 10, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
pypi Issue related to PyPI dependencies
Projects
None yet
Development

No branches or pull requests