Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Installation failing due to shapely=2.0.0 (linux64) issue #2965

Closed
bouweandela opened this issue Dec 13, 2022 · 17 comments
Closed

Installation failing due to shapely=2.0.0 (linux64) issue #2965

bouweandela opened this issue Dec 13, 2022 · 17 comments
Assignees
@valeriupredoi
Labels
bug installation

Comments

@bouweandela
Copy link
Member

bouweandela commented Dec 13, 2022 •
edited by valeriupredoi
Loading

The installation tests are failing because of an incompatibility with shapely. See here for an example. Stack trace:

ERROR:esmvalcore._main:Program terminated abnormally, see stack trace below for more information:
Traceback (most recent call last):
  File "/opt/conda/envs/esmvaltool/lib/python3.10/site-packages/pkg_resources/__init__.py", line 2760, in requires
    deps.extend(dm[safe_extra(ext)])
KeyError: 'vectorized'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/opt/conda/envs/esmvaltool/lib/python3.10/site-packages/esmvalcore/_main.py", line 499, in run
    fire.Fire(ESMValTool())
  File "/opt/conda/envs/esmvaltool/lib/python3.10/site-packages/esmvalcore/_main.py", line 309, in __init__
    self.__setattr__(entry_point.name, entry_point.load()())
  File "/opt/conda/envs/esmvaltool/lib/python3.10/site-packages/pkg_resources/__init__.py", line 2470, in load
    self.require(*args, **kwargs)
  File "/opt/conda/envs/esmvaltool/lib/python3.10/site-packages/pkg_resources/__init__.py", line 2493, in require
    items = working_set.resolve(reqs, env, installer, extras=self.extras)
  File "/opt/conda/envs/esmvaltool/lib/python3.10/site-packages/pkg_resources/__init__.py", line 803, in resolve
    new_requirements = dist.requires(req.extras)[::-1]
  File "/opt/conda/envs/esmvaltool/lib/python3.10/site-packages/pkg_resources/__init__.py", line 2762, in requires
    raise UnknownExtra(
pkg_resources.UnknownExtra: shapely 2.0.0 has no such extra feature 'vectorized'

Also an example from our Github Actions test suite where it's a bit easier to see what deps get in the env etc
@bouweandela bouweandela mentioned this issue Dec 13, 2022
Merged
5 tasks
@valeriupredoi valeriupredoi mentioned this issue Dec 14, 2022
Closed
@valeriupredoi valeriupredoi self-assigned this Dec 14, 2022
@valeriupredoi
Copy link
Contributor

looking into this now 👍

@zklaus
Copy link

zklaus commented Dec 14, 2022

Note that this is during the Julia install, i.e. I think it is the Conda environment that Julia creates inside the Julia environment inside our Conda environment. Ceterum censeo...

@valeriupredoi
Copy link
Contributor

Per aspera ad arseum - it is indeed Julia, and its slowness adapting to new deps - the new shapely=2.0 was released less than 2 days ago, right when we started seeing this problem

@valeriupredoi
Copy link
Contributor

what do you think @zklaus - pin shapely<2.0? 1.8.5 works well with the current julia we have (1.8.3). Maybe raise this with Julia conda feedstock admins too?

@zklaus
Copy link

zklaus commented Dec 14, 2022

Pin how? This is somehow getting pulled in by one of the packages in esmvaltool/install/Julia/Project.toml, right? Can you tell Julia to limit dependencies?

@valeriupredoi
Copy link
Contributor

yes, add shapely=1.8.5 (or whatevs the hash etc) in the Project toml - or just ping the good ole Julia package maintainers @ngam and @mkitti and see what they say?

@mkitti
Copy link

mkitti commented Dec 14, 2022

What do we need from the Julia feedstock? A rebuild?

@bouweandela
Copy link
Member Author

bouweandela commented Dec 14, 2022 •
edited
Loading

I think it's also possible to tell Julia to use the existing conda environment instead of creating its own. Wouldn't that be easier?

@mkitti
Copy link

mkitti commented Dec 14, 2022

Note that this is during the Julia install, i.e. I think it is the Conda environment that Julia creates inside the Julia environment inside our Conda environment. Ceterum censeo...

If the Julia feedstock activate script is working properly, this should not be happening. Via environment variables, Julia should be used to use the existing conda environment.

https://github.com/conda-forge/julia-feedstock/blob/main/recipe/scripts/activate.sh

@valeriupredoi
Copy link
Contributor

valeriupredoi commented Dec 14, 2022 •
edited
Loading

very many thanks @mkitti - that pointer to get Julia to use the current conda env is defo a great idea! I do, however, I feel a bit embarrassed for pinging you good folk, before I recreate the issue locally, and I just did that and found this fail has nothing to do with Julia, see below @bouweandela @zklaus

valeriu@valeriu-PORTEGE-Z30-C:~/ESMValTool$ esmvaltool install Jumia
ERROR:esmvalcore._main:Program terminated abnormally, see stack trace below for more information:
Traceback (most recent call last):
  File "/home/valeriu/miniconda3-fresh/envs/esmvaltool-latest/lib/python3.10/site-packages/pkg_resources/__init__.py", line 2760, in requires
    deps.extend(dm[safe_extra(ext)])
KeyError: 'vectorized'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/valeriu/miniconda3-fresh/envs/esmvaltool-latest/lib/python3.10/site-packages/esmvalcore/_main.py", line 499, in run
    fire.Fire(ESMValTool())
  File "/home/valeriu/miniconda3-fresh/envs/esmvaltool-latest/lib/python3.10/site-packages/esmvalcore/_main.py", line 309, in __init__
    self.__setattr__(entry_point.name, entry_point.load()())
  File "/home/valeriu/miniconda3-fresh/envs/esmvaltool-latest/lib/python3.10/site-packages/pkg_resources/__init__.py", line 2470, in load
    self.require(*args, **kwargs)
  File "/home/valeriu/miniconda3-fresh/envs/esmvaltool-latest/lib/python3.10/site-packages/pkg_resources/__init__.py", line 2493, in require
    items = working_set.resolve(reqs, env, installer, extras=self.extras)
  File "/home/valeriu/miniconda3-fresh/envs/esmvaltool-latest/lib/python3.10/site-packages/pkg_resources/__init__.py", line 803, in resolve
    new_requirements = dist.requires(req.extras)[::-1]
  File "/home/valeriu/miniconda3-fresh/envs/esmvaltool-latest/lib/python3.10/site-packages/pkg_resources/__init__.py", line 2762, in requires
    raise UnknownExtra(
pkg_resources.UnknownExtra: shapely 2.0.0 has no such extra feature 'vectorized'

This needs to be tackled internally in esmvalcore or we have to raise it with Shapely if it's their problem. Sorry @mkitti and @ngam - cheers for your help and happy holidays 🎄

@valeriupredoi
Copy link
Contributor

in fact, that error pops up everytime an instance of esmvaltool is called, even for esmvaltool --help

@valeriupredoi
Copy link
Contributor

shapely=2.0.0 is poorly deployed on conda forge as it seems - at env creation point it is installed from conda-forge, at ESMValTool install point:

Requirement already satisfied: shapely in /home/valeriu/miniconda3-fresh/envs/esmvaltool-latest/lib/python3.10/site-packages (from ESMValTool==2.6.1.dev84+g0d599128a.d20221214) (2.0.0)

but looking at what I have in my env:

valeriu@valeriu-PORTEGE-Z30-C:~/ESMValTool$ conda list shapely
# packages in environment at /home/valeriu/miniconda3-fresh/envs/esmvaltool-latest:
#
# Name                    Version                   Build  Channel
shapely                   2.0.0                    pypi_0    pypi

@valeriupredoi valeriupredoi changed the title Installation failing due to shapely issue Installation failing due to shapely=2.0.0 (linux64) issue Dec 14, 2022
@valeriupredoi valeriupredoi mentioned this issue Dec 14, 2022
Merged
4 tasks
@valeriupredoi
Copy link
Contributor

Opened this conda-forge/shapely-feedstock#121

@valeriupredoi
Copy link
Contributor

and closed it too, after having actually seen the shapely conda forge pkg tests that are correct and pass for linux64, and, more importantly all is fine with our esmvalcore entry points in esmvalcore/main latest dev branch. Am rather puzzleated by this behaviour TBH

@valeriupredoi
Copy link
Contributor

OK some systematic progress wrt this:

  • esmvalcore has absolutely no issues with the new shapely
  • latest esmvaltool envs are identical when we pin shapely or not (<2 as we have now), obviously shapely differs in version but no other dep differs
  • installing the tool with pip install ... --no-deps when shapely=2.0 does not avert the pkg issue; meaning no pip dep is causing this (well, just vprof is from pip)

All these things imply that one or more deps of only esmvaltool (and not of esmvalcore) that depend on shapely is/are causing this (or, in the worst case scenario one or more deps of those FMW); the list od deps that use shapely as dep from esmvaltool is the following, cross-correlated to esmvalcore deps: "Core" means it's also an esmvalcore dep, "NOT Core" means it's not:

  - appdirs                  1.4.4  pyh9f0ad1d_0     conda-forge Core
  - cartopy                 0.21.1  py310hcb7e713_0  conda-forge Core    
  - cattrs                  22.2.0  pyhd8ed1ab_0     conda-forge Core    
  - chart-studio             1.1.0  pyh9f0ad1d_0     conda-forge NOT Core
  - config                   0.5.1  pyhd8ed1ab_0     conda-forge NOT Core
  - configargparse           1.5.3  pyhd8ed1ab_0     conda-forge NOT Core
  - defusedxml               0.7.1  pyhd8ed1ab_0     conda-forge Core    
  - esgf-pyclient            0.3.1  pyh1a96a4e_2     conda-forge Core    
  - esmvalcore               2.7.1  pyhd8ed1ab_0     conda-forge -       
  - fiona                   1.8.22  py310ha325b7b_5  conda-forge Core    
  - geographiclib             1.52  pyhd8ed1ab_0     conda-forge Core    
  - geopy                    2.3.0  pyhd8ed1ab_0     conda-forge Core    
  - humanfriendly             10.0  py310hff52083_4  conda-forge Core    
  - importlib_resources     5.10.1  pyhd8ed1ab_0     conda-forge Core    
  - iris                     3.4.0  pyhd8ed1ab_0     conda-forge Core    
  - iris-esmf-regrid         0.5.0  pyhd8ed1ab_0     conda-forge Core    
  - itsdangerous             2.1.2  pyhd8ed1ab_0     conda-forge Core    
  - latexcodec               2.0.1  pyh9f0ad1d_0     conda-forge Core    
  - mapgenerator             1.0.5  pyhd8ed1ab_1     conda-forge NOT Core
  - munch                    2.5.0  py_0             conda-forge Core    
  - myproxyclient            2.1.0  pyhd8ed1ab_2     conda-forge Core    
  - nested-lookup           0.2.25  pyhd8ed1ab_1     conda-forge Core    
  - plotly                  5.11.0  pyhd8ed1ab_1     conda-forge NOT Core
  - psy-maps                 1.4.2  pyhd8ed1ab_0     conda-forge NOT Core
  - pybtex                  0.24.0  pyhd8ed1ab_2     conda-forge Core    
  - pyshp                    2.3.1  pyhd8ed1ab_0     conda-forge Core    
  - python-stratify      0.2.post0  py310hde88566_3  conda-forge Core    
  - python-xxhash            3.0.0  py310h5764c6d_2  conda-forge Core    
  - requests-cache           0.9.7  pyhd8ed1ab_0     conda-forge Core    
  - retrying                 1.3.3  py_2             conda-forge NOT Core
  - shapely                  2.0.0  py310h8b84c32_0  conda-forge Core    
  - tenacity                 8.1.0  pyhd8ed1ab_0     conda-forge NOT Core
  - ujson                    5.5.0  py310hd8f1fbe_1  conda-forge Core    
  - url-normalize            1.4.3  pyhd8ed1ab_0     conda-forge Core    
  - webob                    1.8.7  pyhd8ed1ab_0     conda-forge Core    
  - xxhash                   0.8.0  h7f98852_3       conda-forge Core    
  - yamale                   4.0.4  pyh6c4a22f_0     conda-forge Core

Am gonna have a closer look at "NOT Core" ones, fortunately they're not many...

@valeriupredoi
Copy link
Contributor

GAH! All this time and the answer was right under my nose and due to informational bias I didn't even think of checking - it's our esmvalcore=2.7.1 package, specifically setup.py off the v2.7.x branch - it needs shapely[vectorized] as dependency 🤦‍♂️ 😠 To be able to use the new shapely we need to either release a 2.7.2 or wait for 2.8.0 - am all about 2.7.2 - gonna open a new issue and discuss there, will close this

@valeriupredoi
Copy link
Contributor

ESMValGroup/ESMValCore#1874

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@valeriupredoi valeriupredoi

Labels
bug installation
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@zklaus @bouweandela @mkitti @valeriupredoi