feat: Conflicting requirements after copier installation, is it possible to bump up them?

[garu57]

Copier v6 installed from git repo master.

After installation i get the following:

# pip check                                                                   
twine 3.7.1 has requirement pkginfo>=1.8.1, but you have pkginfo 1.7.1.       
redis 4.1.0 has requirement packaging>=21.3, but you have packaging 20.9.     
nltk 3.6.7 has requirement regex>=2021.8.3, but you have regex 2020.11.13.    
flask 2.0.2 has requirement Jinja2>=3.0, but you have jinja2 2.11.3.          
black 21.12b0 has requirement pathspec<1,>=0.9.0, but you have pathspec 0.8.1.

Could you please bump up requirements?
Thanks

[pawamoy]

Did you install Copier in an isolated virtualenv? Are you using it as a library?

[garu57]

I'm using Winpython, a scientific distribution that has about 550 packages in the base installation, accurately selected so that they don't have conflicting requirements.
I installed using pip install --user copier. I don't use pipx, I have my scripts that set paths appropriately .
Anyway I normally use project virtual envs with include-system-site-packages = true because i don' want to reinstall packages already in the base installation. When you are going to require certain scientific packages, in an isolated venv, you often need to install tenths of packages and reconcile requirements easily become a real nightmare.
I want to say that this is not a blocking problem, i have the requirement file that allow me to scratch user site and recreate it clean, lets say that's more a wish than a need.
I think that being this a specialized tool I'll try to isolate it in its own venv.

[pawamoy]

Thank you for clarifying 🙂

I installed using pip install --user copier

If you install lots of things with pip install --user I guess it's expected that you end up with conflicts.

When you are going to require certain scientific packages, in an isolated venv, you often need to install tenths of packages and reconcile requirements easily become a real nightmare.

I don't follow here. The resolution is done automatically for you if you install several packages at once with pip. Or it is done by specialized, dependencies management tools like Poetry, PDM, pip-compile, etc.

a scientific distribution that has about 550 packages in the base installation

because i don' want to reinstall packages already in the base installation

I never understood this either, apart maybe that it takes less space on disk if you have lots of projects using the same libraries. But tools like PDM also use a cache, while still allowing isolation of each project. To me, installing several GB of libraries in a single env, even if accurately selected, feels overkill and wasteful 😅 But I'm probably missing something here.

I think that being this a specialized tool I'll try to isolate it in its own venv.

Can't recommend pipx enough 👍

[pawamoy]

About the issue itself: it's true that Copier sets upper bounds on dependencies. I agree that it should relax these constraints, most of them are probably not needed. @yajo what do you think? Would you agree on changing ^x.x.x to >=x.x.x on Copier's dependencies?

[garu57]

I don't follow here. The resolution is done automatically for you if you install several packages at once with pip. Or it is done by specialized, dependencies management tools like Poetry, PDM, pip-compile, etc.

I tried all the possible dependency management tools, Poetry, pip-tools, pipenv and name one, but all of them do not handle correctly venvs with include-system-site-packages = true and make a mess in the base installation. As for PDM, i don't want to invest on a moving target like Pep 582, at least not until it is more stable and adopted by others.

I never understood this either, apart maybe that it takes less space on disk if you have lots of projects using the same libraries. But tools like PDM also use a cache, while still allowing isolation of each project. To me, installing several GB of libraries in a single env, even if accurately selected, feels overkill and wasteful 😅 But I'm probably missing something here.

I guess you don't know what kind of mess lies behind scientific environments. 😲
I don't want to waste your time with technical explanations, but just think about this, if things were that simple as a pip install why a company like Entought would have lost so much time and resources to build and maintain the Anaconda scientific infrastructure with all their self built packages, and why Winpython authors would lost months, self compiling packages to ensure that all the needed copyrighted extensions, that you won't find on Pypy, are installed, to craft a new release? 😕

[pawamoy]

I tried all the possible dependency management tools, Poetry, pip-tools, pipenv and name one, but all of them do not handle correctly venvs with include-system-site-packages = true and make a mess in the base installation.

Ah, I never considered this indeed. It's true these tools have configuration options to enable system site-packages, but maybe they don't work too well with it. I wouldn't know because I never use these options.

I guess you don't know what kind of mess lies behind scientific environments.

I surely don't 😄!
Note that I don't doubt Conda and similar are useful, it's just the "base installation" thing that I have trouble understanding. I probably just wrongly assumed that dependencies could be solved easily in any context with these tools, and therefore base sets of compatible packages were not that useful. Anyway, I don't want to waste your time either, I should learn more about all this myself rather than poking at people on GitHub 😅 My apologies if I offended you 🙂

[garu57]

Surely you didn't offend me 😊, on the contrary i found you very kind in your attempts to give indications according to your experience. 😉

[yajo]

Would you agree on changing ^x.x.x to >=x.x.x on Copier's dependencies?

I think this makes sense.

We could also provide Copier as a nix flake, making it 100% reproducible under any environment.

Finally, I think the original problem is not being properly understood. For example, in your error log you have:

flask 2.0.2 has requirement Jinja2>=3.0, but you have jinja2 2.11.3.

However, we also require a newer Jinja:

copier/pyproject.toml

Line 35 in af0dbac

Jinja2 = "^3.0.2"

So the problem does not seem to be in Copier's dependencies, but in the ones installed in your venv. You'll need to fix those downstream.

I'll close this issue, where there seems to be no action to take in our side, and open the other two.

Thanks everyone, feel free to keep commenting here if needed. 😊