Update optimizer for lib versions and backuper for optimal and last working requirements

[ManPython]

What's the problem this feature will solve?

I'm looking some option to update jupyter or other to this version where other installed libs will be compatible or updated to optmial versions, eg. force re-instal to older main app and related libs. Soo I need some "indication" to right version between all version.
Soo.. when update.. then probably pip should alert and inform possible conflicts and not install new version if older are incompatible in some iteration of update and ask that I'm sure that something can't work or list optimal version to update to minimize impact for some old libs, eg. list of suggested version to update where 75% libs are compatible as tools, etc. or 50%, 25% and of course.. 95% is most of desirable.
Right now I stuck.. fighting with pip to find some optimal version.. but how..

Describe the solution you'd like

From my perspective should exist some tool that will be adequate do build info for:
pip install -r requirements.txt --upgrade
Eg, then tool can build many requirements.txt regarding compatibility range to select, eg.:

1_requirements_25_p.txt 
2_requirements_50_p.txt 
3_requirements_75_p.txt 
4_requirements_80_p.txt 
5_requirements_95_p.txt 

Then user can select 5 as prefix in file 5_requirements_95_p.txt and ok.. not working? Ok.. taking other version of requirements.
I know is big issue and most of out of scope from jupyter, but in fact jupyter is most of advanced env and application where user experiencing problems and doing bad efforts to find solution to repair env and app in case like this ticket.
Soo 5_requirements_95_p.txt must have a conditions for version as < > =< >= that are optimal for update.

Alternative Solutions

Should be compatible with some GUI manager. Eg. dir in pip like requirements/ where can be requirements for some updating app eg. requirements/jupyter/requirements_versions.txt then inside other

1_requirements_25_p.txt 
2_requirements_50_p.txt 
3_requirements_75_p.txt 
4_requirements_80_p.txt 
5_requirements_95_p.txt 

Additional context

jupyter/notebook#7177 (comment)

Code of Conduct

• I agree to follow the PSF Code of Conduct.

Tasks

Beta Give feedback

1. Check libs conditions before update
2. Build list of possible requirements that update touching requirements_list.txt in requirements/lib_name_to_update/ requirements_list.txt
3. Find conditions for requirements_list.txt to build optimal lib versions to update or select for user, eg. many: 1_requirements_25_p.txt
4. Give option to select optimal network of libs to update
5. Create requirements_backup_date.txt to force re-install all previous libs
6. Give option to summary after update about updated libs as some table

Not sure about: https://github.com/jazzband/pip-tools/

[notatallshaw]

Are you referring to external dependencies or python package dependencies?

For external ones this is currently outside the scope of what Python package installers and managers can do, but conda can handle this to some extent for you, so you might want to use that.

For Python package dependencies Pip only resolves install packages, not ones in the environment. There is a pretty easy work around that looks something like this:

pip freeze --quiet --exclude-editable > upgrade_current_environment.txt
sed -i 's/==/>=/g' upgrade_current_environment.txt # Change from pinning to lower bound if you want latest
pip install <my_requirements> -c upgrade_current_environment.txt

Pip-tools also offers a workflow that manages your environment via a constraint based lock file.

Or you could look at Python package environment managers, such as PDM, Poetry, or Pixi.

[ManPython]

Yes, external.

It must be solved and I think should be implemented around PIP..
Conda is not for shell solution I think, and this all is classic dependency hell problem.
What if can look.. it's work for this problem?

[notatallshaw]

Conda is not for shell solution I think, and this all is classic dependency hell problem.

I don't know what either of those two things mean, conda works fine in a shell, and the very latest version of conda has amazing dependency resolution speed.

Maybe you are confusing Anaconda with conda? I would look at using miniforge or miniconda (although there are licensing issues with miniconda I do not like).

What if can look.. it's work for this problem?

There has been some discussion on external dependencies, see https://discuss.python.org/t/pep-725-specifying-external-dependencies-in-pyproject-toml and https://peps.python.org/pep-0725/

But even if implemented this is a long way off a Python package installer or manager understanding, or even resolving on, external dependencies.

[ManPython]

Hhuh.. but the main question.. Is conda or miniconda have option to select right optimal external libs?
Eg. I have installed many addons for jupyter and with this all stuff must be in some mid point of version as optimal.
By this... I should go with this feature request more to minicodna or pip-tools, PDM, Poetry, or Pixi.. etc. as know.
And why not PIP? The PIP is most of like base for update also not only install, why not implement optimizer around?

[pfmoore]

External libraries (i.e., libraries not packaged as Python wheels) are out of scope for pip, and will remain so. PEP 725 might provide a mechanism whereby pip could report that needed libraries were not installed in the system, but frankly I doubt it (as that PEP doesn't provide a cross-platform way of asking "is external library XXX installed?")

So the answer to your question (at least to the extent I understand what you're asking for) is sorry, but no - pip does not support this, and we're not looking to add such support.

[ManPython]

Soo.. the basic value for users, programmers is out of scope..

[pfmoore]

Pip is a python package manager, not a general package manager.

(Note: I'm only responding because I'm concerned we might be misunderstanding each other as far as what we mean by "external libraries" - your original example was Jupyter, which as far as I understand doesn't need any external libraries, at least on Windows, so it's possible you mean something different than I think you do).

[ManPython]

your original example was Jupyter, which as far as I understand doesn't need any external libraries, at least on Windows, so it's possible you mean something different than I think you do).
That's not ture..

Here is elements that Jupyter needs in general as a complex in some basic area:

- jupyter --version
Selected Jupyter core packages...
IPython          : 8.8.0
ipykernel        : 6.20.1
ipywidgets       : 8.0.4
jupyter_client   : 8.6.0
jupyter_core     : 5.1.3
jupyter_server   : 2.12.0
jupyterlab       : 4.0.9
nbclient         : 0.7.2
nbconvert        : 7.2.7
nbformat         : 5.7.1
notebook         : 7.0.6
qtconsole        : 5.4.0
traitlets        : 5.8.1

Touch any of this due other libs can broker whole fundamental ground to work.. what I meet many times dupe install other and update related as required and due lack of knowledge what is stable around all, eg IPython and related.
I'm confused around this policy.. coz similar problems was around .net but more problematic due closed libs.. now have open, but whole environment around python going to be so problematic.. as is easy to resign to working with Pyhon.. let we leave this for AI/GPT etc for future.. (of course is bad case and bad strategy).

https://acrosby.github.io/python/dependency-hell.html

[pfmoore]

Here is elements that Jupyter needs in general as a complex in some basic area:

Those are Python requirements.

Touch any of this due other libs can broker whole fundamental ground to work

So I think what you're actually trying to say is that pip allows you to (for example) upgrade traitlets, and doesn't prevent you doing so if the upgraded traitlets would be inconsistent with the already-installed IPython package?

If that's what you mean, then yes, that's (unfortunately) a deliberate limitation of pip. Many people have environments with thousands of installed packages, and checking everything while we're doing dependency resolution is very costly. So we limit the checks we do to packages that a given pip run knows about (i.e., ones the user has specified on the command line, and their dependencies).

The pip check command lets you run a full-environment consistency check, and we do run a pip check after each install. This is a compromise between safety and usability.

I will note that if I run

pip install "ipython<7"
pip install -U prompt-toolkit

I get the following output:

Requirement already satisfied: prompt-toolkit in c:\users\gustav\.virtualenvs\3342921fc60fa2e\lib\site-packages (1.0.18)
Collecting prompt-toolkit
  Using cached prompt_toolkit-3.0.41-py3-none-any.whl.metadata (6.5 kB)
Requirement already satisfied: wcwidth in c:\users\gustav\.virtualenvs\3342921fc60fa2e\lib\site-packages (from prompt-toolkit) (0.2.12)
Using cached prompt_toolkit-3.0.41-py3-none-any.whl (385 kB)
Installing collected packages: prompt-toolkit
  Attempting uninstall: prompt-toolkit
    Found existing installation: prompt-toolkit 1.0.18
    Uninstalling prompt-toolkit-1.0.18:
      Successfully uninstalled prompt-toolkit-1.0.18
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
ipython 6.5.0 requires prompt-toolkit<2.0.0,>=1.0.15, but you have prompt-toolkit 3.0.41 which is incompatible.
Successfully installed prompt-toolkit-3.0.41

Note the message near the end (which is in red on the console) - this explicitly says that you now have a broken environment. The pip install command gives a "success" return code (which is a debatable choice) but if you want to automate the checking, pip check will give a non-zero exit code if your environment has conflicts.

I hope this helps - it may not be quite as automatic as you'd like, but it is possible to ensure your environments are always in a consistent state using pip check as needed.

[ManPython]

So I think what you're actually trying to say is that pip allows you to (for example) upgrade traitlets, and doesn't prevent you doing so if the upgraded traitlets would be inconsistent with the already-installed IPython package?

In short.. but problem is more advanced.. take again the ticket.. Its about
ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behavior is the source of the following dependency conflicts.
And in the logs (.txt) file is how I must operate individually each lib to back, then update other to "balance" optimal point for all and resign something, but manually.. it's horrible work.

If that's what you mean, then yes, that's (unfortunately) a deliberate limitation of pip. Many people have environments with thousands of installed packages, and checking everything while we're doing dependency resolution is very costly.

Which cost? CPU or coder time? : )
And the feature request is for resolve problem around optimizer when updated or install.. in relation to pip's dependency resolver taking point on area as described in proposal.

I believe in million libs.. but it's not the case for manually checking.. but some tools.. - soo PIP or other.. or wee still can be stuck..

And.. sure.. it's easy to do broken environment, coz..
ipython 6.5.0 requires prompt-toolkit<2.0.0,>=1.0.15, but you have prompt-toolkit 3.0.41 which is incompatible.
And now other need newest prompt-toolkit as 3.0.41maybe, but can maybe have 2.0.0...

The problem is as I described that PIP not checking whole dependency.. and not optimizing.. around some reference libs or app like Jupyter. In fact we can check whole libs outside jupyter to be up to date with all, but must be checked.

We can do more.. eg. after some issue around dependency hell issue user trying uninstall lib, the PIP can ask about problems around to report some relations problems (eg. giving stars for versions).
After new runing PIP this PIP can ask after time about current stage of work last installed libs to reach stars, as working or not.. then can be downgraded suggested last libs to install, eg. there is new version jupyter 30.0 but stable related version is jupyter 25.0.1 version with 1000 libs in case jupyter 30.0 just 500 libs.

Note that is not only traitlets but alos tornado, jupyter-client jupyter-serve jupyterlab-code-formatter then voila-gridstack, voila in general... then cython problems.. etc.. etc..
And python space.. not have knowledge about optimal indicator of version. This is very poor for python (community) and basic about resolve issue around dependency-hell.

Reported pip check not taking all related libs..
To operate with this.. must do now some report about min max version for each.. and find optimal.. maybe excel.. rofl..
In other case.. this one wouldn't exist this ticker for feature request if can exist good optimal tool for installation.

[pfmoore]

I guess if that's your view, then all I can say is that PRs are welcome. I personally think the trade-offs are such that this won't happen, but I'm willing to be convinced otherwise if someone can provide working code that shows it's possible to do it in an efficient manner...

[ManPython]

Not only my.. check link.. PR as communicating negative experience?
I think is not problem to delivery code.. but who can consume time here to solve it.. or who will be funding this.. coz.. open source.
I'm thinking to resolve.. but I should working on other project as main, not environment as tools.. this is problem.. and typical for open source as out of vision and targets.. and worst.. conflict of interest.. - sure exist.