Python 2 only deps restricting upgrade #2651

jakirkham · 2016-06-10T21:59:31Z

Suppose I have package that can be on Python 2 or Python 3, but it requires some backport modules for Python 2. If I start on Python 2, the upgrade to Python 3 gets blocked by the Python 2 dependencies. However, we all know they are not really blockers and we can try to remove them and workaround them. How effective this will be depends on how locked in your stack is to them. Here is a simple example with jsonschema. The recipe comes from conda-forge so we can clearly see functools32 is a Python 2.7 dependency only from the recipe. First I try to go straight to Python 3.

$ conda create -n jssenv python=2 jsonschema
Using Anaconda Cloud api site https://api.anaconda.org
Fetching package metadata: ............
Solving package specifications: .........

Package plan for installation in environment /zopt/conda2/envs/jssenv:

The following NEW packages will be INSTALLED:

    functools32: 3.2.3.2-py27_1 conda-forge
    jsonschema:  2.5.1-py27_0   conda-forge
    openssl:     1.0.2h-1       defaults   
    pip:         8.1.2-py27_0   defaults   
    python:      2.7.11-0       defaults   
    readline:    6.2-2          defaults   
    setuptools:  23.0.0-py27_0  defaults   
    sqlite:      3.13.0-1       conda-forge
    tk:          8.5.19-0       conda-forge
    wheel:       0.29.0-py27_0  defaults   
    zlib:        1.2.8-3        conda-forge

Proceed ([y]/n)? 

Linking packages ...
[      COMPLETE      ]|################################################################################################################################| 100%
#
# To activate this environment, use:
# $ source activate jssenv
#
# To deactivate this environment, use:
# $ source deactivate
#
$ conda install -n jssenv python=3 
Using Anaconda Cloud api site https://api.anaconda.org
Fetching package metadata: ............
Solving package specifications: ....

The following specifications were found to be in conflict:
  - functools32 (target=functools32-3.2.3.2-py27_1.tar.bz2) -> python 2.7.*
  - python 3*
Use "conda info <package>" to see the dependencies for each package.

As this doesn't work, I remove the Python 2 dependencies (in this case only one). Fortunately, it finds an older version of jsonschema where this dependency is not required. However, this is not always the case.

$ conda remove -n jssenv functools32
Using Anaconda Cloud api site https://api.anaconda.org
Fetching package metadata: ............
Solving package specifications: .........

Package plan for package removal in environment /zopt/conda2/envs/jssenv:

The following packages will be REMOVED:

    functools32: 3.2.3.2-py27_1 conda-forge

The following packages will be DOWNGRADED:

    jsonschema:  2.5.1-py27_0   conda-forge --> 2.4.0-py27_0 defaults

Proceed ([y]/n)? 

Unlinking packages ...
[      COMPLETE      ]|################################################################################################################################| 100%
Linking packages ...
[      COMPLETE      ]|################################################################################################################################| 100%
$ conda install -n jssenv python=3 
Using Anaconda Cloud api site https://api.anaconda.org
Fetching package metadata: ............
Solving package specifications: .........

Package plan for installation in environment /zopt/conda2/envs/jssenv:

The following packages will be downloaded:

    package                    |            build
    ---------------------------|-----------------
    jsonschema-2.4.0           |           py35_0          53 KB  defaults

The following NEW packages will be INSTALLED:

    xz:         5.0.5-1       conda-forge

The following packages will be UPDATED:

    jsonschema: 2.4.0-py27_0  defaults    --> 2.4.0-py35_0  defaults   
    pip:        8.1.2-py27_0  defaults    --> 8.1.2-py35_0  defaults   
    python:     2.7.11-0      defaults    --> 3.5.1-0       conda-forge
    setuptools: 23.0.0-py27_0 defaults    --> 23.0.0-py35_0 defaults   
    wheel:      0.29.0-py27_0 defaults    --> 0.29.0-py35_0 defaults   

The following packages will be DOWNGRADED:

    sqlite:     3.13.0-1      conda-forge --> 3.9.2-0       conda-forge

Proceed ([y]/n)? 

Fetching packages ...
jsonschema-2.4 100% |#############################################################################################################| Time: 0:00:00 361.96 kB/s
Extracting packages ...
[      COMPLETE      ]|################################################################################################################################| 100%
Unlinking packages ...
[      COMPLETE      ]|################################################################################################################################| 100%
Linking packages ...
[      COMPLETE      ]|################################################################################################################################| 100%

Now I don't know what the right solution is here. Maybe it is worth telling the user that these packages must be removed as part of the upgrade and ask for confirmation that this is ok. While this example is simple, it becomes less straightforward for larger stacks (if it is possible at all). I would really appreciate hearing some thoughts on this.

The text was updated successfully, but these errors were encountered:

jakirkham · 2016-06-10T22:04:16Z

These problems can also occur between Python 3.4 and Python 3.5 too. I would recommend that people play with a Jupyter Notebook install and cycle through Python versions to get a sense of the work involved.

mcg1969 · 2016-06-10T22:46:33Z

Actually, I think this would be solved by something we're kicking around. The idea is to differentiate between requested packages and those installed solely as dependencies. We'd then allow conda to remove those "dependency" packages if necessary to accomplish upgrade goals. Of course, we'd always minimize the number of such removals, but in a case like this, it seems like it would be worthwhile.

mcg1969 · 2016-06-10T22:47:36Z

This one is related, and amusing: #2480

mcg1969 · 2016-06-10T22:49:28Z

Here's a simple option: build empty versions of these "compatibility" packages. For example, build a version of enum34 that depends on python >=3.4, or functools that depends on python >=3.2. That way, when you upgrade Python, the true package gets removed, replaced with the dummy package.

jakirkham · 2016-06-10T23:02:42Z

The idea is to differentiate between requested packages and those installed solely as dependencies. We'd then allow conda to remove those "dependency" packages if necessary to accomplish upgrade goals. Of course, we'd always minimize the number of such removals, but in a case like this, it seems like it would be worthwhile.

👍 Love this idea.

So, I'm guessing requested means one did conda install <X> and now <X> is added to requested column, but its dependency <Y> is not. Doesn't yum do something similar? Anyways this sounds like a very good idea to me.

Here's a simple option: build empty versions of these "compatibility" packages.

Yeah, I was thinking about that as a workaround. I'm a little worried that someone might get confused and assume it should actually be installed though.

mingwandroid · 2016-06-10T23:13:06Z

On arch linux and msys2, pacman records explicitly requested packages. It makes it really easy to move to a new machine; you list the explicitly requested packages via pacman -Qqe then reinstall that list via pacman -S

jakirkham · 2016-06-10T23:17:05Z

Great info, @mingwandroid! I figured there was a precedent for this behavior elsewhere. It might be good if we can compare some more approaches and see what we can glean from them.

mingwandroid · 2016-06-10T23:32:47Z

Explicitly requested is so important I think. If the solver can use that to bias its actions (or its prompts at least) then that would be awesome.

mcg1969 · 2016-06-11T00:33:17Z

So, I'm guessing requested means one did conda install and now is added to requested column, but its dependency is not. Doesn't yum do something similar? Anyways this sounds like a very good idea.

Indeed, conda already keeps track of the command history applied to each environment, so we can keep track of the packages that we have explicitly installed. But we have to be a bit careful, too. For example, consider:

conda install A B C --- with C depending on B
conda remove B --- C is removed as well
conda install C --- B is reinstalled

After this sequence of commands, the set of "requested" packages is A C. If we had just looked at the install requests, we might conclude A B C. Perhaps not the end of the world if we overestimate like that, but since we have the history, we can be reasonably smart about it.

mcg1969 · 2016-06-11T00:35:39Z

Yeah, I was thinking about that as a workaround. I'm a little worried that someone might get confused and assume it should actually be installed though.

Fair enough. But at least it's benign if they do make that mistake. And once the above "install history" is implemented, these packages will usually be marked for removal.

mcg1969 · 2016-06-11T00:36:50Z

Honestly, I think we need to do a deep dive on pacman and see how much of its existing work we should steal. Though I think namespaces have the potential to provide something truly useful and differentiating.

mingwandroid · 2016-06-11T00:52:29Z

pacman being a rolling package manager means it can discard many considerations that conda can't, is at least not optimise for them.

Pinning for example is more or less verboten. The emphasis is on latest is greatest, which, if enough systems adopt improves the status quo.I

don't mean this as a slight against conda, far from it. We have the additional constraint that people might require specific versions of software.

I've never looked into pacman's solver though. With its relatively simple requirements it has always 'just worked'.

mcg1969 · 2016-07-07T23:33:19Z

As promised, though way overdue, here is the beginning of a document I've begun to talk about Conda namespaces. It doesn't have anything about implementation details yet---rather, it enumerates some guiding principles that I believe namespaces must satisfy in order to represent a good design. Not surprisingly those principles lead to the design I had in mind, but still.

For those of you who are keenly interested in this I invite discussion there. I'm not ready to move this over to a conda issue or WIP PR, because until I flesh out the implementation details, I am concerned I may be missing a fundamental problem with the approach. Once the document is fleshed out I'll make it a full-fledged WIP.

jakirkham · 2016-07-07T23:37:24Z

Sharing it on the conda-forge gitter.

jakirkham · 2016-07-07T23:43:22Z

I found a nit, but I can't figure out how to PR a gist.

jakirkham · 2016-11-29T20:01:30Z

Opened issue ( #3968 ) to discuss this idea of distinguishing between the tracked (user installed) packages and non-user installed packages.

kalefranz · 2017-05-09T03:22:52Z

Looks like discussion has slowed here, and we've got #3968. Going to close this ticket.

github-actions · 2021-10-15T06:02:11Z

Hi there, thank you for your contribution to Conda!

This issue has been automatically locked since it has not had recent activity after it was closed.

Please open a new issue if needed.

jakirkham mentioned this issue Nov 29, 2016

Track requested packages #3968

Closed

kalefranz closed this as completed May 9, 2017

github-actions bot added the locked [bot] locked due to inactivity label Oct 15, 2021

github-actions bot locked as resolved and limited conversation to collaborators Oct 15, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Python 2 only deps restricting upgrade #2651

Python 2 only deps restricting upgrade #2651

jakirkham commented Jun 10, 2016

jakirkham commented Jun 10, 2016 •

edited

mcg1969 commented Jun 10, 2016

mcg1969 commented Jun 10, 2016

mcg1969 commented Jun 10, 2016 •

edited

jakirkham commented Jun 10, 2016

mingwandroid commented Jun 10, 2016

jakirkham commented Jun 10, 2016

mingwandroid commented Jun 10, 2016

mcg1969 commented Jun 11, 2016 •

edited

mcg1969 commented Jun 11, 2016

mcg1969 commented Jun 11, 2016

mingwandroid commented Jun 11, 2016

mcg1969 commented Jul 7, 2016 •

edited

jakirkham commented Jul 7, 2016

jakirkham commented Jul 7, 2016

jakirkham commented Nov 29, 2016

kalefranz commented May 9, 2017

github-actions bot commented Oct 15, 2021

Python 2 only deps restricting upgrade #2651

Python 2 only deps restricting upgrade #2651

Comments

jakirkham commented Jun 10, 2016

jakirkham commented Jun 10, 2016 • edited

mcg1969 commented Jun 10, 2016

mcg1969 commented Jun 10, 2016

mcg1969 commented Jun 10, 2016 • edited

jakirkham commented Jun 10, 2016

mingwandroid commented Jun 10, 2016

jakirkham commented Jun 10, 2016

mingwandroid commented Jun 10, 2016

mcg1969 commented Jun 11, 2016 • edited

mcg1969 commented Jun 11, 2016

mcg1969 commented Jun 11, 2016

mingwandroid commented Jun 11, 2016

mcg1969 commented Jul 7, 2016 • edited

jakirkham commented Jul 7, 2016

jakirkham commented Jul 7, 2016

jakirkham commented Nov 29, 2016

kalefranz commented May 9, 2017

github-actions bot commented Oct 15, 2021

jakirkham commented Jun 10, 2016 •

edited

mcg1969 commented Jun 10, 2016 •

edited

mcg1969 commented Jun 11, 2016 •

edited

mcg1969 commented Jul 7, 2016 •

edited