Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New Resolver: Rollout, Feedback Loops and Development Flow #6536

Open
pradyunsg opened this issue May 25, 2019 · 57 comments
Open

New Resolver: Rollout, Feedback Loops and Development Flow #6536

pradyunsg opened this issue May 25, 2019 · 57 comments

Comments

@pradyunsg
Copy link
Member

@pradyunsg pradyunsg commented May 25, 2019

I've been thinking a bit about #988 (duh!) -- specifically how to roll it out so as to minimize breakage and maximize the opportunity to get useful feedback from users.

Filing this issue now that I finally have both thumbs + time at hand to do so. Obviously, all of what follows is up for discussion. :)


My current plan for rolling out the new resolver is based on exposing the new resolver behind a flag. The flow would be to not document it initially and add big fat warnings on the use of the flag. Once it is less experimental and more beta-ish, we can start inviting users to play with the new resolver. This would involve CTAs to users for asking them to try it out and provide feedback. This information might also be printed when run with the flag.

In terms of feedback management, I am thinking of requesting for feedback on a different repository's issue tracker. The reasoning behind putting issues on a different issue tracker, is to minimize noise here + allow more focused discussions/investigation. I'd bubble up anything that's more than a "bug in the resolution" to the main issue tracker (this one).

In terms of transitioning, I think once there's enough confidence in the new resolution logic, we can look into how we want to handle the transition. Having put this behind a flag, we'll have 2 options -- directly switch over in a release or "stabilize" the new resolver and do a (maybe multi-release?) "transition period". I do think that we can do the transition planning later, when we have a better understanding of the exact trade-offs involved.

In terms of git/GitHub, this is probably the first "experimental" feature implementation within pip. FWIW, I'm planning to do experiments etc on my fork and regularly merging progress to pip's main repository itself (solely code, into pip._internal.resolution). I don't want to be noisy on the main repository but I do want to keep master in sync with work on this.


Note that I'm putting #5051 as a blocker for this work because of how painful dealing with build logic was when building the prototype.

@cjerdonek

This comment has been minimized.

Copy link
Member

@cjerdonek cjerdonek commented May 25, 2019

I don't know how you have it planned out, but one comment is that I would encourage you to try to share code as much as possible between the new code and the current code, and refactor the current code as you're working to allow more sharing between the new and current code paths.

One reason is that if you're sharing more code, there will be less chance of breakage when you're toggling the new behavior off and on, because you'll be exercising that shared code in both states and you won't have as many potential differences in behavior to contend with.

@pfmoore

This comment has been minimized.

Copy link
Member

@pfmoore pfmoore commented May 25, 2019

This would involve CTAs to users for asking them to try it out and provide feedback

Our track record on getting advance feedback on new features has been pretty bad. We've tried beta releases, releasing new features with "opt out" flags that people can use if they hit issues, big publicity drives for breaking changes, and none of them seem to have worked.

My personal feeling is that "make it available and ask for feedback" is an interesting variation on what we've previously tried, but ultimately it won't make much difference. Too many people use the latest pip with default options in their automated build pipelines, and don't test before moving to a new pip version (we saw this with PEP 517).

I wonder - could we get a PSF grant to get resources to either do a big "real world" testing exercise for this feature, or (better still) develop a testing infrastructure for us? Such a project could include a call for projects to let us know their workflows and configurations, so that we can set up testing paths that ensure that new pip versions don't break them. Or even just use a grant to get someone experienced in the communications aspect of getting beta testers for new features to help us set up a better user testing programme?

In terms of git/GitHub, this is probably the first "experimental" feature implementation within pip

I'm not 100% sure what you mean by that. We've certainly had new features in the past that have been added while the "old way" was still present. We've not tended to leave them "off by default, enable to try them out", if that's what you mean, but that's mostly because we've never found any good way to get feedback (see above).

@pradyunsg

This comment has been minimized.

Copy link
Member Author

@pradyunsg pradyunsg commented May 26, 2019

I spent ~60 minutes (re-re-re-re-re-)writing this one post, so now I will go take a look at places in New York! If you don't see an quick response from me, it's because I'll be in tourist mode.


I would encourage you to try to share code as much as possible between the new code and the current code, and refactor the current code as you're working to allow more sharing between the new and current code paths.

Definitely! This is 80% of why I'm putting #5051 ahead of this -- I intend to pay down a lot of the technical debt we've accumulated in our build logic so that it becomes easier to reuse (all of?) it. A bunch of the code will have to be 🔥 and I agree that the rest should definitely be reused as much as reasonable.

We've not tended to leave them "off by default, enable to try them out", if that's what you mean

Yep, indeed. I'm also hinting at development flow here -- IMO it would be okay to merge empty infrastructure (classes with a bunch of methods that are just raise NotImplementedError() that will get fleshed out in subsequent PRs) or one doesn't cover all the cases (half-baked implementation) into the master branch as long as that's only used behind the flag that is explicitly noted as "experimental/alpha".

re: feedback

I'm young, dumb and optimistic -- I want to make this rollout an opt-in, to get proactive feedback and act on it. By "proactive", I mean from folks who are willing to take out some extra time to try out alpha/beta functionality and inform us about how it is. I think if we make make enough noise and strategically target/reach out to people, we can get good "proactive" feedback from folks who have the time and energy to try out new functionality to help iron out the details/issues.

Looking at our recent "major" changes, I think most of the feedback we received was reactive -- from users realizing issues with their workflows when it broke, and then reaching out to inform us about it. A lot of them may not have had the time to help iron out the details of the new functionality, which causes a lot of friction. These also cost us a lot of our "churn budget" [1], which I don't want to spend more of, since Python Packaging doesn't really have much left anyway [2].

FWIW, I plan to borrow some ideas from the PyPI launch, like making blog posts at fairly visible locations (i.e. not my personal blog), possibly going on podcasts, well-timed actionable emails etc. I'm also looking for more prior art/avenues to communicate via. One of the (many!) things I learnt at PyCon, was that there are channels that we don't use, that will help spread information but won't seek out to check if we have any to spread.

To be clear, I'm not criticizing against the rollout approach we took for PEP 517, I think it's going well, especially given the fact that we're all volunteers. I'm trying to see what we can learn and actionable items to try to avoid the problems we had. Most of these items do involve more work from the maintainers and the main reason I am even spending all this time thinking about this, is because I view this as a fun learning exercise of how to do change management.

re: grants

Yep, I think we can definitely use a grant/more experienced person to help us figure out the communication, roll-outs and testing infrastucture. That does however need someone to do the grant-writing work and figuring out more concrete plans than I can make right now, since I don't have a more stable number of hours / week that I can guarantee.

FWIW, PSF has an ongoing contract to help figure out PyPA/Packaging-related communication with Changeset Consulting, so maybe we can leverage that?


I'm intentionally not @-mentioning people since this is fairly early in the planning state to add more people in the conversation.

Footnotes:

  1. A really nice term that @ pganssle used that I'm definitely going to use.
  2. This is why I've put #3164 on the back burner, despite having an implementation of the "pip-cli" package proposed there and having reasonable consensus on how we want the rollout to look like.
@pfmoore

This comment has been minimized.

Copy link
Member

@pfmoore pfmoore commented May 26, 2019

I'm young, dumb and optimistic

:-) And I'm sometimes too old, weary and cynical. Let's go with your philosophy, it sounds much better :-)

@cjerdonek

This comment has been minimized.

Copy link
Member

@cjerdonek cjerdonek commented May 26, 2019

Definitely! This is 80% of why I'm putting #5051 ahead of this -- I intend to pay down a lot of the technical debt we've accumulated in our build logic so that it becomes easier to reuse (all of?) it.

Great!

@brainwane

This comment has been minimized.

Copy link
Member

@brainwane brainwane commented Jun 13, 2019

From IRC just now:

[sumanah] pradyunsg: is there anything we the pip & packaging community can do to help you get more work done faster on the resolver?
....
[pradyunsg] Actually, right now, inputs on #6536 would probably help me figure out how to approach the work / get feedback from people etc.
....
[sumanah] pradyunsg: re: New Resolver: Rollout, Feedback Loops and Development Flow #6536 -- the input you want is something like: is the feature flag approach a good idea? is it a good idea to get feedback via some mechanism other than the pip GitHub issues? is it a good idea to get a grant or similar to get realworld manual testing & robust testing infrastructure built, and/or proactive comms?
...
[pradyunsg] Yep -- whether the ideas I'm suggesting are good. Also any additional ideas/approaches/thoughts that might help the rollout + feedback be smoother would be awesome.

So:

Is the feature flag approach a good idea? Yes.

Is it a good idea to get feedback via some mechanism other than the pip GitHub issues? Yes. We should find automated ways to accept less structured bug reports from less expert users.

Would more robust testing infrastructure help? Yes, a lot, and this is someplace our sponsors might be able to help us out.

Could Changeset (me), under the existing contract with PSF to help with PyPA coordination/communications, help pip with proactive communications to get us more systematic realworld manual testing? Assuming that I have hours remaining in my contract by the time we want to start this rollout, yes.

is it a good idea to get a grant or similar to get more help with user experience, communications/publicity, and testing? Yes. The PSF grants would potentially be of interest, as would NLNet grants (for requests under 30,000 euros), potentially the Chan Zuckerberg essential open source software for science grant, and Mozilla's MOSS. The Packaging WG can be the applicant of record. If @pradyunsg or @pfmoore wants to give a "yeah that sounds interesting" nod, I can start investigating those possibilities with the WG.

@pfmoore

This comment has been minimized.

Copy link
Member

@pfmoore pfmoore commented Jun 13, 2019

If @pradyunsg or @pfmoore wants to give a "yeah that sounds interesting" nod,

It definitely sounds interesting to me :-)

@pradyunsg

This comment has been minimized.

Copy link
Member Author

@pradyunsg pradyunsg commented Jun 13, 2019

@pradyunsg or @pfmoore wants to give a "yeah that sounds interesting" nod

nods yeah that sounds interesting

@pradyunsg

This comment has been minimized.

Copy link
Member Author

@pradyunsg pradyunsg commented Jun 19, 2019

Would more robust testing infrastructure help? Yes, a lot, and this is someplace our sponsors might be able to help us out.

@brainwane Also relevant here is https://github.com/pypa/integration-test. I think getting this set up, is another potential area for funding -- we should add this to https://wiki.python.org/psf/Fundable%20Packaging%20Improvements.

@brainwane

This comment has been minimized.

Copy link
Member

@brainwane brainwane commented Jun 21, 2019

OK! I've started talking with the PSF and with the Chan Zuckerberg Initiative folks about applying for a CZI grant via the Packaging Working Group. I've added some details to the Fundable Packaging Improvements page about why the new pip resolver's important, and added the integration-test project to that list. And I've started gathering names of user experience experts who have the capacity to research our complicated all-on-the-command-line package distribution/installation toolchain, talk with users to understand their mental model of what's happening and what ought to happen, and advise maintainers.

If we get money via grants from MOSS, CZI, or NLNET, I think we'd get the money ... October at the earliest, probably. A grant directly from the PSF would be faster probably but "Our current focus is Python workshops, conferences (esp. for financial aid), and Python diversity/inclusivity efforts."

@techalchemy

This comment has been minimized.

Copy link
Member

@techalchemy techalchemy commented Jun 21, 2019

One consideration is that I know Brett & the folks over on the steering council are talking about investing in project management and looking into having some sort of paid resources for managing these projects (triage, project management, etc) and they are talking with the PSF directly. It may be worth reaching out and finding out what they are doing or thinking, since I heard some talks of long term sustainability and it'd be a good thing to be involved in those.

Feature flags are good, opt-ins are good. One thing you might consider is whether you could randomly prompt users to try out the resolver (like, very very very infrequently and only for an install at a time, i.e. not forcing them to turn it on permanently). Then you could indicate how the resolver was helpful (e.g. what did it do for them? what conflicts did it encounter and resolve?)

People coming from javascript or rust for example will also expect a lockfile of some kind, so that may be something to consider...

Sorry to jump in, glad to see this moving ahead!

@jriddy

This comment has been minimized.

Copy link

@jriddy jriddy commented Jun 27, 2019

My personal feeling is that "make it available and ask for feedback" is an interesting variation on what we've previously tried, but ultimately it won't make much difference. Too many people use the latest pip with default options in their automated build pipelines, and don't test before moving to a new pip version (we saw this with PEP 517).

As one of the people that got bit by some PEP 517 issues for this very reason, I'd actually love to see an opt-in way of testing things out. But I only know about this kinda stuff because i subscribed to all the python packaging news sources I could after the --no-use-pep517 flag issue. What I'm saying is that spreading this kind of news is hard and is probably why feedback is hard to get.

I think more people would be interested in this if the information could be disseminated better. Is that what the resources you are seeking would allow for?

@chrish42

This comment has been minimized.

Copy link

@chrish42 chrish42 commented Jul 5, 2019

To continue on what jriddy is saying, I also feel it'll be really hard to get people to test various feature flags if they have to know about them, make changes to their CI setup for each new flag, etc.

What would seem much more doable, however, is if there is only one feature flag to know about, to test "what's coming up next" in terms of changes that need to be tested. Then people and companies could setup their CI to run that also (without failing builds for errors). I'm thinking of something similar to Rust, where these kinds of changes bake in the "beta" channel of the toolchain, and it's easy to setup another CI channel to run things on the beta toolchain, and send errors to someone.

The key thing is, this setup needs to be learned about and done only once, instead of having to continuously learn about new individual feature flags, modify CI setups or test them manually.

@jriddy

This comment has been minimized.

Copy link

@jriddy jriddy commented Jul 5, 2019

What would seem much more doable, however, is if there is only one feature flag to know about,

In a sense, doesn't this already exist in the form of --pre? Could the beta release channel for pip just be a matter of running pip install --upgrade --pre pip?

@pradyunsg

This comment has been minimized.

Copy link
Member Author

@pradyunsg pradyunsg commented Jul 5, 2019

Sorry to jump in, glad to see this moving ahead!

@techalchemy please, of all people, you definitely don't have to be sorry for pitching in this discussion.

@pradyunsg

This comment has been minimized.

Copy link
Member Author

@pradyunsg pradyunsg commented Jul 6, 2019

Is that what the resources you are seeking would allow for?

To an extent, yes.

reg: beta releases/"channel" for pip

Thanks for chiming in @jriddy and @chrish42. While I think that generally that's definitely an useful/important conversation to have, I also feel it's slightly OT for this issue. None the less, I'll respond here once; if we want to discuss this more, let's open a new issue.

We've tried that in the past -- most recently with pip 10 -- but it hasn't worked out well. I am slightly skeptical of how well that might work going forward too, but I can also imagine that some changes to our process might result in this working smoothly for us. Maybe we could do a "beta only" set of features or something? I'd imagined -X all as a syntax for that in #5727. Maybe we could pick that up as a part of this rollout plan? Idk. We'll need to invest time and energy to figure this out. :)

@msarahan

This comment has been minimized.

Copy link

@msarahan msarahan commented Aug 10, 2019

As mentioned in pypa/packaging-problems#25 (comment), I think it's important to have a consolidated explanation of how a solver changes the pip experience. Lots of people will be frustrated by the shift to a more rigid system (even though things should be more reliable overall, they'll get blocked in places where they are not currently getting blocked.

Having a central explanation of how things have changed and why it is a good change will make responding to those angry people much simpler. Post a link and see if they have any further questions.

The prerelease is a good idea. In conda, we have a prerelease channel, conda-canary. We encourage people to set up a CI job to run against canary in a way that helps them see if conda changes are going to break them. Ideally they let us know before we release that version. That channel has been a pretty dismal failure. The only time people really seem to use it is when they want to get the newest release to fix some bug that they are struggling with. We do not get many reports from our intended early adopters. I still think the prerelease is a good idea, because when a release goes poorly and people are angry with you for breaking their 700 managed nodes, you can say "well, it was available for a week before we released it. Why aren't you testing these things before you roll them out to 700 nodes?" You are giving people an opportunity to make things work better. Help them realize that passing on that opportunity means more pain for them down the line. It's worthwhile investment for them, and if they do it as part of their CI, it costs them no time aside from setup.

Regarding the flag: I think it's better to have a config option (perhaps in addition to a flag). I would not want to pass a flag all the time. I'm not sure if pip has this ability - maybe you tell people who want a more permanent switch to use the corresponding env var?

@pradyunsg

This comment has been minimized.

Copy link
Member Author

@pradyunsg pradyunsg commented Aug 10, 2019

Regarding the flag:

pip's CLI options, automatically get mapped to a configuration file option and an environment variable, with the appropriate names.

@pradyunsg

This comment has been minimized.

Copy link
Member Author

@pradyunsg pradyunsg commented Aug 10, 2019

@msarahan Thanks for chiming in, much appreciated! :)

@ncoghlan

This comment has been minimized.

Copy link
Member

@ncoghlan ncoghlan commented Aug 12, 2019

Regarding the "let me do what I want" option to ignore broken dependencies, I think it would be desirable to structure the feature flag such that it can also serve as the opt out after the resolver gets turned on by default (for example, start with --avoid-conflicts as an opt-in, eventually move to --no-avoid-conflicts as an opt-out, but accept both options from the start)

You'll also want to consider how --ignore-installed interacts with the solver - when it is passed, you should probably ignore all the requirements for already installed packages.

Beyond that, handling things as smaller refactoring patches to make integration of the resolver easier is an excellent way to go (that's the approach that made the new configuration API for CPython possible: a lot of private refactoring that was eventually stable enough to make public)

@pradyunsg

This comment has been minimized.

Copy link
Member Author

@pradyunsg pradyunsg commented Aug 15, 2019

@ncoghlan What does "opting out" of the resolver mean? Completely avoiding dependency resolution (and hence the resolver) is --no-deps. I understand that there's a need for an "ignore version conflicts on this package" or something along those lines.

Personally, I don't see any point in keeping the "keep first seen" resolution logic for longer than a transition period to a new resolver.

However, if there are use cases that these two options would not cover, I'd really like to know about them. :)


More broadly, if there are workflows that have issues with a strict resolver's behavior, I'm curious to know what those look like, as early as possible, to be able to figure out whether/how to support them.

@jriddy

This comment has been minimized.

Copy link

@jriddy jriddy commented Aug 15, 2019

Personally, I don't see any point in keeping the "keep first seen" resolution logic for longer than a transition period to a new resolver.

IDK, I use this "feature" to do some pretty crazy stuff with builds, like...

# install just the packages I've built specifically
pip install --no-index --no-deps --find-links=/path/to/my/local/build/cache -r local-reqs.txt

# ...snip to later in a dockerfile, etc...

# install the deps from public PyPI
pip install -r local-reqs.txt

In this case i'm asking it to resolve my dependencies after i've installed some very pre-determined packages from a local wheelhouse. I suppose i could read my exact versions into that local-reqs file to make a resolver happy, but i've actually found the current behavior of pip quite useful in allowing for these kinds of arbitrary build injections steps. Could be a case of the spacebar heating workflow though, I'll admit.

But maybe the "naive resolution" behavior still has a use.

@pfmoore

This comment has been minimized.

Copy link
Member

@pfmoore pfmoore commented Aug 15, 2019

I agree with @pradyunsg. I don't think it's viable to maintain the existing code and a new resolver indefinitely. Certainly as a pip maintainer I have no interest in doing that.

From an end user POV, I accept that there could well be weird scenarios where the new resolver might not do the right thing. And having an emergency "give me back the old behaviour" flag is an important transition mechanism (although it's arguable whether "temporarily roll back to the previous version of pip" isn't just as good - even though things like common use of CI that automatically uses the latest pip make advocating that option problematic). But long term, why would we need to retain the current behaviour? I can imagine the following main situations:

  1. Resolver bug. Obvious possibility, easy fix - correct the bug in the next release of pip.
  2. Cases where the old resolver is wrong (generates results that fail to satisfy the constraints). We don't intend to support that going forward, surely? (At least not via anything less extreme than the user pinning what they want and using --no-deps to switch off the resolver).
  3. Cases where the old and new resolvers give different results, both of which satisfy the given constraints. Users can add constraints to force the old result (if they can't, that puts us back into (2)). We should give them time to do so, but then drop the old resolver, just like any other deprecated functionality.
  4. An edge case that we consider too complex/weird to support. This is like (3), but where we aren't asserting that the new resolver gives the "right" result. Users can still modify constraints to avoid the weird case, or pin and use --no-deps. But ultimately, we're saying "don't do that", and if users ignore that message, then again at some point we remove the old resolver saying "we warned you".

Are there any others that I've missed? In particular any where deprecating and then removing the old resolver isn't possible?

By the way, where's the best place to post "here's an edge case I thought of" scenarios, so that they don't get lost? I think it would be useful to collect as many weird situations as we can in advance, if only so we can get an early start on writing test cases :-)

PS We should probably also as part of prep work for the new resolver, survey what the "typical" constraint problems are (based on what's on PyPI). For my own part, it's pretty rare that I have anything more complex than "pip install ". It would be a shame to get so bogged down in the complex cases that we lose sight of the vast majority of simpler ones.

@rgommers

This comment has been minimized.

Copy link

@rgommers rgommers commented Aug 15, 2019

  1. resolver is too slow (see conda). If I have to choose between a 20 min plus resolver, or the current behavior, often I want the current behavior (or at least try; in many cases it will happen to give a result that's fine).

  2. metadata wrong. not as much of a problem today, but it's easy to imagine cases that should be solvable but aren't. PyPI metadata is in worse shape than conda/conda-forge metadata, and it's already a problem for conda. if it's wrong and as a user I can't get a solution, I'll want to get some opt-out.

@pradyunsg

This comment has been minimized.

Copy link
Member Author

@pradyunsg pradyunsg commented Aug 15, 2019

@rgommers For 6, the "ignore version conflicts on this package" style option could work, right?

@rgommers

This comment has been minimized.

Copy link

@rgommers rgommers commented Aug 15, 2019

Actually, I'm not sure if pip can even downgrade things. It first had a too aggressive "upgrade everything" strategy, now "upgrade as needed" works pretty well. I've never seen it downgrade anything, and in practice it works pretty well for regular use.

@msarahan

This comment has been minimized.

Copy link

@msarahan msarahan commented Aug 15, 2019

An example downgrade that conda must contend with, but pip will not: users of anaconda or miniconda with python 3 start with python 3.7. Users sometimes need to install something that is available only for python 3.6. The solver must downgrade python and all other non-noarch packages. This is a special case that could perhaps is be optimized by having special behavior for python version changes, but it illustrates the point of how changes in one package may require downgrades to another. "It has worked thus far" is dumb luck, not what actually works all the time.

As for restricting versions, you can't apply that in the solve itself. You have your constraints, and any sort of of "open up by one version" can't hope to be general enough, because it's different for every package. You can cut metadata by version prior to the solver, though. That's what conda's "current_repodata.json" is. Only the latest version. It makes things go really fast when it works, but people would get really mad if that were the only repodata. Things would not be reproducible, and they would get frustrated that specs that work one day may not the next. We provide a fallback to the full repodata, and also plan to introduce time-based subsets, with only the newest versions available at given points in time. Incrementally opening up the available index data might be a more useful concept with the backtracking solver.

@pradyunsg

This comment has been minimized.

Copy link
Member Author

@pradyunsg pradyunsg commented Aug 15, 2019

pip can even downgrade things.

It can -- for pip, downgrading is just an uninstall-install step like upgrading. And it does downgrade when it sees a constraint that it's been asked to satisfy.

The following does downgrade setuptools -- as long as it's the first thing pip sees:

pip install "setuptools < 20.0"
@pfmoore

This comment has been minimized.

Copy link
Member

@pfmoore pfmoore commented Aug 15, 2019

where's the best place to post "here's an edge case I thought of" scenarios, so that they don't get lost?

https://github.com/pradyunsg/zazo/

Thanks, I'll keep that in mind. Although on rethinking, my "pathological case" is not actually that pathological. It's mostly just an extreme case of the fact that in order to know the dependency metadata for a package, you have to download and unpack the package, and in certain cases this can trigger downloading a lot of packages only to reject them. That might be an important limitation of the "simple index" protocol that we need to address, but it's not directly a resolver issue.

@rgommers

This comment has been minimized.

Copy link

@rgommers rgommers commented Aug 15, 2019

An example downgrade that conda must contend with, but pip will not: users of anaconda or miniconda with python 3 start with python 3.7. Users sometimes need to install something that is available only for python 3.6. The solver must downgrade python and all other non-noarch packages. This is a special case that could perhaps is be optimized by having special behavior for python version changes, but it illustrates the point of how changes in one package may require downgrades to another. "It has worked thus far" is dumb luck, not what actually works all the time.

It's also a case where you normally don't want to downgrade. As a user, I'd much rather get an exception telling me the package is not available, and let me explicitly install 3.6 if that's what I want.

Another example is inconsistency between channels. Recent example: we were at Python 3.7.3, then base got 3.7.4. I don't care about those version differences, and most users won't. A default "do nothing" would be way better than "hey .4 > .3, let's upgrade that and then change channels of other packages to base if we have to (even if that downgrades those)".

We provide a fallback to the full repodata, and also plan to introduce time-based subsets, with only the newest versions available at given points in time

That sounds like a very useful improvement.

The problem with "garbage in, garbage out" as a mentality is that as the maintainers of the solver, you are holding the bag. Unless you have very good heuristics for what is garbage, you are powerless.

Yeah, that's true. I think every IDE, distribution or "common interface to a bunch of stuff" has this issue.

The following does downgrade setuptools

That's a user-requested downgrade. An indirect one is what I meant (e.g. setuptools<20.0 in pyproject.toml`). That would work as well I guess, but it is rare in practice.

@pfmoore

This comment has been minimized.

Copy link
Member

@pfmoore pfmoore commented Aug 15, 2019

The problem with "garbage in, garbage out" as a mentality is that as the maintainers of the solver, you are holding the bag.

100% agreed. It's something that has to be handled very carefully. But conversely, trying to work out a sane behaviour for any old junk isn't viable - we have to draw a line somewhere.

Just as a reminder, this discussion was triggered by the question of whether we'd need to retain the existing resolver indefinitely. I'm not sure any of these points really impact that question - the existing resolver isn't right, the best you can say is that it's broken behaviour that people are familiar with. So I stand by what I said above, there's no reason to retain the old resolver beyond an initial transition period.

And actually, for a big chunk of use cases, the old and new resolvers will likely give the same results anyway.

@msarahan

This comment has been minimized.

Copy link

@msarahan msarahan commented Aug 16, 2019

"Right" doesn't matter to users when they get broken by your changes. Any change in behavior will absolutely infuriate some number of users. Telling them that their workflow is wrong and that they need to change hasn't been a terribly effective strategy for me. I think you need to play it by ear. I certainly wouldn't want to maintain the old resolver forever, but you probably need to give people a path to keeping it - like making it a plugin that someone else adopts and maintains, and people can install that separately from pip.

trying to work out a sane behaviour for any old junk isn't viable - we have to draw a line somewhere.

Yes, but what is "old junk?" What distinguishes it? What about bad junk vs good junk (bearing in mind that newer is not always better)? My advice is to spend a lot of time making the solver be very debuggable. Make it easy for users (not just you as the expert) to trace where things go sideways to make it easy to identify when bad metadata is the issue, and what that bad metadata is. This is a skill that most users currently do not have, and honestly most users I've seen don't want to have this skill. I don't think you (or conda for that matter) will ever be able to have a completely accurate automatic heuristic for bad vs. good.

It's also a case where you normally don't want to downgrade. As a user, I'd much rather get an exception telling me the package is not available, and let me explicitly install 3.6 if that's what I want.

@rgommers, conda 4.7 moved to this - requiring the explicit python spec to change minor versions. People have hated it. I have no idea what fraction of the population the vocal ones are, but lots of people really, really don't like that they used to be able to conda install something, and now they can't. They don't care much about the reason, and they're mostly mollified by the answer, but we still get the hostility to deal with in the meantime. Just another example of

"Right" doesn't matter to users when they get broken by your changes.

Another example is inconsistency between channels. Recent example: we were at Python 3.7.3, then base got 3.7.4. I don't care about those version differences, and most users won't. A default "do nothing" would be way better than "hey .4 > .3, let's upgrade that and then change channels of other packages to base if we have to (even if that downgrades those)".

This is a much more complicated point. You're basically proposing different optimization criteria. You might have seen the current 11 steps in our blog post at https://www.anaconda.com/understanding-and-improving-condas-performance/

These are extremely tricky, and there is no global optimum that satisfies every situation. Given binary compatibility, and channels as islands of said compatibility, the strong priority of channels is pretty essential. Python builds don't matter, you're right, but there's currently no way to express a binary compatibility constraint vs. a plain and simple version constraint. You would need that in order to think about your idea of just leaving things alone. Otherwise, you'd quickly be in binary incompatibility hell.

@wolfv

This comment has been minimized.

Copy link

@wolfv wolfv commented Aug 16, 2019

Hi, thanks @msarahan for at-mentioning me in this issue. I wanted to chime in earlier but haven't found the time.

Indeed I experimented quite a bit with using libsolv as a solver for conda specs. And it does work -- the remaining difference now is that conda does not care much about the build number, but libsolv in the way it is coded (and with the backtracking solver) does. Even when using the full repodata, libsolv is really fast – I would go as far as saying that the speed of libsolv is fast enough to not be annoying :)

My big contribution to libsolv was to make it cross-platform compatible so that it now compiles on Windows, OS X and Linux.

I would totally advocate to use libsolv for a new resolver, even though it's a blob of compiled code (at least it's fast) and @mlschroe might be available to help out – he helped a lot with the libsolv support for the conda matchspec.

On the other hand I am not sure in what stage the resolver development is and wether it's already too late now, and wether compiled code is acceptable or not.

@rgommers

This comment has been minimized.

Copy link

@rgommers rgommers commented Aug 16, 2019

You might have seen the current 11 steps in our blog post at https://www.anaconda.com/understanding-and-improving-condas-performance/

Indeed I did. That was a nice blog post.

You're basically proposing different optimization criteria.

not really. I think your "binary compatibility constraint vs. a plain and simple version constraint" just points to something I'm missing probably. What I'd expect is that no package should have python >= 3.7.4 or == 3.7.4 in its metadata, it's always == 3.7 (just checked for scipy, meta.yaml says python and conda_forge.yml says max_py_ver: '37' - makes sense). So introducing a 3.7.4 should do nothing - the resolver choosing 3.7.3 and not changing anything else is much cheaper (and valid according to your 11 steps) than forcing 3.7.4 and triggering a chain of up/down-grades.

Guess this part is getting off-topic for pip rollout plans, so happy to take this somewhere else.

My advice is to spend a lot of time making the solver be very debuggable.

+1

Also: make (keep) it as repairable as possible. That's the nice thing with pip now, if it messes up usually one can do cd site-packages && rm -rf troublesome_package (possibly followed by reinstall with --no-deps) and things work again. The likes of conda, apt and friends are much harder to repair that way.

@msarahan

This comment has been minimized.

Copy link

@msarahan msarahan commented Aug 16, 2019

You're basically proposing different optimization criteria.

I don't think you're factoring the concept of channels into your thinking enough. I don't know how relevant it is to pip. Definitely much less than it is to conda, but I'm not sure if it's totally irrelevant. People can still gather packages from more than one index at a time, right?

Packages don't have python >=3.7.4, nor ==3.7.4. The standard in conda packaging is to have an upper and lower bound. These are generally automatically determined by conda-build using information provided by the recipe author regarding how many places of the version to consider a compatible range. Packages have constraints like >=3.7.2,<3.8.0a0, with the 0a0 awkwardness being there to account for the fact that prerelease versions are below .0 releases, and would thus match a <3.8.0 spec where people don't really expect it to.

Packages also have a channel associated with them. This channel is effectively part of the version optimization: https://github.com/conda/conda/blob/4.6.7/conda/resolve.py#L1074 - the channel is like a super-version, one place ahead of the package's major version. If a solve should not change python, then the python spec can't be python ==3.7 - that is a range, and channel will affect that range. Specifically, having a python ==3.7 spec and starting with an install from the defaults channel, then adding the conda-forge channel will result in a lot of churn, because you have introduced new python packages that are higher in "version" (including channel), and your python spec is permissive of that change.

Conda 4.7 introduced much more aggressive "freezing" of specs, and I'm pretty sure that behavior is what you're after. That's pretty complicated, though. It boils down to only freezing things that don't conflict with your explicit specs. How you determine "conflict" is the hard part. We think it's better to not freeze things that would prevent the solver from giving the user the newest packages that are part of the graph of dependencies for that package. This freezing is worth mentioning because it can be done on a per-spec basis for pip in a way that conda can't. I think it might be a great optimization for a backtracking solver.

Also: make (keep) it as repairable as possible. That's the nice thing with pip now, if it messes up usually one can do cd site-packages && rm -rf troublesome_package (possibly followed by reinstall with --no-deps) and things work again. The likes of conda, apt and friends are much harder to repair that way.

Yes, this is very important. I think pip has done a good job of judiciously vendoring things so that it's harder to break. That is very wise, and conda is learning from that. If you do end up using any compiled code, make sure that it is statically linked or otherwise impossible for dynamic loading to cause problems.

@ncoghlan

This comment has been minimized.

Copy link
Member

@ncoghlan ncoghlan commented Aug 17, 2019

(libsolv tangent: back when I used to work for RH, I grabbed https://pypi.org/project/solv/ to close the "pip install solv" security loophole on Fedora, since the libsolv build process doesn't generate an sdist or wheel archive at the moment, let alone publish it to PyPI. Happy to chat to anyone that might be interested in make those real library bindings with a bundle copy of libsolv, rather than the innocuous placeholder it is now)

Regarding my "opting out" comment, I don't mean "fall back to the old installation logic", I mean "provide an option to skip installing packages that would cause constraint violations, rather than failing the entire installation request".

Yum/DNF offer that via their --skip-broken option (in DNF that flag is an alias for --setopt=strict=0), and I think pip's resolver should offer a similar option.

@pradyunsg

This comment has been minimized.

Copy link
Member Author

@pradyunsg pradyunsg commented Aug 18, 2019

@ncoghlan Ah right. That makes sense.

"ignore version conflicts on this package" style option

I'd already mentioned that we'd do that, which is why I got confused by your comment.

We're on the same page then. :)

@brainwane

This comment has been minimized.

Copy link
Member

@brainwane brainwane commented Sep 2, 2019

@ncoghlan replied to my proposed timeline on distutils-sig and said it sounds reasonable.

@pradyunsg - looking forward to your next monthly update!

@pradyunsg

This comment has been minimized.

Copy link
Member Author

@pradyunsg pradyunsg commented Nov 10, 2019

I spent some time taking a look at this again, and filed #7317.

I think we're almost there w.r.t. the abstractions -- thanks to a lot of work on pip's index interaction, dependency resolution + build logic separation and a bunch of general cleanup.

@pradyunsg

This comment has been minimized.

Copy link
Member Author

@pradyunsg pradyunsg commented Nov 12, 2019

I just closed #7317. As far as I can tell, the dependency resolution is now decoupled (enough) from the metadata build logic. The build logic refactor has progressed well and it is no longer a blocker for further progress now.

We can now begin working on implementing the resolvelib abstractions in pip, taking reference from passa and poetry's resolver where appropriate. :)

@sdispater

This comment has been minimized.

Copy link

@sdispater sdispater commented Nov 12, 2019

@pradyunsg I am planning on extracting the base resolver (based on PubGrub) from the Poetry codebase (see https://github.com/sdispater/poetry/tree/master/poetry/mixology). It's mostly decoupled from the rest of the code but there are still references to internal parts that I need to abstract.

If you are interested in helping with that please let me know. The idea is to have a standalone implementation of the PubGrub algorithm that can be used by third parties and will be put in https://pypi.org/project/mixology/ which currently holds the code of the old resolver.

@pradyunsg

This comment has been minimized.

Copy link
Member Author

@pradyunsg pradyunsg commented Nov 13, 2019

@sdispater Definitely! I don't know if I can help directly (time constraints) but it'd be awesome if you could decouple the PubGrub port from the rest of poetry!

One of the things that'd be really nice, would be to have a consistent abstraction layer, such that pip, poetry and pipenv use the same abstractions. Right now, we have zazo (mine), mixology (poetry's) and resolvelib (pipenv's) -- all define an abstraction layer of some sort and they're slightly different but (ridiculously!) similar. If you're open to this, do let us know!

@SylvainCorlay

This comment has been minimized.

Copy link

@SylvainCorlay SylvainCorlay commented Nov 25, 2019

FYI, we (@wolfv, and the @QuantStack team in general) have responded to the RfP for the pip dependency resolver.

The proposed approach is to adopt the libsolv C library, and to contribute the support for pip's version constraints format to libsolv. We would expose the C library through new Python bindings.

Libsolv is battled-hardened library underlying the RPM ecosystem, and therefore already used at industrial scale.

  • Libsolv's is distributed under the BSD-3-Clause license.
  • Libsolv supports multiple packages and repository formats, such as rpm, deb,
    haiku, conda, arch. Importantly, the variety of formats and ways to express
    dependency constraints shows that it is a pluggable system that should be able
    to accommodate pip's syntax for constraints on dependency versions..
  • Using libsolv instead of conda's solver in the thin mamba wrapper, we were
    able to improve significantly upon conda's performances. (conda's slowness in package resolution with large channels was our main motivation for working on mamba).
  • It works cross-platform on Windows, OS X and Linux. (@wolfv did the windows port of libsolv)
  • It performs full SAT solving to find optimal dependency combinations and if it does not succeed, it returns actionable hints for conflict resolution.
@pfmoore

This comment has been minimized.

Copy link
Member

@pfmoore pfmoore commented Nov 25, 2019

The proposed approach is to adopt the libsolv C library

There would need to be a fallback for platforms that libsolv does not support (for example, AIX support in pip is actively being worked on, and you didn't mention BSD). So libsolv as a performance option where available is plausible to me, but we're not really in a position to only use it. (Is there a pure Python version of libsolve, i.e. something that gives the same results, just slower?)

Also, how would get-pip.py work? Would we have to include binaries for libsolv for all possible platforms? Again, I'd assume not, we'd use a pure-python fallback.

Not being able to use external C code has been an annoyance for pip for a long time now. I'd like to see a good solution for it, but (a) I'm not sure there is one (short of some form of "mini-pip" bootstrapper solution that allows us to de-vendor altogether) and (b) it's a big enough piece of work that I'd hate the new resolver to depend on it.

@wolfv

This comment has been minimized.

Copy link

@wolfv wolfv commented Nov 25, 2019

Hi @pfmoore

I think additional platform support shouldn't be that hard to achieve as libsolv is fairly straight-forward C code. I am pretty sure that there is no pure Python version of libsolv, though (which I understand is a drawback, but there is also no pure Python version of Python, or the Python standard lib, so in my mind it shouldn't be a blocker).

I think for bootstrapping, one could have a pure Python pip that uses the current mechanism for resolving, which then installs the necessary resolver lib based on libsolv. E.g. one could pin the exact package of libsolv + pip-specific Python bindings, and install them from the boostrap-pip as you describe. It sounds totally doable to me, but you might know better what would be involved ...

@pfmoore

This comment has been minimized.

Copy link
Member

@pfmoore pfmoore commented Nov 25, 2019

I think additional platform support shouldn't be that hard to achieve

To be clear, I've no vested interest in the more niche platforms, I just think we have to be very clear if we're impacting what platforms pip is supported on (which at the moment is basically "anything that can run Python"). There's also the deployment question of how we ship a C extension (as pip is currently shipped as a "universal" wheel, and that's important for some use cases like get-pip.py)

I think for bootstrapping, one could have a pure Python pip that uses the current mechanism for resolving

I've argued above, and I'll repeat it here, I don't really want to have to maintain two resolvers in pip indefinitely.

But I don't want to be too negative about the proposal - I just wanted to flag some of the reasons we don't currently allow dependencies on C extensions, in case you weren't aware of them.

@uranusjr

This comment has been minimized.

Copy link
Member

@uranusjr uranusjr commented Nov 25, 2019

The discussion is probably better suited elsewhere, and might be totally redundant when/if more detail is revealed, but I really want to let my questions out now.

From my understanding, libsolv uses a fully SAT solver for dependency resolution, and requires loading in dependency information first before it starts solving. But PyPI as of now stores dependency metadata per-package. Even if you ignore setup.py’s runtime-dependent nature, it’d be difficult to efficiently fetch the information needed for the SAT solver.

How do you plan to handle this problem? Do you plan to implement infrastructure (and propose additional spec for third-party repo implementations) to generate .solv files when packages are uploaded? Or do you have some stragegies up your sleeves generathing appropriate solvable data as the solver goes, and maybe implement some backtracking as new dependency data comes in?

I am not aware of any existing work in this regard, and most resources I can find suggest the Python packaging landscape need something else/more than a straight-up SAT solver. So I’m very interested in any possibilities here.

@mlschroe

This comment has been minimized.

Copy link

@mlschroe mlschroe commented Nov 26, 2019

Those .solv files are just for caching, libsolv doesn't need them for solving. But I do agree that the dynamic nature of PyPI's dependencies makes it hard to use a SAT solver.

(See https://docs.google.com/document/d/1x_VrNtXCup75qA3glDd2fQOB2TakldwjKZ6pXaAjAfg/edit for more information)

(Note that a SAT solver is just a backtracking solver that also does clause learning if it runs into a conflict, so I think it is possible to use a SAT solver for PyPI. But it needs to be a solver that allows to dynamically add clauses while solving.)

@techalchemy

This comment has been minimized.

Copy link
Member

@techalchemy techalchemy commented Nov 26, 2019

Sat solvers have some significant capabilities these days including dynamic additions, restarts, backtracking, random restarts, etc. But I think the challenges here are partly going to be technical ones related to the need to support platforms where there is no guarantee that you can build a C-based solver.

@pradyunsg

This comment has been minimized.

Copy link
Member Author

@pradyunsg pradyunsg commented Nov 27, 2019

I'm at the airport right now so I can't respond to the points that have been raised right now but... I'm sure we shouldn't be discussing the technical choices/tradeoffs here -- this is more scoped to how we communicate and manage the rollout vs what we rollout. :)

I filed #7406 for further discussion on the technical trade-offs -- @sdispater, @techalchemy, @uranusjr, @wolfv I'd appreciate if we could have the further discussion around the various choices for the resolver design.

To set expectations early, I'm gonna be traveling for the next 2 weeks and hopefully will be able to catch up with all the discussion on ~Dec 9th.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.