Give the user (optional) control over the backtracking behaviour via timeout

[potiuk]

What's the problem this feature will solve?

As discussed in #10258 (which is about presenting conflict as they appear) to split-off the discussion I start a new issue as proposed by @pradyunsg.

The feature request here is a proposal to give the user control over the backtracking process via timeout, especially in case it goes into a long backtracking loop. There are some issues describing such backtracking loop happening, for example #10924 and discussing potential changes in the way how algorithm work (for example #10884, #10788).

There are also similar discussions and proposals (for example #10417, #10235) but none of them (as far as I see) proposed timeout as a solution.

The problem this proposal is supposed to handle is to handle the case of users using pip to find and resolve installation of packages in CI environments. Many of the users use pip to aither install packages in CI or build container images (example Apache Airflow but there are plenty of others) and they would like to be able to determine the non-conflicting set of dependencies using pip resolver. This is useful if you build libraries or applications where you have "open" set of requirements and you want to make sure that your users are able to continue installing your application or library so that their pip install commend will complete in reasonable time.

In this case what the users of pip need is to use pip resolver for the job it is designed - to find the right set of dependencies fulfilling the limitations of packages to install, but they also need ot be able to diagnoes and fix the cases when pip enters the backtrack loop and it takes much more time than usual.

In such case the users should be able to stop backtracking automatically (CI systems are designed to run in unattended way) and get information that will enable them to inspect the root cause of the conflicts that lead to backtracking.

Currently even if such backtracking is stoped at a timeout, the logs of pip do not contain enough information to figure out what was the root of the conflict that led to backtracking (details in #10258 (comment))

Describe the solution you'd like

What I would like to see is:

• Add --resolver-timeout (for example) flag to pip install that will fail resolver if the time limit is reached
• Add diagnostics information printed by pip when such timeout happens pointing to the (probable if it cannot be assesed exactly) cause for the too-long-backtracking.

Alternative Solutions

• Limiting the number of backtracking similar to Max Backtracking Option and print out current failure casues #10417.

The problem with this one is that users of pip have no idea of the backtracking algorithm and backtracking depth. See "additional context" below for explanation.

• Solving it using external timeout wthout internal information from pip

The problem cannot be easily solved (IMHO) without pip providing the right diagnostics information. In Apache Airflow we've implemented some ways to get more information when such backtracking happens (and we added our own timeout), but the investigation after such capturing are based mostly on guessing. We are looking on what has changed since the last succesful run (which we store in constraints) and provide some clies to a process to manually figure out the root cause. This is however based on trial and error, and this is mostly based on gueses. Information provided by pip itself could be much more helpful and could lead to much less time (and energy) consuming solution.

Additional context

I perfectly understand that pip cannot decide upfront on whether to print the context , and yeah - it is an extremely complex problem to solve, so I also sympathise with pip maintainers here.

But if the algorithm cannot decide, let the user decide when they want to stop and how much of backtracking to do and when "too long" or "too much" of the backtracking to do. Simply letting it running for an indefinite amount of time when things go wrong is the "worst" UI choice from the user perspective. Because the user cannot do anything about it when it happens.

Most of the users do not understand the complexity of the resolution algorithm and reasons for backtracking and some of the details that pip maintainers know initimately. From the users perspective, the users only know that suddenly a) backtracking happens (but they might not understand why and even what backtracking is) and b) pip starts to download awfully lot of data c) it takes whole lot longer for the resolver to run and it might even never complete.

Out of those three observations of the users, the "time" is the easiest one to reason about from the user perspective. Their observation is that suddenly the install command they run take a lot longer than normal (and they know from the past what the "normal" is. So their reaction might be to add the timeout parameter and rerun the failed job for investigation. This is unlike to backtracking depth - where users would have to first understand all the details of the algorithm before understanding what limit they should put to investigate.

While I understand that it's extremely "hard" to decide when "long" is "too long" in generic case (this is the problem pip maintainers have and I understand it is difficult), delegating it to the user (if they choose to) is the next best thing that can be done. Most of the users do not have "generic" problem to solve. They just want to make sure their particular command works - and if they use it in CI context especially, this is much "narrower" problem space (because they repetitively run pretty much the same command and they expect similar output) and they can make much better decisions for their particular case than the "generic" solution and they can decide on their own when "long" is "too long". And then they should get the right diagnostic information if they hit the limit.

This is what I propose - give the users (those who need it - should be optional) the ability to make their own decision when "long" is "too long" for their particular case, and print diagnostics information if this limit is hit.

I think not all decisions have to be made by the algorithm and pip maintainers - some of those can simply be delegated to users who need it and know what they are doing and what the "usual" time for resolution is.

Code of Conduct

• I agree to follow the PSF Code of Conduct.

[pfmoore]

This has been discussed a number of times in the past. Generally, there's been a lot of ideas about what "might work", but the problem is translating them into code that implements those ideas in the context of how the resolver actually works. The best way to make progress here is probably for someone to provide an implementation (in the form of a PR) of a particular proposal, and we can then discuss the trade-offs based on something concrete.

If anyone in the community wants to step up and produce an implementation, that would be great (although just to be clear, there's no promise that just because someone writes a PR, it'll automatically be accepted - a PR is simply the first stage in getting something agreed).

[potiuk]

If anyone in the community wants to step up and produce an implementation, that would be great (although just to be clear, there's no promise that just because someone writes a PR, it'll automatically be accepted - a PR is simply the first stage in getting something agreed).

I am willing to take a look but before attempting to learn how resolver works internally, can the maintainers confirm that the general direction is ok ? I understand that there are some rules to follow and the code has to be good etc. But the idea here is rather simple to explain without attempting to write the code:

• configurable timeout on resolver as a flag
• printing disagnostic when the timeout happens

Is this possible to confirm that the general idea is a good direction (assuming it passes the rewiews, tests, is of good quality and meets all other expectations of reviewers) ?

[potiuk]

The reason I am asking is that, it requires quite some effort to deep dive and understand how resolver works and I would like to know that this effort is not really done in vain from the beginning.

[pfmoore]

I am willing to take a look but before attempting to learn how resolver works internally, can the maintainers confirm that the general direction is ok?

Personally, no, I can't give that sort of commitment. It depends on what the implementation looks like, how it works in practice, etc. All I'm willing to say at this point is that I'm not interested in further discussing what "might" be a solution, if no-one is willing to put in the effort to do the practical work of delivering actual code.

[potiuk]

Personally, no, I can't give that sort of commitment. It depends on what the implementation looks like, how it works in practice, etc. All I'm willing to say at this point is that I'm not interested in further discussing what "might" be a solution, if no-one is willing to put in the effort to do the practical work of delivering actual code.

Very disheartening statement it is. Basically this means that you ask people to spend their time and reserve the right to tell at the end that what they worked on was not acceptable to begin with as an idea ? Am I reading it right?

[pfmoore]

I'm pointing out that someone is going to have to write some code at some point, and I no longer have any energy for the discussion in the absence of code. So please don't feel compelled to spend time if you don't think it's worthwhile, but conversely please don't expect me to give my blessing to a solution that offers no benefit to me personally, and which has triggered extremely long debates that would take a big chunk of my time to review, with no certainty that there's a consensus anyway.

[potiuk]

All i am asking for is reaching consensus for the idea specified here from maintainer team of pip not necessary yours @pfmoore. If you have no time and energy for that, maybe others might have time for it.

Maybe consensus can be reached without You personally since you have no time and energy ? Maybe you personally do not have to be involved in all discussions in pip that solve problems for your users and other maintainers can have more time and energy to confirm the direction is fine ? And please don't feel obliged to respond - maybe others can?

I am not sure if that's the case here, but If everything that happens in pip has to be blessed by one person that has no time and energy to discuss and reach consensus, then that single person becomes a single point of failure and obstruction, and blocks development and scalability of the project? Maybe it's time to think to increase the team of maintainers and let more people take decisions? I don't know of course the governance model of pip, so I might be wrong in the assessment, but this looks awfully like that.

And please don't treat it as personal attack. Seems that in the discussions here it happens that there are needless emotions and being personally 'hurt' where this is purely about some frictions in the process.

I am just asking whether we can agree that the direction is good to solve a very real problem of the users. Yes you stated clearly the problem is not yours, but IMHO, he only reason pip exists are the users of pip. At least that is my experience as a project maintainer, that problems of my users are rarely 'my' problems.

Yet if maintainers don't even provide the guidance and decision making for the users who want to help and donate their own time to solve their own problems, this does not seem like problems are being addressed at all. And raises the questions if this is really best approach.

But regardless of that, i think I might spend a little time on it in the coming days and weeks, just to see and learn and maybe i will provide some PR to discuss it (i have about 40 users waiting for my help in guidance and directions and reviews in my own project and i do not want to let them down). No promise from my side - we have a workable workaround for now so no hurry for that, maybe indeed in the meantime someone will be happy to both donate their time and trust that the direction is right to solve real problem of pip without hearing from pip maintainers.

I think the chances for that are slim and such approach might lead to 'stalling' of such a feature proposal without actually rejecting it and reasoning why. But i hope i am wrong about that.