Implement backjumping over unrelated states

[bennati]

The current backtracking logic assumes that the last state in the stack
is the state that caused the incompatibility. This is not always the case.
For example, given three requirements A, B and C, with dependencies
A1, B1 and C1, where A1 and B1 are incompatible.
The requirements are processed one after the other, so the last state
is related to C, while the incompatibility is caused by B.

The current behavior causes significant slowdowns in case there are
many candidates for B and C, as all their combination are evaluated
before a compatible version of B can be found.

The new behavior discards a state if the packages that cause the
incompatibility are not found among the direct dependencies
of the candidate in the current state.
In our example, this causes the state related to C to be dropped
without evaluating any of its candidates, until a compatible candidate
of B is found.

This change can be tested with the following requirements:

boto3==1.10.16
s3fs
seaborn

[pombredanne]

This is an awesome finding!

[pombredanne]

@bennati Could you add a test?

[bennati]

I realized the error is something else entirely, please review again :)

[pradyunsg]

If the change does what the message is implying, I'd prefer that we rename the function to _backjump :)

[bennati]

Why do you want to rename the function? The function is not jumping anywhere, it is backtracking states in order until it reaches the correct state.

[bennati]

@pombredanne can you elaborate on what kind test you would like to have?
Basing the test on the time it takes to process a requirements file does not seem ideal.

[pradyunsg]

https://en.m.wikipedia.org/wiki/Backjumping is why I've suggested the name that I suggested. :)

[bennati]

The Lint CI fails due to "unused" imports and variables, but removing them causes the next step in the CI (mypy) to fail because the same imports and variables are not defined.
Not sure how to solve these issues.

[uranusjr]

You can use TYPE_CHECKING to conditionally import those, so they are only imported for type checking.

[pradyunsg]

Actually, @notatallshaw the issue you're reporting here seems to have been introduced in 0.9.0 -- I've just reproduced that in pip + resolvelib 0.9.0 in addition to this branch. My understanding is that #111 is where we introduced the change (which is trimming information aggresively).

[pradyunsg]

If this is not too much to ask making a release candidate would help a lot testing downstream.

Yes, and, you can do pip install https://github.com/sarugaku/resolvelib/archive/refs/heads/main.zip if you really want the latest and greatest. :)

[notatallshaw]

Actually, @notatallshaw the issue you're reporting here seems to have been introduced in 0.9.0 -- I've just reproduced that in pip + resolvelib 0.9.0 in addition to this branch. My understanding is that #111 is where we introduced the change (which is trimming information aggresively).

Agreed, I've opened a new issue with a reproducible set of steps: #120

[frostming]

Hi, @pradyunsg @uranusjr do you have any preference on when a new release will be out with this fix?

[uranusjr]

Since the associated tests are done I think I’ll cut a release. I think this is also a good chance to go 1.0 as well? I’ll wait 24 hours in case anyone objects.

[pradyunsg]

Go ahead and cut the release. :)

[uranusjr]

Still a few hours to 24 but I’ll call it.

[notatallshaw]

FYI, I've raised a PR to remove backjumping: #142