Ensure pip-tools unions dependencies of multiple declarations of a package with different extras

[richafrank]

Fixes #1451
Fixes #852

Fix for #1451 that uses the same simplification in #1452 but handles the case of combining editables (that lack names) as well, e.g. test_editable_package_constraint_without_non_editable_duplicate.

This reverts the original solution for the test case above and replaces it. The original solution was to prepare the nameless ireqs early, so that they have a name. However:

    Preparing the ireq is side-effect-ful and can only be done once for an
    instance, so we use a proxy instead. combine_install_requirements may
    use the given ireq as a template for its aggregate result, mutating it
    further by combining extras, etc. In that situation, we don't want that
    aggregate ireq to be prepared prior to mutation, since its dependencies
    will be frozen with those of only a subset of extras.

    i.e. We both want its name early (via preparation), but we also need to
    prepare it after any mutation for combination purposes. So we use a
    proxy here for the early preparation.

This avoids issues with the ireq instances and the cache needing updates after mutating the extras for the combined_ireq, specifically #1451 . It means an extra preparation step for ireqs lacking names (preparing the proxy in addition to the ireq itself eventually), but that seems worthwhile (in my opinion).

Contributor checklist

• Provided the tests for the changes.
• Assure PR title is short, clear, and good to be included in the user-oriented changelog

Maintainer checklist

• Assure one of these labels is present: backwards incompatible, feature, enhancement, deprecation, bug, dependency, docs or skip-changelog as they determine changelog listing.
• Assign the PR to an existing or new milestone for the target version (following Semantic Versioning).

[AndydeCleyre]

I'm not sure if it would be expected, but I notice that this does not resolve #852, which is reproducible as (courtesy of @atugushev):

$ printf '%s\n' '-e git+https://github.com/edx/django-rest-framework-oauth.git@0a43e8525f1e3048efe4bc70c03de308a277197c#egg=djangorestframework-oauth==1.1.1' 'edx-enterprise==1.7.2' >requirements.in
$ seq 1 10 | while read i; do echo "---- $i ----" && pip-compile -rqo- requirements.in | grep '^edx-opaque-keys'; done
---- 1 ----
edx-opaque-keys[django]==1.0.1
---- 2 ----
edx-opaque-keys==1.0.1
---- 3 ----
edx-opaque-keys==1.0.1
---- 4 ----
edx-opaque-keys[django]==1.0.1
---- 5 ----
edx-opaque-keys==1.0.1
---- 6 ----
edx-opaque-keys[django]==1.0.1
---- 7 ----
edx-opaque-keys[django]==1.0.1
---- 8 ----
edx-opaque-keys==1.0.1
---- 9 ----
edx-opaque-keys==1.0.1
---- 10 ----
edx-opaque-keys==1.0.1

[richafrank]

I'm not sure if it would be expected, but I notice that this does not resolve #852

Ah, interesting! Looks like related code, but a separate issue. I think I have a fix, but it might take a little time to formulate a good unit test (that doesn't require all of django). I can work on another PR for that.

[richafrank]

@AndydeCleyre Since this is still open, I pushed the fix for #852 here.

[AndydeCleyre]

Thank you, this is greatly appreciated!

[AndydeCleyre]

Can you say whether it would, or why it would not, make sense to integrate the new prep-proxy-to-get-name-to-get-key behavior in our utils.key_from_ireq function?

[richafrank]

Can you say whether it would, or why it would not, make sense to integrate the new prep-proxy-to-get-name-to-get-key behavior in our utils.key_from_ireq function?

Great question. IMO based on the general intent of key_from_ireq, I think it probably does make sense to integrate this behavior there. I imagine some bugs (reported and otherwise) would get squashed. A few considerations though:

1. Preparing the proxy requires a piptools.repositories.base.BaseRepository instance and means a new dependency for everything that calls key_from_ireq.
2. key_from_ireq is very low level (as per its location in utils), and as I look at updating it to accept a BaseRepository argument, many, many call sites (across many modules) need updating. This isn't a tiny change, so while this change may still prove beneficial, I wouldn't recommend including it as part of this bug fix, if the two can be delivered incrementally (which I think they can).
3. I think I have most of the compile tests passing with this change, but the sync code paths don't already offer a BaseRepository instance to pass around, so we might need to discuss the best approach there.
4. Because of the breadth of the key_from_ireq call sites, all these changes should definitely get many eyes on them. I'm certainly not an expert on all of them.
5. We would need to assess whether each call site requires this new behavior and dependency, but also whether there are any speed concerns that require caching, like I included in the one call site I have here, and what the lifetime of that cache and its elements are. (That's a great question for reviewers of this PR too.)

Let me know what you think.

[AndydeCleyre]

@richafrank Thanks, you've articulated everything perfectly. I'm approving this now.

@atugushev When you can please have a look at this and the above comment.

[AndydeCleyre]

This is looking great and good to go as far as I'm concerned, with the note that we should figure out what to do about key_from_ireq as per the discussion, as a follow up task. This is an important fix, thank you!

[atugushev]

@richafrank, awesome fix! 🎉

[atugushev]

Could you split each test block into different tests or parametrize tests if possible?

[richafrank]

Sounds good. I split it into two tests.

[atugushev]

When you can please have a look at this and the above comment.

@AndydeCleyre Nice thoughts about key_from_ireq improvement. Let's discuss it in a separate issue 👍🏻

[atugushev]

@richafrank I've updated the PR's description to link fixed issues. Could you check please if everything is correct?

[richafrank]

I've updated the PR's description to link fixed issues. Could you check please if everything is correct?

Yep, looks good to me.