Latest x packages breaks pinned version functionality

[miicha]

My usecase

• I'd like to have a slim mirror for offline usage based on a selection of a few packages
• packages are initially given in a requirmentss.txt and externally expanded to a requirements.txt with all dependencies (including version range)
• To keep the required storage low, I'd like to fetch max X versions while honoring the pinned versions given in the requirements.txt

Problem

Because all filters are applied in a loop in package.py, the last X versions plugin wins and older required versions are not downloaded:

if not all(plugin.filter(release_data) for plugin in release_filters):
    del self.releases[version]

Possible solution

• The pinned filter should have priority and "pre-filter" the available versions
• the next step is not yet clear to me, but I can see 3 options:
  1. afterwards we can either filter the remaining versions to keep only X, but this is problematic because we don't know if different packages have different version ranges since we would need to keep at least one version from each range
  2. alternatively last X version filtering might be suppressed for packages that have pinned versions / version range leading to more packages to be downloaded
  3. the pinned versions filtering itself takes care of fetching only the newest version from each given range (would be my favorite solution)

Since I need this functionality quite urgently, I would create a "work in progress" PR with solution 2. If you have some additional ideas or requirements, I would adjust it accordingly.

[cooperlees]

Thanks for proposing and looking into this. We've never prioritized getting plugin's ordering correct so we apply correctly as it really increases the complexity of the plugin system, as you're seeing.

I like the sounds of checking if there are specific per package version ranges etc. and exempting them from the keep option.

If you could prove this all works with unit tests it will increase my ability to merge this efficiently. I'd even accept adding this to the integration tests we run too.

• that is: https://github.com/pypa/bandersnatch/blob/main/test_runner.py when TOXENV=INTEGRATION is set ...