[WIP] Pep 503 Sans I/O

[dstufft]

This is a fork of #87 rewritten to use Sans I/O techniques instead. This creates an API where Repository.fetch() returns a fetcher object which looks something like:

class Fetcher:
    finished = False  # Whether the fetcher has gotten all of the files it needs or not

    def pending_requests(self):
        """
        Return an iterable of requests that need to be completed
        """

    def add_response(self, req, content, headers=None):
        """
        Send a response to one of our requests back into the fetcher.
        """

    def get_files(self):
        """
        Return a list of files that have been found, if the fetcher isn't yet finished it raises an error.
        """

This includes a utility function that allows you to implement a Fetcher using a generator like:

from packaging.repositories._utils import as_fetcher

@as_fetcher
def fetch(project):
    resp = yield Request(url="...")
    return []    # This is 3.3+ only syntax, will need something else in it's place for 2.x

The way you use this API is with something like:

import requests

from packaging.repositories import (
    ProjectFilter, FormatFilter,
    FilteredRepository, MultiRepository,
    SimpleRepository,
)


repo = FilteredRepository(
    MultiRepository(repositories=[
        # SimpleRepository("https://pypi.example.com/"),
        FilteredRepository(
            SimpleRepository("https://pypi.org/simple/"),
            ProjectFilter("!myinternallibrary"),
        ),
    ]),
    FormatFilter(wheel=True, sdist=False),
)


fetcher = repo.fetch("pip")

while not fetcher.finished:
    for req in fetcher.pending_requests():
        resp = requests.get(req.url)
        resp.raise_for_status()
        fetcher.add_response(req, resp.content, headers=resp.headers)

files = fetcher.get_files()

However, that's a sort of manual way of doing it, and it wouldn't be very hard to write a small wrapper that combined the Sans I/O style Repository class with an explicit transport to remove the need to drive the I/O directly, using something like:

import attr
import requests

from packaging.repositories import (
    ProjectFilter, FormatFilter,
    FilteredRepository, MultiRepository,
    SimpleRepository,
)


@attr.s(cmp=False, slots=True)
class RequestsRepositoryHelper:

    repository = attr.ib()
    session = attr.ib(default=attr.Factory(requests.session), repr=False)

    def fetch(self, project):
        fetcher = repo.fetch("pip")

        while not fetcher.finished:
            for req in fetcher.pending_requests():
                resp = requests.get(req.url)
                resp.raise_for_status()
                fetcher.add_response(req, resp.content, headers=resp.headers)

        return fetcher.get_files()


repo = RequestsRepositoryHelper(
    FilteredRepository(
        MultiRepository(repositories=[
            # SimpleRepository("https://pypi.example.com/"),
            FilteredRepository(
                SimpleRepository("https://pypi.org/simple/"),
                ProjectFilter("!myinternallibrary"),
            ),
        ]),
        FormatFilter(wheel=True, sdist=False),
    )
)

files = repo.fetch("pip")

You can see the delta from just the Deferred style API by looking at dee9f15.

[pradyunsg]

@dstufft Is this still on the roadmap?

[brettcannon]

I'm going to close this as a bit outdated. 😉 There is also a parser in mousebender which we can upstream if people want (see #424 ).