Switch to depend on requests library

[peterbe]

I have had more than one problem with the built-in urlopen() to download certain URLs.
This one for example fails on some SSL trouble in urlopen but opens nicely with requests from my OSX laptop.

I thought Python 3 was supposed to have requests built in so maybe this is only necessary for Python 2.

[peterbe]

What do you think @graingert @elidickinson @ionelmc ?

[graingert]

python3 doesn't have requests built in.

[peterbe]

python3 doesn't have requests built in.

So that was just a rumor I heard then. Well, it makes it easier to just depend on the new external dependency.

[graingert]

I think you should go for something like http://weasyprint.org/docs/tutorial/#url-fetchers

[peterbe]

Interesting but I'm not sure I follow. How is that better/different than requests?

[graingert]

Their default implementation looks pretty robust:

https://github.com/Kozea/WeasyPrint/blob/2d7f68e8a17821374d3974ea987710eecae586ad/weasyprint/urls.py#L241

And users of this library can override it as they see fit. Eg for my site I checked for

def my_fetcher(url):
    if url.stats_with('/static/'):
        return django_static_file_finder(urlparse(url).path)
    else:
        return weasyprint.default_url_fetcher(url)

weasyprint.HTML(doc, url_fetcher=my_fetcher)

[graingert]

You could also create a requests implementation:

try:
    import requests
except ImportError:
    def _default_url_fetcher(url):
        ...
else:
    def _default_url_fetcher(url):
        ...
        fileobj = session.get(url, stream=True).raw
        ...


# For autocomplete and static analysis etc
def default_url_fetcher(url):
    return _default_url_fetcher(url)


def premailer(doc, url_fetcher=default_url_fetcher):
    ...

[graingert]

I'd also add an argument in for a HTTP 'Accept' header so for:

 <link rel="stylesheet" type="text/css" href="https://example.com/theme.css">

You'd call:

default_url_fetcher(url='https://example.com/theme.css', mimetype='text/css')

[ionelmc]

I think it's fine having some default_url_fetcher abstraction in case someone wants to use something else than requests.

[peterbe]

But why do you need all of that? What's wrong with simply doing requests.get('https://example.com/theme.css').text. That'll get the CSS payload as a unicode string. And if the response is gzipped, requests will take care of that.

[graingert]

Accept header is nothing to do with gzip. The servers can return different documents depending on the Accept header

[peterbe]

But is that likely? Have you had problems with doing urlopen('some.com/url.css') and gotten the wrong content because of headers? Note, in premailer we only use urlopen to download external stylesheets.

(Sorry if I sound like I'm "fighting". I'm just trying to learn. And keep things simple :) at the same time)

[graingert]

It's just part of the http spec. I've seen some sites render their entire page off of '/' using the accept header

[peterbe]

Haha! Well, they're nuts! :)

[graingert]

What was the consensus on this?

[peterbe]

My view is that anything overly complex is going to mean more code to maintain. A switch to relying on requests is straightforward and it's an established and good package. It just swaps out the urllib.urlopen for a different opener and that's about it. Doesn't try to solve other problems as well.

[alexphelps]

I just ran into this bug today. This is the error when using an SSL url for the css file. The only fix I could find was to not use ssl on the url which makes configuring my app harder because all other pages, files, etc need ssl.

URLError: <urlopen error [Errno 1] _ssl.c:510: error:14077438:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert internal error>

[peterbe]

@alexphelps I bet if you take that same URL (the https one) and try to do requests.get(that_url) you'll find that it works.
@graingert is working on a fancy wrapper API on top of requests.get but if he doesn't have time, which is fine, we'll just replace urllib with requests.

[p12tic]

@peterbe: I think this issue can be closed.