Planning for breaking changes on TRFKA1.17

[haikuginger]

So, we're taking a step back and considering/refactoring all the breaking changes we want to make in the near future that'll break current API contracts on some level. This issue is basically a dump of my thoughts/plans so far, which are open for discussion to all comers.

• Method of SSL/TLS verification: There's been an interest in having platform-independent hostname verification pre-enabled for SessionManager. From talking to @Lukasa, there are two main ways to go about this - first, to either import or package certifi, or to require PyOpenSSL, certitude, and other binary packages to be able to reliably use the platform cert bundles. At this point, certifi is a more practical option, so we basically need to decide if we want to bundle it or require it.
• Scope of SSL/TLS verification: @shazow has indicated that he'd like autoconfig on SessionManager, as a high-level configuration object. Personally, I think that if we're putting in the effort to set up a secure default, all of our connection levels should be able to benefit from it - including ConnectionPool and PoolManager. This is up for discussion, and brings us to...
• Settings handling in general: I'd like to unify all sorts of autoconfiguration code onto a Settings object that can be passed into any RequestMethods subclass. This would include things like the SSLContext to be used, any platform-specific flags that need to be considered (e.g., GAE), or anything else. My vision would be to have any newly-created Settings object carry smart defaults, and any newly-created RequestMethods subclass get either a Settings object in arguments or one of a fresh instantiation of same or a link to a urllib3-wide Settings object. It'd look something like this:

import urllib3

# We can call like this...
sm = urllib3.SessionManager()

# Or like this...
insecure_settings = urllib3.Settings(verify_ssl=False)
sm = urlib3.SessionManager(settings=dumb_settings)

# Or you could even do this!
class InsecurePoolManager(urllib3.PoolManager):

    def __init__(self, *args, **kwargs):
        insecure_settings = Settings(verify_ssl=False)
        kwargs[settings] = insecure_settings
        super(InsecurePoolManager, self).__init__(*args, **kwargs)

• While we're on RequestMethods, support will be added for explicitly-passed params and body fields. (Expand RequestMethods optional arguments #885)
• SessionManager will get a rewrite to be more flexible in general. The class will have a default list of ContextHandlers, which at first will include just CookieHandler, but when instantiating or subclassing SessionManager, an alternate list can be provided. Each request/response pair will be passed to each handler in the list for mutation and data extraction. And we're going to include a bonus BasicAuthHandler that people can use if they want. This will also solve for SessionManager: Ability to disable cookies #888, as a list without a CookieHandler can be passed to __init__().

import urllib3
from urllib3.context_handlers import CookieHandler, BasicAuthHandler

basic_auth = BasicAuthHandler('mysite.com', 'username', 'password')
cookies = CookieHandler()
handlers = [cookies, basic_auth]
sm = urllib3.SessionManager(handlers=handlers)

• I think I'm going to cut the link between our Request object and urllib.request.Request. We'll mimic some small features of the interface so that cookielib can still handle our version, but the main goal will be to have data stored in a way that we can natively expand into urlopen(). This will also make for a cleaner API for additional ContextHandlers to work with.

[haikuginger]

Oh, @Lukasa, do we want to make the WHATWG parser part of this? As long as we're swinging the hammer around anyway.

[Lukasa]

Eh, I'm ok with leaving that out.

[haikuginger]

A bunch of these items are addressed in new commits on #849 and in #903, and are ready for review. SSL/TLS verification and the proposed Settings object are tied together, so I'm waiting until we have a discussion on what the best implementation looks like to proceed with those.

[IvanLauLinTiong]

pyOpenSSL is deprecated and will be removed in future release version 2.x (#2691).