Merging new release version: 1.9.1

CHANGES.rst

@@ -1,6 +1,27 @@
 Changes
 =======
 
+1.9.1 (2014-09-13)
+++++++++++++++++++
+
+* Apply socket arguments before binding. (Issue #427)
+
+* More careful checks if fp-like object is closed. (Issue #435)
+
+* Fixed packaging issues of some development-related files not
+  getting included. (Issue #440)
+
+* Allow performing *only* fingerprint verification. (Issue #444)
+
+* Emit ``SecurityWarning`` if system clock is waaay off. (Issue #445)
+
+* Fixed PyOpenSSL compatibility with PyPy. (Issue #450)
+
+* Fixed ``BrokenPipeError`` and ``ConnectionError`` handling in Py3.
+  (Issue #443)
+
+
+
 1.9 (2014-07-04)
 ++++++++++++++++
 

CONTRIBUTORS.txt

@@ -117,5 +117,15 @@ In chronological order:
 * Arthur Grunseid <http://grunseid.com>
   * source_address support and tests (with https://github.com/bui)
 
+* Ian Cordasco <graffatcolmingov@gmail.com>
+  * PEP8 Compliance and Linting
+  * Add ability to pass socket options to an HTTP Connection
+
+* Erik Tollerud <erik.tollerud@gmail.com>
+  * Support for standard library io module.
+
+* Krishna Prasad <kprasad.iitd@gmail.com>
+  * Google App Engine documentation
+
 * [Your name or handle] <[email or website]>
   * [Brief summary of your changes]

MANIFEST.in

@@ -1,2 +1,5 @@
 include README.rst CHANGES.rst LICENSE.txt CONTRIBUTORS.txt dev-requirements.txt Makefile
-recursive-include dummyserver *.*
+recursive-include dummyserver *
+recursive-include test *
+recursive-include docs *
+recursive-exclude docs/_build *

Makefile

@@ -30,6 +30,7 @@ clean:
 	find . -name "*.py[oc]" -delete
 	find . -name "__pycache__" -delete
 	rm -f $(REQUIREMENTS_OUT)
+	rm -rf docs/_build
 
 test: requirements
 	nosetests

README.rst

@@ -5,6 +5,9 @@ urllib3
 .. image:: https://travis-ci.org/shazow/urllib3.png?branch=master
         :target: https://travis-ci.org/shazow/urllib3
 
+.. image:: https://www.bountysource.com/badge/tracker?tracker_id=192525
+        :target: https://www.bountysource.com/trackers/192525-urllib3?utm_source=192525&utm_medium=shield&utm_campaign=TRACKER_BADGE
+
 
 Highlights
 ==========

dev-requirements.txt

@@ -1,5 +1,7 @@
 nose==1.3.3
 mock==1.0.1
-tornado==3.2.2
 coverage==3.7.1
 tox==1.7.1
+
+# Tornado 3.2.2 makes our tests flaky, so we stick with 3.1
+tornado==3.1.1

docs/index.rst

@@ -34,7 +34,7 @@ Highlights
 
 - Tested on Python 2.6+ and Python 3.2+, 100% unit test coverage.
 
-- Works with AppEngine, gevent, and eventlib.
+- Works with AppEngine, gevent, eventlib, and the standard library :mod:`io` module.
 
 - Small and easy to understand codebase perfect for extending and building upon.
   For a more comprehensive solution, have a look at
@@ -83,6 +83,20 @@ For more on making secure SSL/TLS HTTPS requests, read the :ref:`Security
 section <security>`.
 
 
+urllib3's responses respect the :mod:`io` framework from Python's
+standard library, allowing use of these standard objects for purposes
+like buffering:
+
+.. doctest ::
+
+    >>> http = urllib3.PoolManager()
+    >>> r = http.urlopen('GET','http://example.com/', preload_content=False)
+    >>> b = io.BufferedReader(r, 2048)
+    >>> firstpart = b.read(100)
+    >>> # ... your internet connection fails momentarily ...
+    >>> secondpart = b.read()
+
+
 Components
 ==========
 
@@ -196,7 +210,7 @@ set for the entire pool or per-request.
     >>> r = http.request('GET', 'http://httpbin.org/delay/1')
 
     >>> # Manager with 2 second timeout for the read phase, no limit for the rest.
-    >>> http = PoolManager(timeout=Timeout(read=2.0)) 
+    >>> http = PoolManager(timeout=Timeout(read=2.0))
     >>> r = http.request('GET', 'http://httpbin.org/delay/1')
 
     >>> # Manager with no timeout but a request with a timeout of 1 seconds for
@@ -296,7 +310,7 @@ benefits from this library.
   biggest impact for progress. Periods of  3 to 10 days allow a contributor to
   tackle substantial complex issues which are otherwise left to linger until
   somebody can't afford to not fix them.
-  
+
   Contact `@shazow <https://github.com/shazow>`_ to arrange a grant for a core
   contributor.
 

docs/security.rst

@@ -13,6 +13,41 @@ Luckily, it's not too hard to enable verified HTTPS requests and there are a
 few ways to do it.
 
 
+Python with SSL enabled
+-----------------------
+
+First we need to make sure your Python installation has SSL enabled. Easiest
+way to check is to simply open a Python shell and type `import ssl`::
+
+    >>> import ssl
+    Traceback (most recent call last):
+      ...
+    ImportError: No module named _ssl
+
+If you got an ``ImportError``, then your Python is not compiled with SSL support
+and you'll need to re-install it. Read
+`this StackOverflow thread <https://stackoverflow.com/questions/5128845/importerror-no-module-named-ssl>`_
+for details.
+
+Otherwise, if ``ssl`` imported cleanly, then we're ready to setup our certificates:
+:ref:`certifi-with-urllib3`.
+
+
+Enabling SSL on Google AppEngine
+++++++++++++++++++++++++++++++++
+
+If you're using Google App Engine, you'll need to add ``ssl`` as a library
+dependency to your yaml file, like this::
+
+    libraries:
+    - name: ssl
+      version: latest
+
+If it's still not working, you may need to enable billing on your account
+to `enable using sockets
+<https://developers.google.com/appengine/docs/python/sockets/>`_.
+
+
 .. _certifi-with-urllib3:
 
 Using Certifi with urllib3
@@ -102,7 +137,6 @@ Now you can continue using urllib3 as you normally would.
 For more details, check the :mod:`~urllib3.contrib.pyopenssl` module.
 
 
-
 InsecureRequestWarning
 ----------------------
 
@@ -125,4 +159,3 @@ you can use :func:`~urllib3.disable_warnings`::
     urllib3.disable_warnings()
 
 Making unverified HTTPS requests is strongly discouraged. ˙ ͜ʟ˙
-

test/contrib/test_pyopenssl.py

@@ -1,9 +1,15 @@
+from nose.plugins.skip import SkipTest
+from urllib3.packages import six
+
+if six.PY3:
+    raise SkipTest('Testing of PyOpenSSL disabled on PY3')
+
 try:
     from urllib3.contrib.pyopenssl import (inject_into_urllib3,
                                            extract_from_urllib3)
 except ImportError as e:
-    from nose.plugins.skip import SkipTest
-    raise SkipTest('Could not import pyopenssl: %r' % e)
+    raise SkipTest('Could not import PyOpenSSL: %r' % e)
+
 
 from ..with_dummyserver.test_https import TestHTTPS, TestHTTPS_TLSv1
 from ..with_dummyserver.test_socketlevel import TestSNI, TestSocketClosing

test/test_response.py

@@ -182,6 +182,37 @@ def test_io_bufferedreader(self):
         br.close()
         self.assertEqual(resp.closed, True)
 
+        b = b'fooandahalf'
+        fp = BytesIO(b)
+        resp = HTTPResponse(fp, preload_content=False)
+        br = BufferedReader(resp, 5)
+
+        br.read(1)  # sets up the buffer, reading 5
+        self.assertEqual(len(fp.read()), len(b) - 5)
+
+        # This is necessary to make sure the "no bytes left" part of `readinto`
+        # gets tested.
+        while not br.closed:
+            br.read(5)
+
+    def test_io_readinto(self):
+        # This test is necessary because in py2.6, `readinto` doesn't get called
+        # in `test_io_bufferedreader` like it does for all the other python
+        # versions.  Probably this is because the `io` module in py2.6 is an
+        # old version that has a different underlying implementation.
+
+
+        fp = BytesIO(b'foo')
+        resp = HTTPResponse(fp, preload_content=False)
+
+        barr = bytearray(3)
+        assert resp.readinto(barr) == 3
+        assert b'foo' == barr
+
+        # The reader should already be empty, so this should read nothing.
+        assert resp.readinto(barr) == 0
+        assert b'foo' == barr
+
     def test_streaming(self):
         fp = BytesIO(b'foo')
         resp = HTTPResponse(fp, preload_content=False)

test/test_util.py

@@ -21,6 +21,8 @@
     InsecureRequestWarning,
 )
 
+from urllib3.util import is_fp_closed
+
 from . import clear_warnings
 
 # This number represents a time in seconds, it doesn't mean anything in
@@ -324,3 +326,32 @@ def test_resolve_cert_reqs(self):
         self.assertEqual(resolve_cert_reqs('REQUIRED'), ssl.CERT_REQUIRED)
         self.assertEqual(resolve_cert_reqs('CERT_REQUIRED'), ssl.CERT_REQUIRED)
 
+    def test_is_fp_closed_object_supports_closed(self):
+        class ClosedFile(object):
+            @property
+            def closed(self):
+                return True
+
+        self.assertTrue(is_fp_closed(ClosedFile()))
+
+    def test_is_fp_closed_object_has_none_fp(self):
+        class NoneFpFile(object):
+            @property
+            def fp(self):
+                return None
+
+        self.assertTrue(is_fp_closed(NoneFpFile()))
+
+    def test_is_fp_closed_object_has_fp(self):
+        class FpFile(object):
+            @property
+            def fp(self):
+                return True
+
+        self.assertTrue(not is_fp_closed(FpFile()))
+
+    def test_is_fp_closed_object_has_neither_fp_nor_closed(self):
+        class NotReallyAFile(object):
+            pass
+
+        self.assertRaises(ValueError, is_fp_closed, NotReallyAFile())

test/with_dummyserver/test_connectionpool.py

@@ -595,7 +595,9 @@ def test_source_address(self):
             pool = HTTPConnectionPool(self.host, self.port,
                     source_address=addr, retries=False)
             r = pool.request('GET', '/source_address')
-            assert r.data == b(addr[0])
+            assert r.data == b(addr[0]), (
+                "expected the response to contain the source address {addr}, "
+                "but was {data}".format(data=r.data, addr=b(addr[0])))
 
     def test_source_address_error(self):
         for addr in INVALID_SOURCE_ADDRESSES:

test/with_dummyserver/test_https.py

@@ -1,3 +1,4 @@
+import datetime
 import logging
 import ssl
 import sys
@@ -20,12 +21,14 @@
 from urllib3.connection import (
     VerifiedHTTPSConnection,
     UnverifiedHTTPSConnection,
+    RECENT_DATE,
 )
 from urllib3.exceptions import (
     SSLError,
     ReadTimeoutError,
     ConnectTimeoutError,
     InsecureRequestWarning,
+    SystemTimeWarning,
 )
 from urllib3.util.timeout import Timeout
 
@@ -218,6 +221,23 @@ def test_assert_invalid_fingerprint(self):
         https_pool.assert_fingerprint = 'AA'
         self.assertRaises(SSLError, https_pool.request, 'GET', '/')
 
+    def test_verify_none_and_bad_fingerprint(self):
+        https_pool = HTTPSConnectionPool('127.0.0.1', self.port,
+                                         cert_reqs='CERT_NONE',
+                                         ca_certs=DEFAULT_CA_BAD)
+
+        https_pool.assert_fingerprint = 'AA:AA:AA:AA:AA:AAAA:AA:AAAA:AA:' \
+                                        'AA:AA:AA:AA:AA:AA:AA:AA:AA'
+        self.assertRaises(SSLError, https_pool.request, 'GET', '/')
+
+    def test_verify_none_and_good_fingerprint(self):
+        https_pool = HTTPSConnectionPool('127.0.0.1', self.port,
+                                         cert_reqs='CERT_NONE',
+                                         ca_certs=DEFAULT_CA_BAD)
+
+        https_pool.assert_fingerprint = 'CC:45:6A:90:82:F7FF:C0:8218:8e:' \
+                                        '7A:F2:8A:D7:1E:07:33:67:DE'
+        https_pool.request('GET', '/')
 
     @requires_network
     def test_https_timeout(self):
@@ -308,6 +328,27 @@ def test_enhanced_ssl_connection(self):
 
         https_pool._make_request(conn, 'GET', '/')
 
+    def test_ssl_correct_system_time(self):
+        with warnings.catch_warnings(record=True) as w:
+            warnings.simplefilter('always')
+            self._pool.request('GET', '/')
+
+        self.assertEqual([], w)
+
+    def test_ssl_wrong_system_time(self):
+        with mock.patch('urllib3.connection.datetime') as mock_date:
+            mock_date.date.today.return_value = datetime.date(1970, 1, 1)
+
+            with warnings.catch_warnings(record=True) as w:
+                warnings.simplefilter('always')
+                self._pool.request('GET', '/')
+
+            self.assertEqual(len(w), 1)
+            warning = w[0]
+
+            self.assertEqual(SystemTimeWarning, warning.category)
+            self.assertTrue(str(RECENT_DATE) in warning.message.args[0])
+
 
 class TestHTTPS_TLSv1(HTTPSDummyServerTestCase):
     certs = DEFAULT_CERTS.copy()

urllib3/__init__.py

@@ -4,7 +4,7 @@
 
 __author__ = 'Andrey Petrov (andrey.petrov@shazow.net)'
 __license__ = 'MIT'
-__version__ = '1.9'
+__version__ = '1.9.1'
 
 
 from .connectionpool import (
@@ -57,7 +57,7 @@ def add_stderr_logger(level=logging.DEBUG):
 
 # Set security warning to only go off once by default.
 import warnings
-warnings.simplefilter('module', exceptions.InsecureRequestWarning)
+warnings.simplefilter('module', exceptions.SecurityWarning)
 
 def disable_warnings(category=exceptions.HTTPWarning):
     """