Provide ciphers list for HTTPS requests

[damiengermonville]

Hi,

Since Python 2.7, the ssl module implements a way to provide a list of available ciphers to the SSL object [1]. It would be nice if this feature could be reflected in Requests (and in Urllib3 as well).

I know it's not a key feature but some web servers have a Long Handshake intolerance [2] and that makes tools (and Python scripts) based on OpenSSL 1.0.1+ fail at handshake. A workaround is to provide a short list of ciphers instead of all the available ones.

I've looked into trough the Requests documentation and googled. I didn't find a way to do it. Maybe there is, if so i would appreciate a bit of help :)

Thanks !

[1] http://docs.python.org/2/library/ssl.html?highlight=ssl#ssl.wrap_socket
[2] https://github.com/ssllabs/research/wiki/Long-Handshake-Intolerance

[medecau]

Let us pick and choose what version of SSL we use.

curl https://www.bbva.pt/
unless
curl --sslv3 https://www.bbva.pt/

[Lukasa]

The accepted way of doing this in Requests is to use the Transport Adapter functionality. You can see an example of doing exactly this here.

[medecau]

@Lukasa thanks, this solves my problem.

[damiengermonville]

@Lukasa Thanks for the quick answer. I've looked at your post and I think it's a good start to solve my problem maybe (which is setting the ciphers, not the SSL version :p)

The VerifiedHTTPSConnection wraps the socket and that's where I need to provide ciphers (this is the only way apparently). I'm not sure i'll be able to create a pool and then use it in a adapter, all without having to copy/paste existing code (I'm looking for a clean way of doing it)

Many thanks again

[sigmavirus24]

@damiengermonville you might then need to petition over at shazow/urllib3 for an easier way to pass in the ciphers to the VerifiedHTTPSConnection object. We have no clue what your code looks like, but I imagine this might make it easier. It's also something I don't believe @shazow would be that adverse to, but I'm pretty sick at the moment so it's probable that I'm wrong. :)

[shazow]

You should already be able to select what version of SSL/TLS you want to force, iirc. Either way, I'm open to adjustments to the VerifiedHTTPSConnection object as long as they're sensible and don't significantly increase code complexity. :)

[damiengermonville]

@shazow I'm afraid that adjustments have to be made in VerifiedHTTPSConnection but also in urllib3/util.py ssl_wrap_socket() so it can give the ciphers argument to ssl.wrap_socket().

[shazow]

@damiengermonville Sounds fine. :)

[cecemel]

Hi,

I had exactly the same needs, and I needed some time to find how to add the specific ciphers. So if I may, I will just add what I did so people can find it (please correct me if I am completely wrong):

from requests.packages.urllib3.contrib import pyopenssl
pyopenssl.DEFAULT_SSL_CIPHER_LIST = 'your list her'

This will also be handy for people bumping into issues with the OpenSSL version of Ubuntu 14.04 and Requests 2.7 and python 2.7.6
(see: here and here )

[athoik]

Hi,

The following worked for me!

import requests

requests.packages.urllib3.util.ssl_.DEFAULT_CIPHERS += ':RC4-SHA'
try:
    requests.packages.urllib3.contrib.pyopenssl.DEFAULT_SSL_CIPHER_LIST += ':RC4-SHA'
except AttributeError:
    # no pyopenssl support used / needed / available
    pass

Taken from: http://stackoverflow.com/questions/32650984/why-does-python-requests-ignore-the-verify-parameter/32651967#32651967

[Lukasa]

Please do not do it this way. Instead, follow the approach in this blog post and change the ciphers only for specific sites. RC4 is extremely dangerous and broken, and enabling it in the way shown in @athoik's comment will enable it for all sites, opening you up to trivial attacks. If you must allow RC4 or other removed ciphers, do so on a site-by-site basis.

[athoik]

@Lukasa, I had to enable only the AES256-SHA in my case.

requests.packages.urllib3.util.ssl_.DEFAULT_CIPHERS = 'AES256-SHA'

I just copy / pasted the original solution.

[Lukasa]

Ah, I see. In general this is still using a sledgehammer to crack a nut, and a per-host approach as discussed above is the better way to go.

[alainrobert]

I am wondering why does requests provide a list of DEFAULT_CYPHERS in ssl_ when ssl already provides one.
It took me a long time to discover that it was the reason for the issue I was having. I find SSL is hard to debug enough to have different layer adding their own filters, it is only a the end of my journey now that I know the problem that I find this useful discussion and blog post.
The solution you are proposing is nice but it requires that we change the code and to use a session, this means possibly a lot of refactoring.
Also because the server is controlling the encoding there is nothing we can do in the client code, so restricting the list just makes the connection fail.
Shouldn't we let people restrict the list easily with publicly documented methods rather than making hard for everyone to use the default values?

[Lukasa]

I am wondering why does requests provide a list of DEFAULT_CYPHERS in ssl_ when ssl already provides one.

A number of reasons. For example:

1. On older Python versions the ssl module does not provide a list of default ciphers.
2. The Python ssl module list is not updated after the release of the module: we can update ours much faster than others. For example, we removed 3DES from our list and got out a release within a month, whereas it took many months for Python to do so, and most Python users still have not seen this update.

The solution you are proposing is nice but it requires that we change the code and to use a session, this means possibly a lot of refactoring.

Yes. However, Sessions are a good thing that should be used in all but the simplest of codebases, so it's a major help. And if your code is simple, you can just create a global session and change all your calls to requests.x to session.x. The refactoring is intentionally very simple.

Also because the server is controlling the encoding there is nothing we can do in the client code, so restricting the list just makes the connection fail.

This represents a severe misunderstanding of how TLS works. The client and server together decide which cipher suite is used, based on the server selecting its preference from the list the client supports. The client is responsible for defending itself against the server making bad choices. Clients need to have an opinion about what ciphers are acceptable for use.

If your assertion is that the client should allow whatever cipher suite the server chooses, then you are essentially saying that TLS does not serve a security purpose for you at all: it is merely an operational inconvenience. In that case I recommend you don't use HTTPS: you're just adding complexity to your work for no reason.

Otherwise, the client is responsible for preventing old and badly configured servers from leaking client data.

Shouldn't we let people restrict the list easily with publicly documented methods rather than making hard for everyone to use the default values?

No.

Requests has a strong cultural bias towards "secure by default". Essentially, if you use Requests in the default configuration then we guarantee that you are using a configuration that is secure based on our understanding of the network threat ecosystem at the time we released.

Users should not opt-in to secure configurations because that creates an ecosystem in which only those who have the knowledge to understand the risks are secure. Requests aims to ensure all our users are secure, not just those who are keeping up to date with the crypto literature.

So our policy is that we will be secure, and give you opt-outs. We don't promise to make them easy: we'll let you point the gun at your foot, but we won't necessarily provide an easy switch to do it. If you have to do a little bit of work to reduce your security, then that is a reasonable trade-off for ensuring that casual users don't accidentally do it.

[alainrobert]

Thank you, you are making very good points. I wish the customization would be documented in requests doc along with some troubleshooting tips, the interface is awesome but this was not a good experience...

[Lukasa]

Yeah, this is an understandable concern. Generally we don't want to make things like this super easy to find just to discourage their use a bit more, which is why we've traditionally resorted to "documenting" this kind of thing in blog posts rather than inline.

[IanNMarshall]

Hi,
I've been going through this and several other similar threads for several hours now, and none of the solutions suggested to me seem to work. I think my issue is the older protocol / cipher, but not sure as nothing here has worked.

Error is on this line -> bad handshake / unexpected EOF
s.get('https://www.elections.ny.gov/regbycounty.html')

I'm using Python 2.7.11
Here's the output of pip freeze:

curl makes connection fine with TLSv1.0 and Cipher: RSA_WITH_3DES_EDE_CBC_SHA1

I've tried both the not recommended way (adding my cipher string to the DEFAULT_CIPHERS) and the preferred way from https://lukasa.co.uk/2017/02/Configuring_TLS_With_Requests/ with no luck. I think DES-CBC3-SHA is the string I need to be adding, not sure if I'm just doing it wrong (I've tried concatenating to the end of CIPHERS making it the only string in CIPHERS etc.) or if there's something else going on here.

import requests
from requests.adapters import HTTPAdapter
from requests.packages.urllib3.util.ssl_ import create_urllib3_context

# This is the 2.11 Requests cipher string, containing 3DES.
CIPHERS = (
    'ECDH+AESGCM:DH+AESGCM:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:ECDH+HIGH:'
    'DH+HIGH:ECDH+3DES:DH+3DES:RSA+AESGCM:RSA+AES:RSA+HIGH:RSA+3DES:!aNULL:'
    '!eNULL:!MD5'
)
#CIPHERS += (':DES-CBC3-SHA')

class DESAdapter(HTTPAdapter):
    """
    A TransportAdapter that re-enables 3DES support in Requests.
    """
    def init_poolmanager(self, *args, **kwargs):
        context = create_urllib3_context(ciphers=CIPHERS)
        kwargs['ssl_context'] = context
        return super(DESAdapter, self).init_poolmanager(*args, **kwargs)

    def proxy_manager_for(self, *args, **kwargs):
        context = create_urllib3_context(ciphers=CIPHERS)
        kwargs['ssl_context'] = context
        return super(DESAdapter, self).proxy_manager_for(*args, **kwargs)

print CIPHERS
s = requests.Session()
s.mount('https://www.elections.ny.gov/regbycounty.html', DESAdapter())
r = s.get('https://www.elections.ny.gov/regbycounty.html')

And the output (bad handshake)

Thanks in advance!

[autumnjolitz]

Hi @IanNMarshall

I've been bitten in the ass by this "bug" as well. It's not a requests issue, nor do the requests authors really care whether or not their documentation RE old ciphers is actually useful, given that it's unreasonable to detect if and when the provided OpenSSL support libraries have the ciphers you ask for.

After a lot of work, I've determined that if you have pyOpenSSL installed, you will get the above error. At least, that's what I thought the issue was, given that removal of pyOpenSSL magically allowed your test program to pass (after enabling the DES-CBC3-SHA cipher)

Looking closer, I determined that the binary assets being distributed for cryptography for OpenSSL fundamentally lacks RC4, 3DES, et al ciphers, which makes pyOpenSSL fail for forcing to "just" that cipher. It appears that requests will use pyOpenSSL and failsafe to built-in ssl if it goes missing.

If you can prove that the following works with your desired cipher:

(cpython27) btj-gt$ openssl ciphers -V '3DES'
          0xC0,0x12 - ECDHE-RSA-DES-CBC3-SHA  SSLv3 Kx=ECDH     Au=RSA  Enc=3DES(168) Mac=SHA1
          0xC0,0x08 - ECDHE-ECDSA-DES-CBC3-SHA SSLv3 Kx=ECDH     Au=ECDSA Enc=3DES(168) Mac=SHA1
          0xC0,0x1C - SRP-DSS-3DES-EDE-CBC-SHA SSLv3 Kx=SRP      Au=DSS  Enc=3DES(168) Mac=SHA1
          0xC0,0x1B - SRP-RSA-3DES-EDE-CBC-SHA SSLv3 Kx=SRP      Au=RSA  Enc=3DES(168) Mac=SHA1
          0xC0,0x1A - SRP-3DES-EDE-CBC-SHA    SSLv3 Kx=SRP      Au=SRP  Enc=3DES(168) Mac=SHA1
          0x00,0x16 - EDH-RSA-DES-CBC3-SHA    SSLv3 Kx=DH       Au=RSA  Enc=3DES(168) Mac=SHA1
          0x00,0x13 - EDH-DSS-DES-CBC3-SHA    SSLv3 Kx=DH       Au=DSS  Enc=3DES(168) Mac=SHA1
          0x00,0x10 - DH-RSA-DES-CBC3-SHA     SSLv3 Kx=DH/RSA   Au=DH   Enc=3DES(168) Mac=SHA1
          0x00,0x0D - DH-DSS-DES-CBC3-SHA     SSLv3 Kx=DH/DSS   Au=DH   Enc=3DES(168) Mac=SHA1
          0xC0,0x17 - AECDH-DES-CBC3-SHA      SSLv3 Kx=ECDH     Au=None Enc=3DES(168) Mac=SHA1
          0x00,0x1B - ADH-DES-CBC3-SHA        SSLv3 Kx=DH       Au=None Enc=3DES(168) Mac=SHA1
          0xC0,0x0D - ECDH-RSA-DES-CBC3-SHA   SSLv3 Kx=ECDH/RSA Au=ECDH Enc=3DES(168) Mac=SHA1
          0xC0,0x03 - ECDH-ECDSA-DES-CBC3-SHA SSLv3 Kx=ECDH/ECDSA Au=ECDH Enc=3DES(168) Mac=SHA1
          0x00,0x0A - DES-CBC3-SHA            SSLv3 Kx=RSA      Au=RSA  Enc=3DES(168) Mac=SHA1
          0x00,0x8B - PSK-3DES-EDE-CBC-SHA    SSLv3 Kx=PSK      Au=PSK  Enc=3DES(168) Mac=SHA1

Then you may be able to just pip uninstall -y pyOpenSSL and it may work.

I was able to make everything "work" (even with pyOpenSSL) by making a usable cryptography installation via

CPPFLAGS=-I/usr/local/opt/openssl/include LDFLAGS=-L/usr/local/opt/openssl/lib pip install --no-binary cryptography cryptography