SSLError raised while 'r.config._ssl_url = None'

[Stevoisiak]

Issue Description

According to a comment on issue #202, you can disable SSL by adding .config._ssl_url = None to a script.

However, even with SSL disabled, I am encountering multiple SSL Errors when trying to use PRAW on a company network that uses self-signed certificates.

Code

import praw

reddit = praw.Reddit(client_id='####',
                     client_secret='####',
                     user_agent='windows:testapp (by /u/stevoisiak)')
reddit.config._ssl_url = None  # Disable SSL

for submission in reddit.subreddit('learnpython').hot(limit=10):
    print(submission.title)

Traceback

Traceback (most recent call last):
  File "C:\Users\svascellar\AppData\Local\Programs\Python\Python36-32\lib\site-packages\urllib3-1.22-py3.6.egg\urllib3\connectionpool.py", line 601, in urlopen
    chunked=chunked)
  File "C:\Users\svascellar\AppData\Local\Programs\Python\Python36-32\lib\site-packages\urllib3-1.22-py3.6.egg\urllib3\connectionpool.py", line 346, in _make_request
    self._validate_conn(conn)
  File "C:\Users\svascellar\AppData\Local\Programs\Python\Python36-32\lib\site-packages\urllib3-1.22-py3.6.egg\urllib3\connectionpool.py", line 850, in _validate_conn
    conn.connect()
  File "C:\Users\svascellar\AppData\Local\Programs\Python\Python36-32\lib\site-packages\urllib3-1.22-py3.6.egg\urllib3\connection.py", line 326, in connect
    ssl_context=context)
  File "C:\Users\svascellar\AppData\Local\Programs\Python\Python36-32\lib\site-packages\urllib3-1.22-py3.6.egg\urllib3\util\ssl_.py", line 329, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "C:\Users\svascellar\AppData\Local\Programs\Python\Python36-32\lib\ssl.py", line 407, in wrap_socket
    _context=self, _session=session)
  File "C:\Users\svascellar\AppData\Local\Programs\Python\Python36-32\lib\ssl.py", line 814, in __init__
    self.do_handshake()
  File "C:\Users\svascellar\AppData\Local\Programs\Python\Python36-32\lib\ssl.py", line 1068, in do_handshake
    self._sslobj.do_handshake()
  File "C:\Users\svascellar\AppData\Local\Programs\Python\Python36-32\lib\ssl.py", line 689, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\svascellar\AppData\Local\Programs\Python\Python36-32\lib\site-packages\requests-2.18.4-py3.6.egg\requests\adapters.py", line 440, in send
    timeout=timeout
  File "C:\Users\svascellar\AppData\Local\Programs\Python\Python36-32\lib\site-packages\urllib3-1.22-py3.6.egg\urllib3\connectionpool.py", line 639, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "C:\Users\svascellar\AppData\Local\Programs\Python\Python36-32\lib\site-packages\urllib3-1.22-py3.6.egg\urllib3\util\retry.py", line 388, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='www.reddit.com', port=443): Max retries exceeded with url: /api/v1/access_token (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)'),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\svascellar\AppData\Local\Programs\Python\Python36-32\lib\site-packages\prawcore\requestor.py", line 47, in request
    return self._http.request(*args, timeout=TIMEOUT, **kwargs)
  File "C:\Users\svascellar\AppData\Local\Programs\Python\Python36-32\lib\site-packages\requests-2.18.4-py3.6.egg\requests\sessions.py", line 508, in request
    resp = self.send(prep, **send_kwargs)
  File "C:\Users\svascellar\AppData\Local\Programs\Python\Python36-32\lib\site-packages\requests-2.18.4-py3.6.egg\requests\sessions.py", line 618, in send
    r = adapter.send(request, **kwargs)
  File "C:\Users\svascellar\AppData\Local\Programs\Python\Python36-32\lib\site-packages\requests-2.18.4-py3.6.egg\requests\adapters.py", line 506, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: HTTPSConnectionPool(host='www.reddit.com', port=443): Max retries exceeded with url: /api/v1/access_token (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)'),))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:/Users/svascellar/.PyCharmCE2018.1/config/scratches/praw test.py", line 8, in <module>
    for submission in reddit.subreddit('learnpython').hot(limit=10):
  File "C:\Users\svascellar\AppData\Local\Programs\Python\Python36-32\lib\site-packages\praw\models\listing\generator.py", line 52, in __next__
    self._next_batch()
  File "C:\Users\svascellar\AppData\Local\Programs\Python\Python36-32\lib\site-packages\praw\models\listing\generator.py", line 62, in _next_batch
    self._listing = self._reddit.get(self.url, params=self.params)
  File "C:\Users\svascellar\AppData\Local\Programs\Python\Python36-32\lib\site-packages\praw\reddit.py", line 371, in get
    data = self.request('GET', path, params=params)
  File "C:\Users\svascellar\AppData\Local\Programs\Python\Python36-32\lib\site-packages\praw\reddit.py", line 486, in request
    params=params)
  File "C:\Users\svascellar\AppData\Local\Programs\Python\Python36-32\lib\site-packages\prawcore\sessions.py", line 182, in request
    params=params, url=url)
  File "C:\Users\svascellar\AppData\Local\Programs\Python\Python36-32\lib\site-packages\prawcore\sessions.py", line 125, in _request_with_retries
    retries, saved_exception, url)
  File "C:\Users\svascellar\AppData\Local\Programs\Python\Python36-32\lib\site-packages\prawcore\sessions.py", line 91, in _do_retry
    params=params, url=url, retries=retries - 1)
  File "C:\Users\svascellar\AppData\Local\Programs\Python\Python36-32\lib\site-packages\prawcore\sessions.py", line 125, in _request_with_retries
    retries, saved_exception, url)
  File "C:\Users\svascellar\AppData\Local\Programs\Python\Python36-32\lib\site-packages\prawcore\sessions.py", line 91, in _do_retry
    params=params, url=url, retries=retries - 1)
  File "C:\Users\svascellar\AppData\Local\Programs\Python\Python36-32\lib\site-packages\prawcore\sessions.py", line 113, in _request_with_retries
    data, files, json, method, params, retries, url)
  File "C:\Users\svascellar\AppData\Local\Programs\Python\Python36-32\lib\site-packages\prawcore\sessions.py", line 98, in _make_request
    params=params)
  File "C:\Users\svascellar\AppData\Local\Programs\Python\Python36-32\lib\site-packages\prawcore\rate_limit.py", line 32, in call
    kwargs['headers'] = set_header_callback()
  File "C:\Users\svascellar\AppData\Local\Programs\Python\Python36-32\lib\site-packages\prawcore\sessions.py", line 142, in _set_header_callback
    self._authorizer.refresh()
  File "C:\Users\svascellar\AppData\Local\Programs\Python\Python36-32\lib\site-packages\prawcore\auth.py", line 299, in refresh
    self._request_token(grant_type='client_credentials')
  File "C:\Users\svascellar\AppData\Local\Programs\Python\Python36-32\lib\site-packages\prawcore\auth.py", line 138, in _request_token
    response = self._authenticator._post(url, **data)
  File "C:\Users\svascellar\AppData\Local\Programs\Python\Python36-32\lib\site-packages\prawcore\auth.py", line 29, in _post
    data=sorted(data.items()))
  File "C:\Users\svascellar\AppData\Local\Programs\Python\Python36-32\lib\site-packages\prawcore\requestor.py", line 49, in request
    raise RequestException(exc, args, kwargs)
prawcore.exceptions.RequestException: error with request HTTPSConnectionPool(host='www.reddit.com', port=443): Max retries exceeded with url: /api/v1/access_token (Caused by SSLError(SSLError(1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)'),))

System Information

• PRAW Version: 5.4.0
• Python Version: 3.6.4
• Operating System: Windows 7 Professional - Service Pack 1 (64 bit)

[bboe]

The approach used then isn't going to work today.

PRAW uses requests, which has support for disabling certificate validation via verify=False: http://docs.python-requests.org/en/master/user/advanced/#ssl-cert-verification

This use to be a first-class feature in PRAW3, but it looks like it didn't make it to PRAW4+, and you appear to be the first to ask for it. Perhaps you can figure out what changes need to be made to change the internal request session to disable verification? It might be as easy as something like reddit.http.verify=False, but I'm guessing it's a bit more complicated than that.

[Stevoisiak]

@bboe Thanks for the helpful response. I'll try seeing if I can configure requests to disable SSL.

Side note: There's a Stack Overflow question on this issue from 2014.

[Stevoisiak]

Where does PRAW initialize Requests?

There's a docstring referencing Requests.session in reddit.py, but I can't find where the library is called.

praw/praw/reddit.py

Lines 110 to 120 in 49f3315

	import json, betamax, requests
	class JSONDebugRequestor(Requestor):
	def request(self, *args, **kwargs):
	response = super().request(*args, **kwargs)
	print(json.dumps(response.json(), indent=4))
	return response
	my_session = betamax.Betamax(requests.Session())
	reddit = Reddit(..., requestor_class=JSONDebugRequestor,
	requestor_kwargs={'session': my_session})

[bboe]

It nested in prawcore:

https://github.com/praw-dev/prawcore/blob/master/prawcore/requestor.py#L33

[bboe]

Note that Requestor.request passes all its arguments directly to its session's request method. So you might not need to go that deep.

[jarhill0]

@Stevoisiak I was able to use a custom requests Session using the following code:

import requests

import praw

s = requests.Session()
s.verify = '/path/to/certfile'
reddit = praw.Reddit(requestor_kwargs={'session': s})  # and other params

I believe this is what you need to do when using self-signed certificates (source: Requests documentation), but I don't have experience with that so I may be wrong.

I hope this helps!

[Stevoisiak]

@jarhill0 Specifying a .pem certificate worked perfectly.

import praw
import requests

session = requests.Session()
session.verify = "C:/path/to/cert/certificate.pem"
reddit = praw.Reddit(client_id='###',
                     client_secret='###',
                     user_agent='windows:testapp (by /u/stevoisiak)',
                     requestor_kwargs={'session': session})

for submission in reddit.subreddit('learnpython').hot(limit=10):
    print(submission.title)

Alternatively, you can fully disable SSL with session.verify = False. Although this will cause each request to raise an InsecureRequestWarning.

(Be warned that disabling SSL is strongly discouraged by the urllib3 documentation)

import praw
import requests

session = requests.Session()
session.verify = False
reddit = praw.Reddit(client_id='###',
                     client_secret='###',
                     user_agent='windows:testapp (by /u/stevoisiak)',
                     requestor_kwargs={'session': session})

for submission in reddit.subreddit('learnpython').hot(limit=10):
    print(submission.title)

[bboe]

Perhaps someone can add a page to the documentation with this information?

[Stevoisiak]

@bboe I have compiled this information as an answer on StackOverflow.

[jarhill0]

I'll write it up for the docs, unless you'd like to @Stevoisiak.

[Stevoisiak]

@jarhill0 You can write up the docs. Feel free to copy from my SO answer if you'd like.

Thank you for your help on this issue!