Decoding issues on Linux cause Segmentation fault for given sites

[gjthompson1]

Summary.

I have tested the following on my Mac and I get both unicode errors:

For the following:

r = requests.get('http://xn--w8jujla2a2c4e0lg4403e52qmkrfye.biz', stream=True)

I get:

UnicodeError: decoding with 'idna' codec failed (UnicodeError: Invalid character '\x80')

For:

r = requests.get('http://xn--meiner-dta.hamburg', stream=True)

I get:

UnicodeError: ('IDNA does not round-trip', b'xn--meiner-dta', b'meissner')

Which I am good with.

But on linux I get:

Segmentation fault (core dumped)

Crashing my program.

Note the versions of python are slightly different but I guess the following is true for ubuntu.

https://forums.aws.amazon.com/thread.jspa?threadID=255687

Expected Result

Expect it to error out like on MacOS.

System Information

$ python -m requests.help

On Mac

{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.6"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.6.2"
  },
  "platform": {
    "release": "16.3.0",
    "system": "Darwin"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.18.4"
  },
  "system_ssl": {
    "version": "100020cf"
  },
  "urllib3": {
    "version": "1.22"
  },
  "using_pyopenssl": false
}

On Linux

{
  "chardet": {
    "version": "3.0.4"
  },
  "cryptography": {
    "version": ""
  },
  "idna": {
    "version": "2.6"
  },
  "implementation": {
    "name": "CPython",
    "version": "3.5.2"
  },
  "platform": {
    "release": "4.4.0-1032-aws",
    "system": "Linux"
  },
  "pyOpenSSL": {
    "openssl_version": "",
    "version": null
  },
  "requests": {
    "version": "2.18.4"
  },
  "system_ssl": {
    "version": "1000207f"
  },
  "urllib3": {
    "version": "1.22"
  },
  "using_pyopenssl": false
}

[Lukasa]

Hrm. Requests cannot be the cause of this segfault because it's pure-Python code. Can you use GDB or something similar to examine the coredump and check the Python stack to see what code was executing when the segfault occurred?

[gjthompson1]

Lukas, thanks for getting back to me. I tried to run using GDB. I get the following:

test.py

import requests
print('Hey')
r = requests.get('http://xn--w8jujla2a2c4e0lg4403e52qmkrfye.biz', stream=True)

Then

(gdb) run test.py
Starting program: /usr/bin/python3 test.py
warning: Error disabling address space randomization: Operation not permitted
Hey
During startup program terminated with signal SIGSEGV, Segmentation fault.

I guess there is something wrong here because its not catching anything..

[Lukasa]

Sorry, we need a Python backtrace. See this article

[gjthompson1]

Does this work?

(gdb) run test.py
Starting program: /usr/bin/python3 test.py
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Hey

Program received signal SIGSEGV, Segmentation fault.
0x00007ffff673f767 in ?? () from /usr/lib/python3.5/lib-dynload/_ssl.cpython-35m-x86_64-linux-gnu.so

(gdb) bt
#0  0x00007ffff673f767 in ?? () from /usr/lib/python3.5/lib-dynload/_ssl.cpython-35m-x86_64-linux-gnu.so
#1  0x00007ffff6739231 in ?? () from /usr/lib/python3.5/lib-dynload/_ssl.cpython-35m-x86_64-linux-gnu.so
#2  0x00000000004e9bc7 in PyCFunction_Call ()
#3  0x0000000000524414 in PyEval_EvalFrameEx ()
#4  0x000000000052e12b in PyEval_EvalCodeEx ()
#5  0x00000000004ebdd7 in ?? ()
#6  0x00000000005b7167 in PyObject_Call ()
#7  0x00000000004f413e in ?? ()
#8  0x00000000005b7167 in PyObject_Call ()
#9  0x000000000054d359 in ?? ()
#10 0x000000000055d17c in ?? ()
#11 0x00000000005b7167 in PyObject_Call ()
#12 0x0000000000528d06 in PyEval_EvalFrameEx ()
#13 0x000000000052d2e3 in ?? ()
#14 0x0000000000528eee in PyEval_EvalFrameEx ()
#15 0x000000000052d2e3 in ?? ()
#16 0x0000000000528eee in PyEval_EvalFrameEx ()
#17 0x0000000000528814 in PyEval_EvalFrameEx ()
#18 0x000000000052d82f in ?? ()
#19 0x0000000000529332 in PyEval_EvalFrameEx ()
#20 0x000000000052d2e3 in ?? ()
#21 0x0000000000528eee in PyEval_EvalFrameEx ()
#22 0x000000000052d2e3 in ?? ()
#23 0x0000000000528eee in PyEval_EvalFrameEx ()
#24 0x000000000052e12b in PyEval_EvalCodeEx ()
#25 0x00000000004ebdd7 in ?? ()
#26 0x00000000005b7167 in PyObject_Call ()
#27 0x00000000005262af in PyEval_EvalFrameEx ()
#28 0x000000000052e12b in PyEval_EvalCodeEx ()
#29 0x00000000004ebdd7 in ?? ()
#30 0x00000000005b7167 in PyObject_Call ()
#31 0x00000000005262af in PyEval_EvalFrameEx ()
#32 0x00000000004ed325 in ?? ()

[Lukasa]

Nope, we need the py-bt

[gjthompson1]

hmm

Undefined command: "py-bt". Try "help".

[gjthompson1]

Got it.

Needed:

sudo apt-get install gdb python3.5-dbg

(gdb) py-bt
Traceback (most recent call first):
  <built-in method _wrap_socket of SSLContext object at remote 0x7ffff4c43f98>
  File "/usr/lib/python3.5/ssl.py", line 745, in __init__
    server_hostname)
  File "/usr/lib/python3.5/ssl.py", line 377, in wrap_socket
    _context=self)
  File "/usr/lib/python3/dist-packages/urllib3/util/ssl_.py", line 305, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 252, in connect
    ssl_version=resolved_ssl_version)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 787, in _validate_conn
    conn.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 346, in _make_request
    self._validate_conn(conn)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 560, in urlopen
    body=body, headers=headers)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 376, in send
    timeout=timeout
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 576, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 195, in resolve_redirects
    **adapter_kwargs
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 597, in <listcomp>
    history = [resp for resp in gen] if allow_redirects else []
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 597, in send
    history = [resp for resp in gen] if allow_redirects else []
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 468, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/api.py", line 53, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/api.py", line 67, in get
    return request('get', url, params=params, **kwargs)
  File "test.py", line 3, in <module>

[Lukasa]

Ok, this segfault is coming from your interpreter. How did you install it?

[gjthompson1]

You can reproduce this if you use the following Dockerfile

FROM ubuntu:latest

RUN apt-get update -y
RUN apt-get install -y python3-pip python3-dev libffi-dev

RUN apt-get install -y vim

COPY requirements.txt /tmp/
RUN pip3 install --requirement /tmp/requirements.txt

requirements.txt

requests==2.18.4

[steven2308]

I have the same problems (the unicode ones, not the segmentation fault) in Ubuntu 16.10 with Python 3.6.0b2 and aiohttp==2.2.5

This reproduces the problem:

import asyncio
import aiohttp


async def process():
    url = 'http://xn--x9dj.com/'
    session = aiohttp.ClientSession()
    async with session.get(url) as response:
        print(response)
    session.close()

loop = asyncio.get_event_loop()
loop.run_until_complete(
    process()
)
loop.close()

Traceback:

Traceback (most recent call last):
  File "test.py", line 14, in <module>
    process()
  File "/usr/lib/python3.6/asyncio/base_events.py", line 449, in run_until_complete
    return future.result()
  File "/usr/lib/python3.6/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "test.py", line 8, in process
    async with session.get(url) as response:
  File "/myenv/lib/python3.6/site-packages/aiohttp/client.py", line 603, in __aenter__
    self._resp = yield from self._coro
  File "/myenv/lib/python3.6/site-packages/aiohttp/client.py", line 226, in _request
    session=self)
  File "/myenv/lib/python3.6/site-packages/aiohttp/client_reqrep.py", line 96, in __init__
    self.update_host(url)
  File "/myenv/lib/python3.6/site-packages/aiohttp/client_reqrep.py", line 123, in update_host
    if not url.host:
  File "/myenv/lib/python3.6/site-packages/yarl/__init__.py", line 59, in __get__
    val = self.wrapped(inst)
  File "/myenv/lib/python3.6/site-packages/yarl/__init__.py", line 433, in host
    return raw.encode('ascii').decode('idna')
  File "/myenv/lib/python3.6/encodings/idna.py", line 214, in decode
    result.append(ToUnicode(label))
  File "/myenv/lib/python3.6/encodings/idna.py", line 139, in ToUnicode
    raise UnicodeError("IDNA does not round-trip", label, label2)
UnicodeError: ('IDNA does not round-trip', b'xn--x9dj', b'xn--c09aja')
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7fe5774205f8>

if I use this domain instead:

xn---1-9ld5cqq.com

Then I get this:

Traceback (most recent call last):
  File "/myenv/lib/python3.6/encodings/idna.py", line 214, in decode
    result.append(ToUnicode(label))
  File "/myenv/lib/python3.6/encodings/idna.py", line 134, in ToUnicode
    label2 = ToASCII(result)
  File "/myenv/lib/python3.6/encodings/idna.py", line 76, in ToASCII
    label = nameprep(label)
  File "/myenv/lib/python3.6/encodings/idna.py", line 58, in nameprep
    raise UnicodeError("Violation of BIDI requirement 3")
UnicodeError: Violation of BIDI requirement 3

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "test.py", line 44, in <module>
    process()
  File "/usr/lib/python3.6/asyncio/base_events.py", line 449, in run_until_complete
    return future.result()
  File "/usr/lib/python3.6/asyncio/tasks.py", line 239, in _step
    result = coro.send(None)
  File "test.py", line 12, in process
    async with session.get(url) as response:
  File "/myenv/lib/python3.6/site-packages/aiohttp/client.py", line 603, in __aenter__
    self._resp = yield from self._coro
  File "/myenv/lib/python3.6/site-packages/aiohttp/client.py", line 226, in _request
    session=self)
  File "/myenv/lib/python3.6/site-packages/aiohttp/client_reqrep.py", line 96, in __init__
    self.update_host(url)
  File "/myenv/lib/python3.6/site-packages/aiohttp/client_reqrep.py", line 123, in update_host
    if not url.host:
  File "/myenv/lib/python3.6/site-packages/yarl/__init__.py", line 59, in __get__
    val = self.wrapped(inst)
  File "/myenv/lib/python3.6/site-packages/yarl/__init__.py", line 433, in host
    return raw.encode('ascii').decode('idna')
UnicodeError: decoding with 'idna' codec failed (UnicodeError: Violation of BIDI requirement 3)
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x7f31f464c5f8>

I hope this helps clarifying the problem.
Thank you.

[nateprewitt]

Since this is an issue with the Python interpreter and not Requests' code base, I'm going to close this out.

@steven2308 you appear to be using aiohttp and the stacktrace you've provided doesn't contain any Requests code. It appears you're supplying an invalid URI to IDNA which is causing issues. I suggest you follow up with aiohttp if you feel there's an issue.