New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Uncaptured exception instead of 400 when receiving non-ascii bytes in request url #64

Closed
dpetukhov opened this Issue Jul 3, 2014 · 4 comments

Comments

Projects
None yet
4 participants
@dpetukhov

dpetukhov commented Jul 3, 2014

Hi.

I have waitress installed on my production server and it generates strange exceptions. Like this:

uncaptured python exception, closing channel <waitress.channel.HTTPChannel connected 127.0.0.1:54006 at 0xb034478c>
(<class 'UnicodeDecodeError'>:'ascii' codec can't decode byte 0xd0 in position 124: ordinal not in range(128)
[/usr/lib/python3.2/asyncore.py|read|83]
[/usr/lib/python3.2/asyncore.py|handle_read_event|449]
[/home/novo/py32/libpython3.2/site-packages/waitress/channel.py|handle_read|175]
[/home/novo/py32/lib/python3.2/site-packages/waitress/channel.py|received|192]
[/home/novo/py32/lib/python3.2/site-packages/waitress/parser.py|received|102]
[/home/novo/py32/lib/python3.2/site-packages/waitress/parser.py|parse_header|206]
[/home/novo/py32/lib/python3.2/site-packages/waitress/parser.py|split_uri|254]
[/usr/lib/python3.2/urllib/parse.py|urlsplit|314]
[/usr/lib/python3.2/urllib/parse.py|_coerce_args|101]
[/usr/lib/python3.2/urllib/parse.py|_decode_args|85]
[/usr/lib/python3.2/urllib/parse.py
<genexpr>|85])

I found that these exceptions generated when some stupid bot sends request with raw bytes \xd0 in http header in first line. Here is a line from nginx log (nginx proxy queries to waitress):

54.209.115.31 - myhost.ru - [03/Jul/2014:15:41:46 +0000] "GET /lands/lands/%D0%B1%D0%B0%D0%B9%D0%BA%D0%B0%D0%BB/%D0%BC%D0%B0%D0%BB%D0%BE%D0%B5-%D0%BC%D0%BE%D1%80%D0%B5/lands/\xD0\xB1\xD0\xB0\xD0\xB9\xD0\xBA\xD0\xB0\xD0\xBB/\xD0\xBC\xD0\xB0\xD0\xBB\xD0\xBE\xD0\xB5-\xD0\xBC\xD0\xBE\xD1\x80\xD0\xB5/\xD0\xB7\xD0\xB0\xD0\xBB\xD0\xB8\xD0\xB2-\xD0\xBA\xD1\x83\xD1\x80\xD0\xBA\xD1\x83\xD1\x82\xD1\x81\xD0\xBA\xD0\xB8\xD0\xB9/\xD0\xBA\xD1\x83\xD1\x80\xD0\xBA\xD1\x83\xD1\x82/id330/ HTTP/1.1" 502 172 "-" "" "-"

As you can see, here the client sent both urlescaped and raw bytes in GET. According to RFC, url must by escaped. But nginx doesn't care.

I also wrote simple POC to demonstrate the issue:

def send_xdo(host, port=80):
    s = socket.create_connection((host, port))
    req = b"GET /\xd0 HTTP/1.1\n\n"
    s.send(req)
    s.recv(4096)
    s.close()

send_xdo('localhost', 6543)

I think Waitress should be able to handle this exception. It will be better just send 400 Bad Request to the client. Take a look at the rfc - http://tools.ietf.org/html/rfc2616#section-5.1.2

The Request-URI is transmitted in the format specified in section
3.2.1. If the Request-URI is encoded using the "% HEX HEX" encoding
[42], the origin server MUST decode the Request-URI in order to
properly interpret the request. Servers SHOULD respond to invalid
Request-URIs with an appropriate status code.

@mcdonc

This comment has been minimized.

Show comment
Hide comment
@mcdonc

mcdonc Jul 14, 2014

Member

FWIW, as a note to myself, the UnicodeDecodeError doesn't occur under Python 2, only under Python 3.

Member

mcdonc commented Jul 14, 2014

FWIW, as a note to myself, the UnicodeDecodeError doesn't occur under Python 2, only under Python 3.

@Flimm

This comment has been minimized.

Show comment
Hide comment
@Flimm

Flimm Jun 1, 2017

Contributor

What about modifying the test in the way that I suggested in this pull request #162 ?

Contributor

Flimm commented Jun 1, 2017

What about modifying the test in the way that I suggested in this pull request #162 ?

bertjwregeer added a commit that referenced this issue Aug 16, 2017

Merge pull request #162 from Flimm/fix.issue64-flimm
Fix issue #64 Uncaptured exception instead of 400 when receiving non-ascii bytes in request url
@Flimm

This comment has been minimized.

Show comment
Hide comment
@Flimm

Flimm Aug 16, 2017

Contributor

Thank you @bertjwregeer !

To be clear to anyone lurking, the fix has been merged but not yet released on PyPI.

Contributor

Flimm commented Aug 16, 2017

Thank you @bertjwregeer !

To be clear to anyone lurking, the fix has been merged but not yet released on PyPI.

@Flimm

This comment has been minimized.

Show comment
Hide comment
@Flimm

Flimm Oct 11, 2017

Contributor

waitress 1.1.0 has been released on PyPI, and it includes this fix.

Contributor

Flimm commented Oct 11, 2017

waitress 1.1.0 has been released on PyPI, and it includes this fix.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment