Invalid Unicode byte in Authorization token raises an DjangoUnicodeDecodeError

[osantana]

When we sent an invalid/unicode byte in token authentication we got a 500 Internal Server Error (DjangoUnicodeDecodeError) instead of a 401 Unauthorized:

headers = {'Authorization':'token cfbcf941f2fa06a0647b7dc4cb7839199b495c2b¸'}  # <-- last char is an invalid unicode char

Raises the following traceback:

Stacktrace (most recent call last):

  File "django/core/handlers/base.py", line 132, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "python3.4/contextlib.py", line 30, in inner
    return func(*args, **kwds)
  File "django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "rest_framework/viewsets.py", line 85, in view
    return self.dispatch(request, *args, **kwargs)
  File "rest_framework/views.py", line 452, in dispatch
    response = self.handle_exception(exc)
  File "rest_framework/views.py", line 440, in dispatch
    self.initial(request, *args, **kwargs)
  File "rest_framework/views.py", line 354, in initial
    self.perform_authentication(request)
  File "rest_framework/views.py", line 292, in perform_authentication
    request.user
  File "rest_framework/request.py", line 491, in __getattribute__
    return super(Request, self).__getattribute__(attr)
  File "rest_framework/request.py", line 266, in user
    self._authenticate()
  File "rest_framework/request.py", line 454, in _authenticate
    user_auth_tuple = authenticator.authenticate(self)
  File "rest_framework/authentication.py", line 167, in authenticate
    return self.authenticate_credentials(auth[1])
  File "rest_framework/authentication.py", line 171, in authenticate_credentials
    token = self.model.objects.select_related('user').get(key=key)
  File "django/db/models/query.py", line 325, in get
    clone = self.filter(*args, **kwargs)
  File "django/db/models/query.py", line 679, in filter
    return self._filter_or_exclude(False, *args, **kwargs)
  File "django/db/models/query.py", line 697, in _filter_or_exclude
    clone.query.add_q(Q(*args, **kwargs))
  File "django/db/models/sql/query.py", line 1304, in add_q
    clause, require_inner = self._add_q(where_part, self.used_aliases)
  File "django/db/models/sql/query.py", line 1331, in _add_q
    current_negated=current_negated, connector=connector, allow_joins=allow_joins)
  File "django/db/models/sql/query.py", line 1203, in build_filter
    condition = self.build_lookup(lookups, col, value)
  File "django/db/models/sql/query.py", line 1096, in build_lookup
    return final_lookup(lhs, rhs)
  File "django/db/models/lookups.py", line 96, in __init__
    self.rhs = self.get_prep_lookup()
  File "django/db/models/lookups.py", line 134, in get_prep_lookup
    return self.lhs.output_field.get_prep_lookup(self.lookup_name, self.rhs)
  File "django/db/models/fields/__init__.py", line 727, in get_prep_lookup
    return self.get_prep_value(value)
  File "django/db/models/fields/__init__.py", line 1125, in get_prep_value
    return self.to_python(value)
  File "django/db/models/fields/__init__.py", line 1121, in to_python
    return smart_text(value)
  File "django/utils/encoding.py", line 56, in smart_text
    return force_text(s, encoding, strings_only, errors)
  File "django/utils/encoding.py", line 102, in force_text
    raise DjangoUnicodeDecodeError(s, *e.args)

[kevin-brown]

Does this happen if the extra bytes are encoded ahead of time?

And is that character allowed in the Authorization header? Related httpie ticket.

[xordoquy]

@kevin-brown even though we should gracefully return the issue.

[xordoquy]

@osantana do you have a test case for this issue or was it discovered on a running server ?

[osantana]

I've a testcase but it's an application that we're developing/deploying privately for a company, so, we did not have a public API to provide.

But I can reproduce this issue and send you more information about it. The stacktrace provided is the full stacktrace and we don't use any custom Authentication method.

# settings.py
REST_FRAMEWORK = {
    'DEFAULT_PERMISSION_CLASSES': (
        'apps.core.permissions.ModelPermission',
    ),
    'DEFAULT_AUTHENTICATION_CLASSES': (
        'rest_framework.authentication.TokenAuthentication',
        'rest_framework.authentication.SessionAuthentication',
    ),
    'DEFAULT_FILTER_BACKENDS': ('rest_framework.filters.DjangoFilterBackend',)
}

[xordoquy]

@osantana I gave it a try to reproduce the issue but so far I've not been able to do it within a testcase.

[tomchristie]

If we can't replicate this, there's no way we'll be able to progress the issue.
Going to close the ticket off for now, but if there's a demonstrable way to replicate it, then we'll reopen.

[osantana]

Sorry, I forgot to mention a very important thing: this project runs with Python 3.4 (and this exception raises only in Py3).

Besides that I've forked DRF-tutorial project and created a test to reproduce this issue:

https://github.com/osantana/rest-framework-tutorial/blob/master/snippets/tests.py

You can just clone this repository, install requirements and run ./manage.py tests.

[xordoquy]

@osantana will give it a try tonight and reopen if I can reproduce it. Thanks for the details.

[osantana]

News about this issue?

[xordoquy]

@osantana at the moment no. I unsuccessfully tried to reproduce it.

[osantana]

Even with the tests that I created for rest-framework-tutorial? Have you tried with Python 3.4? This is a Py3-related issue...

[xordoquy]

Sorry, I missed that part and didn't have the to test against your test case yet.

[xordoquy]

@osantana here's the result on my box:

bash-3.2$ pip freeze -l
You are using pip version 6.1.1, however version 7.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Django==1.8.2
djangorestframework==3.1.2
Markdown==2.6.2
Pygments==2.0.2
requests==2.7.0
bash-3.2$ python --version
Python 3.4.3
bash-3.2$ python manage.py test
Creating test database for alias 'default'...
....
----------------------------------------------------------------------
Ran 4 tests in 0.790s

OK
Destroying test database for alias 'default'...
bash-3.2$ 

[xordoquy]

What OS are you using and what are your locals ?

[osantana]

I'm using Ubuntu 14.04 and locale(?) is set with en_US.UTF-8 but I've other language packs installed and generated.

Edited: I'm using 14.04! :)

[osantana]

But I can try to reproduce this issue on other environment... (I've a Mac OS X 10.10 to make tests)

[xordoquy]

@osantana I'm not saying this issue doesn't exist or is void. I'm simply pointing out that it doesn't look like a simple py3 related issue - at least not an obvious one at all.

[osantana]

Ok, I understand this... What I'm trying to do is to know if you managed to reproduce the issue or if I need provide more information to help you :)

It's don't even know if it's a "big issue" but I believe that people can use it to generate a DoS attack in affected APIs...

[kevin-brown]

@osantana At this point I think there might be more involved that needs to be looked at.

If you can replicate the test within the DRF repository and create a (failing) pull request, that might be better than trying to reproduce it outside of the repository.

[osantana]

I don't know if it's the right fix but I submitted the PR #3006 with test and fix.