Invalid Unicode byte in Authorization token raises an DjangoUnicodeDecodeError

[osantana]

When we sent an invalid/unicode byte in token authentication we got a 500 Internal Server Error (DjangoUnicodeDecodeError) instead of a 401 Unauthorized:

headers = {'Authorization':'token cfbcf941f2fa06a0647b7dc4cb7839199b495c2b¸'}  # <-- last char is an invalid unicode char

Raises the following traceback:

Stacktrace (most recent call last):

  File "django/core/handlers/base.py", line 132, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "python3.4/contextlib.py", line 30, in inner
    return func(*args, **kwds)
  File "django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "rest_framework/viewsets.py", line 85, in view
    return self.dispatch(request, *args, **kwargs)
  File "rest_framework/views.py", line 452, in dispatch
    response = self.handle_exception(exc)
  File "rest_framework/views.py", line 440, in dispatch
    self.initial(request, *args, **kwargs)
  File "rest_framework/views.py", line 354, in initial
    self.perform_authentication(request)
  File "rest_framework/views.py", line 292, in perform_authentication
    request.user
  File "rest_framework/request.py", line 491, in __getattribute__
    return super(Request, self).__getattribute__(attr)
  File "rest_framework/request.py", line 266, in user
    self._authenticate()
  File "rest_framework/request.py", line 454, in _authenticate
    user_auth_tuple = authenticator.authenticate(self)
  File "rest_framework/authentication.py", line 167, in authenticate
    return self.authenticate_credentials(auth[1])
  File "rest_framework/authentication.py", line 171, in authenticate_credentials
    token = self.model.objects.select_related('user').get(key=key)
  File "django/db/models/query.py", line 325, in get
    clone = self.filter(*args, **kwargs)
  File "django/db/models/query.py", line 679, in filter
    return self._filter_or_exclude(False, *args, **kwargs)
  File "django/db/models/query.py", line 697, in _filter_or_exclude
    clone.query.add_q(Q(*args, **kwargs))
  File "django/db/models/sql/query.py", line 1304, in add_q
    clause, require_inner = self._add_q(where_part, self.used_aliases)
  File "django/db/models/sql/query.py", line 1331, in _add_q
    current_negated=current_negated, connector=connector, allow_joins=allow_joins)
  File "django/db/models/sql/query.py", line 1203, in build_filter
    condition = self.build_lookup(lookups, col, value)
  File "django/db/models/sql/query.py", line 1096, in build_lookup
    return final_lookup(lhs, rhs)
  File "django/db/models/lookups.py", line 96, in __init__
    self.rhs = self.get_prep_lookup()
  File "django/db/models/lookups.py", line 134, in get_prep_lookup
    return self.lhs.output_field.get_prep_lookup(self.lookup_name, self.rhs)
  File "django/db/models/fields/__init__.py", line 727, in get_prep_lookup
    return self.get_prep_value(value)
  File "django/db/models/fields/__init__.py", line 1125, in get_prep_value
    return self.to_python(value)
  File "django/db/models/fields/__init__.py", line 1121, in to_python
    return smart_text(value)
  File "django/utils/encoding.py", line 56, in smart_text
    return force_text(s, encoding, strings_only, errors)
  File "django/utils/encoding.py", line 102, in force_text
    raise DjangoUnicodeDecodeError(s, *e.args)

[kevin-brown]

Does this happen if the extra bytes are encoded ahead of time?

And is that character allowed in the Authorization header? Related httpie ticket.

[xordoquy]

@kevin-brown even though we should gracefully return the issue.

[xordoquy]

@osantana do you have a test case for this issue or was it discovered on a running server ?

[osantana]

I've a testcase but it's an application that we're developing/deploying privately for a company, so, we did not have a public API to provide.

But I can reproduce this issue and send you more information about it. The stacktrace provided is the full stacktrace and we don't use any custom Authentication method.

# settings.py
REST_FRAMEWORK = {
    'DEFAULT_PERMISSION_CLASSES': (
        'apps.core.permissions.ModelPermission',
    ),
    'DEFAULT_AUTHENTICATION_CLASSES': (
        'rest_framework.authentication.TokenAuthentication',
        'rest_framework.authentication.SessionAuthentication',
    ),
    'DEFAULT_FILTER_BACKENDS': ('rest_framework.filters.DjangoFilterBackend',)
}

[xordoquy]

@osantana I gave it a try to reproduce the issue but so far I've not been able to do it within a testcase.

[lovelydinosaur]

If we can't replicate this, there's no way we'll be able to progress the issue.
Going to close the ticket off for now, but if there's a demonstrable way to replicate it, then we'll reopen.

[osantana]

Sorry, I forgot to mention a very important thing: this project runs with Python 3.4 (and this exception raises only in Py3).

Besides that I've forked DRF-tutorial project and created a test to reproduce this issue:

https://github.com/osantana/rest-framework-tutorial/blob/master/snippets/tests.py

You can just clone this repository, install requirements and run ./manage.py tests.

[xordoquy]

@osantana will give it a try tonight and reopen if I can reproduce it. Thanks for the details.

[osantana]

News about this issue?

[xordoquy]

@osantana at the moment no. I unsuccessfully tried to reproduce it.