Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Invalid Unicode byte in Authorization token raises an DjangoUnicodeDecodeError #2928

Closed
osantana opened this issue May 12, 2015 · 20 comments
Closed
Labels
Milestone

Comments

@osantana
Copy link
Contributor

osantana commented May 12, 2015

When we sent an invalid/unicode byte in token authentication we got a 500 Internal Server Error (DjangoUnicodeDecodeError) instead of a 401 Unauthorized:

headers = {'Authorization':'token cfbcf941f2fa06a0647b7dc4cb7839199b495c2b¸'}  # <-- last char is an invalid unicode char

Raises the following traceback:

Stacktrace (most recent call last):

  File "django/core/handlers/base.py", line 132, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "python3.4/contextlib.py", line 30, in inner
    return func(*args, **kwds)
  File "django/views/decorators/csrf.py", line 58, in wrapped_view
    return view_func(*args, **kwargs)
  File "rest_framework/viewsets.py", line 85, in view
    return self.dispatch(request, *args, **kwargs)
  File "rest_framework/views.py", line 452, in dispatch
    response = self.handle_exception(exc)
  File "rest_framework/views.py", line 440, in dispatch
    self.initial(request, *args, **kwargs)
  File "rest_framework/views.py", line 354, in initial
    self.perform_authentication(request)
  File "rest_framework/views.py", line 292, in perform_authentication
    request.user
  File "rest_framework/request.py", line 491, in __getattribute__
    return super(Request, self).__getattribute__(attr)
  File "rest_framework/request.py", line 266, in user
    self._authenticate()
  File "rest_framework/request.py", line 454, in _authenticate
    user_auth_tuple = authenticator.authenticate(self)
  File "rest_framework/authentication.py", line 167, in authenticate
    return self.authenticate_credentials(auth[1])
  File "rest_framework/authentication.py", line 171, in authenticate_credentials
    token = self.model.objects.select_related('user').get(key=key)
  File "django/db/models/query.py", line 325, in get
    clone = self.filter(*args, **kwargs)
  File "django/db/models/query.py", line 679, in filter
    return self._filter_or_exclude(False, *args, **kwargs)
  File "django/db/models/query.py", line 697, in _filter_or_exclude
    clone.query.add_q(Q(*args, **kwargs))
  File "django/db/models/sql/query.py", line 1304, in add_q
    clause, require_inner = self._add_q(where_part, self.used_aliases)
  File "django/db/models/sql/query.py", line 1331, in _add_q
    current_negated=current_negated, connector=connector, allow_joins=allow_joins)
  File "django/db/models/sql/query.py", line 1203, in build_filter
    condition = self.build_lookup(lookups, col, value)
  File "django/db/models/sql/query.py", line 1096, in build_lookup
    return final_lookup(lhs, rhs)
  File "django/db/models/lookups.py", line 96, in __init__
    self.rhs = self.get_prep_lookup()
  File "django/db/models/lookups.py", line 134, in get_prep_lookup
    return self.lhs.output_field.get_prep_lookup(self.lookup_name, self.rhs)
  File "django/db/models/fields/__init__.py", line 727, in get_prep_lookup
    return self.get_prep_value(value)
  File "django/db/models/fields/__init__.py", line 1125, in get_prep_value
    return self.to_python(value)
  File "django/db/models/fields/__init__.py", line 1121, in to_python
    return smart_text(value)
  File "django/utils/encoding.py", line 56, in smart_text
    return force_text(s, encoding, strings_only, errors)
  File "django/utils/encoding.py", line 102, in force_text
    raise DjangoUnicodeDecodeError(s, *e.args)
@kevin-brown
Copy link
Member

kevin-brown commented May 12, 2015

Does this happen if the extra bytes are encoded ahead of time?

And is that character allowed in the Authorization header? Related httpie ticket.

@xordoquy
Copy link
Collaborator

xordoquy commented May 12, 2015

@kevin-brown even though we should gracefully return the issue.

@xordoquy
Copy link
Collaborator

xordoquy commented May 13, 2015

@osantana do you have a test case for this issue or was it discovered on a running server ?

@xordoquy xordoquy added this to the 3.1.3 Release milestone May 13, 2015
@osantana
Copy link
Contributor Author

osantana commented May 14, 2015

I've a testcase but it's an application that we're developing/deploying privately for a company, so, we did not have a public API to provide.

But I can reproduce this issue and send you more information about it. The stacktrace provided is the full stacktrace and we don't use any custom Authentication method.

# settings.py
REST_FRAMEWORK = {
    'DEFAULT_PERMISSION_CLASSES': (
        'apps.core.permissions.ModelPermission',
    ),
    'DEFAULT_AUTHENTICATION_CLASSES': (
        'rest_framework.authentication.TokenAuthentication',
        'rest_framework.authentication.SessionAuthentication',
    ),
    'DEFAULT_FILTER_BACKENDS': ('rest_framework.filters.DjangoFilterBackend',)
}

@xordoquy
Copy link
Collaborator

xordoquy commented May 16, 2015

@osantana I gave it a try to reproduce the issue but so far I've not been able to do it within a testcase.

@tomchristie
Copy link
Member

tomchristie commented May 18, 2015

If we can't replicate this, there's no way we'll be able to progress the issue.
Going to close the ticket off for now, but if there's a demonstrable way to replicate it, then we'll reopen.

@tomchristie tomchristie removed this from the 3.1.3 Release milestone May 18, 2015
@osantana
Copy link
Contributor Author

osantana commented May 18, 2015

Sorry, I forgot to mention a very important thing: this project runs with Python 3.4 (and this exception raises only in Py3).

Besides that I've forked DRF-tutorial project and created a test to reproduce this issue:

https://github.com/osantana/rest-framework-tutorial/blob/master/snippets/tests.py

You can just clone this repository, install requirements and run ./manage.py tests.

@xordoquy
Copy link
Collaborator

xordoquy commented May 18, 2015

@osantana will give it a try tonight and reopen if I can reproduce it. Thanks for the details.

@osantana
Copy link
Contributor Author

osantana commented Jun 2, 2015

News about this issue?

@xordoquy
Copy link
Collaborator

xordoquy commented Jun 2, 2015

@osantana at the moment no. I unsuccessfully tried to reproduce it.

@osantana
Copy link
Contributor Author

osantana commented Jun 2, 2015

Even with the tests that I created for rest-framework-tutorial? Have you tried with Python 3.4? This is a Py3-related issue...

@xordoquy
Copy link
Collaborator

xordoquy commented Jun 2, 2015

Sorry, I missed that part and didn't have the to test against your test case yet.

@xordoquy
Copy link
Collaborator

xordoquy commented Jun 2, 2015

@osantana here's the result on my box:

bash-3.2$ pip freeze -l
You are using pip version 6.1.1, however version 7.0.3 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
Django==1.8.2
djangorestframework==3.1.2
Markdown==2.6.2
Pygments==2.0.2
requests==2.7.0
bash-3.2$ python --version
Python 3.4.3
bash-3.2$ python manage.py test
Creating test database for alias 'default'...
....
----------------------------------------------------------------------
Ran 4 tests in 0.790s

OK
Destroying test database for alias 'default'...
bash-3.2$ 

@xordoquy
Copy link
Collaborator

xordoquy commented Jun 2, 2015

What OS are you using and what are your locals ?

@osantana
Copy link
Contributor Author

osantana commented Jun 2, 2015

I'm using Ubuntu 14.04 and locale(?) is set with en_US.UTF-8 but I've other language packs installed and generated.

Edited: I'm using 14.04! :)

@osantana
Copy link
Contributor Author

osantana commented Jun 2, 2015

But I can try to reproduce this issue on other environment... (I've a Mac OS X 10.10 to make tests)

@xordoquy
Copy link
Collaborator

xordoquy commented Jun 3, 2015

@osantana I'm not saying this issue doesn't exist or is void. I'm simply pointing out that it doesn't look like a simple py3 related issue - at least not an obvious one at all.

@osantana
Copy link
Contributor Author

osantana commented Jun 3, 2015

Ok, I understand this... What I'm trying to do is to know if you managed to reproduce the issue or if I need provide more information to help you :)

It's don't even know if it's a "big issue" but I believe that people can use it to generate a DoS attack in affected APIs...

@kevin-brown
Copy link
Member

kevin-brown commented Jun 3, 2015

@osantana At this point I think there might be more involved that needs to be looked at.

If you can replicate the test within the DRF repository and create a (failing) pull request, that might be better than trying to reproduce it outside of the repository.

@osantana
Copy link
Contributor Author

osantana commented Jun 3, 2015

I don't know if it's the right fix but I submitted the PR #3006 with test and fix.

@xordoquy xordoquy added the Bug label Jun 16, 2015
@xordoquy xordoquy added this to the 3.1.4 Release milestone Jun 16, 2015
@tomchristie tomchristie modified the milestones: 3.1.4 Release, 3.2.0 Release Jul 30, 2015
@tomchristie tomchristie modified the milestones: 3.2.0 Release, 3.1.4 Release Jul 30, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants