Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Moving from Python 2 -> 3 and hit snag with webob and JSON serializer, pickle works #431

Closed
russellballestrini opened this issue Jul 1, 2021 · 5 comments

Comments

@russellballestrini
Copy link

russellballestrini commented Jul 1, 2021

I'm taking the time to try to upgrade some of my Pyramid projects from Python 2.7 to 3.9, things are going well but I hit a snag with webob, and I'm wondering if it is a defect or something obvious I'm doing wrong.

I have a work around by falling back the the pickle serializer so there is no rush but I would prefer to use the JSON serializer for signed cookie based sessions.

Here is a snippet of my code in question and the full traceback.

         else:
             # if no active cart in session, create one.
             cart = Cart()

             # The issue appeared when attempting to port from Python 2 to 3.

             # When the serializer is JSON instead of pickle I get the following:
             # TypeError: Unicode-objects must be encoded before hashing
             #request.session["active_cart_id"] = str(cart.id)

             # Still using the JSON serializer when I encode to
             # to utf-8 in order to work with bytes, I get this error:
             # TypeError: Object of type bytes is not JSON serializable
             #request.session["active_cart_id"] = str(cart.id).encode("utf-8")

             # this appears to work fine with pickle.
             request.session["active_cart_id"] = str(cart.id)

             # note for reviewer: cart.id is of type uuid object.

             request.dbsession.add(cart)
             request.dbsession.flush()


2021-07-01 11:08:32,657 ERROR [waitress:404][waitress-0] Exception while serving /
Traceback (most recent call last):
  File "/home/fox/git/make_post_sell/env/lib64/python3.9/site-packages/waitress/channel.py", line 397, in service
    task.service()
  File "/home/fox/git/make_post_sell/env/lib64/python3.9/site-packages/waitress/task.py", line 168, in service
    self.execute()
  File "/home/fox/git/make_post_sell/env/lib64/python3.9/site-packages/waitress/task.py", line 434, in execute
    app_iter = self.channel.server.application(environ, start_response)
  File "/home/fox/git/make_post_sell/env/lib64/python3.9/site-packages/pyramid/router.py", line 270, in __call__
    response = self.execution_policy(environ, self)
  File "/home/fox/git/make_post_sell/env/lib64/python3.9/site-packages/pyramid_retry/__init__.py", line 127, in retry_policy
    response = router.invoke_request(request)
  File "/home/fox/git/make_post_sell/env/lib64/python3.9/site-packages/pyramid/router.py", line 248, in invoke_request
    request._process_response_callbacks(response)
  File "/home/fox/git/make_post_sell/env/lib64/python3.9/site-packages/pyramid/request.py", line 85, in _process_response_callbacks
    callback(self, response)
  File "/home/fox/git/make_post_sell/env/lib64/python3.9/site-packages/pyramid/session.py", line 258, in set_cookie_callback
    self._set_cookie(response)
  File "/home/fox/git/make_post_sell/env/lib64/python3.9/site-packages/pyramid/session.py", line 325, in _set_cookie
    serializer.dumps((self.accessed, self.created, dict(self)))
  File "/home/fox/git/make_post_sell/env/lib64/python3.9/site-packages/webob/cookies.py", line 660, in dumps
    sig = hmac.new(self.salted_secret, cstruct, self.digestmod).digest()
  File "/usr/lib64/python3.9/hmac.py", line 170, in new
    return HMAC(key, msg, digestmod)
  File "/usr/lib64/python3.9/hmac.py", line 93, in __init__
    self.update(msg)
  File "/usr/lib64/python3.9/hmac.py", line 113, in update
    self._inner.update(msg)
TypeError: Unicode-objects must be encoded before hashing

Thanks again for anything you can do to nudge me in the right direction. This code works in Python 2 with JSON serializer.

@mmerickel
Copy link
Member

It sounds like you've written a custom serializer and it's not returning bytes.

@russellballestrini
Copy link
Author

@mmerickel thanks for the rapid call out, here is another snippet:

from pyramid.config import Configurator

# Cookie only session, not encrypted but signed to prevent tampering.
from pyramid.session import SignedCookieSessionFactory

from miscutils import get_children_settings


def session_serializer(serializer_name):
    if serializer_name == "json":
        import json
        return json
    import pickle
    return pickle


def main(global_config, **settings):
    """This function returns a Pyramid WSGI application."""

    # setup session factory to use unencrypted but signed cookies
    session_settings = get_children_settings(settings, "session")
    session_settings["serializer"] = session_serializer(
        session_settings.get("serializer")
    )
    session_factory = SignedCookieSessionFactory(**session_settings)

    with Configurator(settings=settings, session_factory=session_factory) as config:
        # all of the models for persisting data into our database.
        config.include(".models")
        # all of the web application routes.
        config.include(".routes")
        # all of the data we attach to each inbound request.
        config.include(".request_methods")
        # enable jinja2 templating engine.
        config.include(".config_jinja2")
        # scan each of these includes for additional configuration.
        config.scan()

    return config.make_wsgi_app()

@mmerickel
Copy link
Member

json.dumps() returns a unicode string, you'll need to json.dumps().encode('utf8') to conform to the serializer contract.

@russellballestrini
Copy link
Author

Thanks, I figured as much. Here is a first pass at making json compatible with the serializer contract. Let me know if there is a better way.

First I created a new file / module which I may import and pass to the session configurator:

import json

loads = json.loads

def dumps(obj, *args, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, cls=None, indent=None, separators=None, default=None, sort_keys=False, **kw):
    result = json.dumps(obj, *args, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, cls=None, indent=None, separators=None, default=None, sort_keys=False, **kw)
    return result.encode("utf-8")

and then in my main function above, I changed the session_serializer function to look like so:

def session_serializer(serializer_name):
    if serializer_name == "json":
        import make_post_sell.lib.json_session_serializer as json_serializer
        return json_serializer
    import pickle
    return pickle

This appears to work so I will close this issue. Please feel free to comment after closing if my approach is bad.

@russellballestrini
Copy link
Author

Actually looks like we already have a batteries included solution, see: https://docs.pylonsproject.org/projects/pyramid/en/latest/api/session.html#pyramid.session.JSONSerializer

def session_serializer(serializer_name):
    if serializer_name == "json":
        from pyramid.session import JSONSerializer
        return JSONSerializer()

    from pyramid.session import PickleSerializer
    return PickleSerializer()

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants