TypeError: bad tzinfo state arg when encoding/decoding timezone-aware datetimes

[mandx]

Here's the Gist

I'm using Django (and I have pytz installed, and Django uses it) to parse datetime values with timezone information. The problem appears to be when encoding these values and then decoding them in another Python process (like a background queue).

This is an example of what I'm pickling. The TEST_DATES_STR is not the actual value I'm saving; parse_datetime lives in django.utils.dateparse, and that code it's already included in the gist, so there's no need to install Django to reproduce the bug:

TEST_DATES_STR = ["2015-08-12T00:30:00Z", "2015-10-02T08:30:00Z"]
TEST_DATES = [parse_datetime(d) for d in TEST_DATES_STR]

Basically we have encode.py (first invocation) which simply does:

sys.stdout.write(jsonpickle.dumps(TEST_DATES))

and we have decode.py that simply does:

data = sys.stdin.read()
print(data)
dates = jsonpickle.loads(data)
assert dates == TEST_DATES

The traceback (execution never reaches the assert statement):

  File "decode.py", line 9, in <module>
    dates = jsonpickle.loads(data)
  File "/home/mandx/github/jsonpickle/tests/jsonpickle/__init__.py", line 148, in decode
    return unpickler.decode(string, backend=backend, keys=keys)
  File "/home/mandx/github/jsonpickle/tests/jsonpickle/unpickler.py", line 26, in decode
    return context.restore(backend.decode(string), reset=reset)
  File "/home/mandx/github/jsonpickle/tests/jsonpickle/unpickler.py", line 119, in restore
    value = self._restore(obj)
  File "/home/mandx/github/jsonpickle/tests/jsonpickle/unpickler.py", line 159, in _restore
    return restore(obj)
  File "/home/mandx/github/jsonpickle/tests/jsonpickle/unpickler.py", line 382, in _restore_list
    children = [self._restore(v) for v in obj]
  File "/home/mandx/github/jsonpickle/tests/jsonpickle/unpickler.py", line 382, in <listcomp>
    children = [self._restore(v) for v in obj]
  File "/home/mandx/github/jsonpickle/tests/jsonpickle/unpickler.py", line 159, in _restore
    return restore(obj)
  File "/home/mandx/github/jsonpickle/tests/jsonpickle/unpickler.py", line 235, in _restore_object
    instance = handler(self).restore(obj)
  File "/home/mandx/github/jsonpickle/tests/jsonpickle/handlers.py", line 178, in restore
    return cls.__new__(cls, *params)
TypeError: bad tzinfo state arg

This is while jsonpickle is deserializing the second datetime value (the first value is properly deserialized). I tried using make_refs=False when calling jsonpickle.dumps, but no dice. I tracked down the bug to this method (jsonpickle/handlers.py:166):

class DatetimeHandler(BaseHandler):
    # ...
    def restore(self, data):
        cls, args = data['__reduce__']
        unpickler = self.context
        restore = unpickler.restore
        cls = restore(cls, reset=False)
        value = util.b64decode(args[0])
        params = (value,) + tuple([restore(i, reset=False) for i in args[1:]])
        return cls.__new__(cls, *params)

Where the params tuple ends up with the last item being a _Proxy object with the instance attribute set to None. If we add these lines just before the return statement:

try:
    if params[-1].instance is None:
        params = params[:-1]
except (IndexError, AttributeError):
    pass

Of course the TypeError is gone, but it will return the correct datetime but without timezone information.

[jorritb]

Seems like this hasn't been fixed yet in 1.2? Is this the only workaround?

[talmachani]

Hi all, got some new info.

installed version: jsonpickle==1.4.2

First, here is a simple script of how to reproduce the issue fast and easy.

import datetime
from pprint import pprint

import jsonpickle
import pytz

d1 = datetime.datetime(2020, 6, 6, 6, 6, 6, 6, tzinfo=pytz.UTC)
d2 = datetime.datetime(2020, 6, 6, 6, 6, 6, 6, tzinfo=pytz.UTC)
l = [d1, d2]
encoded_list = jsonpickle.encode(l, make_refs=False)
pprint(encoded_list)
decoded_list = jsonpickle.decode(encoded_list)

I have been doing some investigation, the issue happens only when make_refs=False.
and source of it is in the _in_cycle method, specifically this condition (not make_refs and id(obj) in objs) something is not right here.

As per my understanding when flatten is called for pytz.UTC object since it's a singleton it has the same ID and it will be in objs so _in_cycle will return true and repr will be called.

My proposed solution is when make_refs is False, self._objs should not be populated.

[davvid]

This is another issue that is solved by the proposed solution in #338 -- please give it a try. I added unit tests with pytz.UTC tzinfo as part of that change to verify that it's working as intended.

The changes are currently in the refs branch in my davvid fork. Let me know if that works for you.

[davvid]

Merged and fixed 😺