Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

TypeError: Decimal(...) is not JSON serializable with use_decimal=True #34

Closed
ngnpope opened this Issue May 10, 2012 · 3 comments

Comments

Projects
None yet
2 participants

ngnpope commented May 10, 2012

Hi,

I seem to have run into a problem when attempting to encode decimal types. I am running multiple applications on multiple workers using uWSGI. This problem essentially boils down to the use of multiple sub-interpreters.

In uWSGI a work around for this problem is to use the --single-interpreter option but this prevents multiple applications per process.

The problem in simplejson, I believe, is due to the caching of the Decimal type in the speed-ups C extension. This leads to a problem when checking if we have a decimal because the cached type is the same across all sub-interpreters, but the Decimal object compared is different across each sub-interpreter, i.e. the cache type in the extension is the type from the first sub-interpreter that imported simplejson and thus for subsequent sub-interpreters effectively "Decimal != Decimal".

I originally thought this may have been a problem with uWSGI and was pointed to the same issue that existed in psycopg2 < 2.4.3. I have updated psycopg2 and ruled that out as the cause, and deleting the _speedups.so library to force simplejson to use its python-based encoder caused the problem to disappear.

See the following links for more information:

I'm not very familiar with Python's C API unfortunately, otherwise I would have tried to offer you a patch.

I hope this won't be too awkward to fix.

Cheers,

Nick

ngnpope commented May 10, 2012

As a follow up, the workaround I am currently using is to use DjangoJSONEncoder to force the Decimal conversion to string to happen Python-side. Alternatively (for non-Django users) the following could be used to prevent comparison of Decimal types in the C extension while still being able to have the benefit of the speed-ups for everything else:

import decimal, simplejson
class DecimalJSONEncoder(simplejson.JSONEncoder):
    def default(self, o):
        if isinstance(o, decimal.Decimal):
            return str(o)
        return super(DecimalJSONEncoder, self).default(o)

data = decimal.Decimal('10.0000000000')
print simplejson.dumps(data, cls=DecimalJSONEncoder)
Owner

etrepum commented May 10, 2012

I believe that v2.5.1 should resolve your issue, let me know if it doesn't. I haven't tested it in an actual sub-interpreter environment.

607a489

@etrepum etrepum closed this May 10, 2012

ngnpope commented May 11, 2012

Just tested... Works wonderfully, thanks :)

jperkin pushed a commit to joyent/pkgsrc-legacy that referenced this issue Dec 9, 2013

Update to 2.5.2:
Version 2.5.2 released 2012-05-10

* Fix for regression introduced in 2.5.1
  simplejson/simplejson#35

Version 2.5.1 released 2012-05-10

* Support for use_decimal=True in environments that use Python
  sub-interpreters such as uWSGI
  simplejson/simplejson#34

Version 2.5.0 released 2012-03-29

* New item_sort_key option for encoder to allow fine grained control of sorted
  output

Version 2.4.0 released 2012-03-06

* New bigint_as_string option for encoder to trade JavaScript number precision
  issues for type issues.
  simplejson/simplejson#31

Version 2.3.3 released 2012-02-27

* Allow unknown numerical types for indent parameter
  simplejson/simplejson#29

Version 2.3.2 released 2011-12-30

* Fix crashing regression in speedups introduced in 2.3.1

Version 2.3.1 released 2011-12-29

* namedtuple_as_object now checks _asdict to ensure that it
  is callable.
  simplejson/simplejson#26

Version 2.3.0 released 2011-12-05

* Any objects with _asdict() methods are now considered for
  namedtuple_as_object.
  simplejson/simplejson#22

jsonn pushed a commit to jsonn/pkgsrc that referenced this issue Oct 11, 2014

Update to 2.5.2:
Version 2.5.2 released 2012-05-10

* Fix for regression introduced in 2.5.1
  simplejson/simplejson#35

Version 2.5.1 released 2012-05-10

* Support for use_decimal=True in environments that use Python
  sub-interpreters such as uWSGI
  simplejson/simplejson#34

Version 2.5.0 released 2012-03-29

* New item_sort_key option for encoder to allow fine grained control of sorted
  output

Version 2.4.0 released 2012-03-06

* New bigint_as_string option for encoder to trade JavaScript number precision
  issues for type issues.
  simplejson/simplejson#31

Version 2.3.3 released 2012-02-27

* Allow unknown numerical types for indent parameter
  simplejson/simplejson#29

Version 2.3.2 released 2011-12-30

* Fix crashing regression in speedups introduced in 2.3.1

Version 2.3.1 released 2011-12-29

* namedtuple_as_object now checks _asdict to ensure that it
  is callable.
  simplejson/simplejson#26

Version 2.3.0 released 2011-12-05

* Any objects with _asdict() methods are now considered for
  namedtuple_as_object.
  simplejson/simplejson#22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment