OverflowError on large long values

[ishirav]

Large long values cannot be encoded by ujson (cjson does not have the same problem). Example code:

import ujson, cjson
d = {'num': 2**65}
ujson.encode(d)
>> OverflowError: long too big to convert
cjson.encode(d)
>> '{"num": 36893488147419103232}'

[jskorpan]

As designed.

On 25 jul 2013, at 10:55, "ishirav" <notifications@github.commailto:notifications@github.com> wrote:

Large long values cannot be encoded by ujson (cjson does not have the same problem). Example code:

import ujson, cjson
d = {'num': 2**65}
ujson.encode(d)

OverflowError: long too big to convert
cjson.encode(d)
'{"num": 36893488147419103232}'

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/99.

[ishirav]

Well... this is a deviation from the JSON standard, which does not impose a limit on the length of integers.

It's also a "landmine" that might blow up in a production system - because everything works when you test it with numbers that are small enough, but unexpectedly fails when somewhere, someone passes a very large number.

I don't think it a good idea to keep this limitation in place, even if it's "as designed".

Thanks!

[ethompsy]

Wouldn't it be better to return SOME value instead of an error? For example it should be simple enough to convert the long to it's string representation and return that. Refactoring for a returned string is easier than refactoring to avoid presenting a long to the renderer.

Is this possible?

_Some context:_ I am trying to use ultrajson with a production system and as soon as I installed it all my JSON endpoints are dead with this error because we use longs as our Primary Key IDs on most of our models. So I would need to refactor ALL of my endpoints to try to avoid presenting a long to the renderer. This is a mess. I agree with @ishirav that this is not a good limitation, and I would say this limitation is arguably poor design if it was intended.

[ishirav]

@ethompsy, I think returning a string value when a number is expected is even worse than throwing an error, because you're hiding the problem and also changing the content of the JSON document.

[ethompsy]

@ishirav, Is it worse than NOT supporting long integers?

I use ujson==1.33 with drf-ujson==1.2 and I just spent part of a day going through my serializers and converting any longs to strings so that they can make it through to the front end. Now I will resolve any front-end issues this causes. I am still going to refactor the front end, but a little help with passing through the values would have been nice. I had to toggle JSON renderers in order to see the values that were causing the errors. If I was not truly believing in the 2.3 X faster rendering I would not have done all this.

I think that ease of use should be a consideration when making these design decisions. I do think your point is still the most useful suggestion here: "this is a deviation from the JSON standard, which does not impose a limit on the length of integers". Sticking to agreed standards is one way to ensure ease of use.

[byaka]

Finded some really strange bug
ujson version 1.35

> ujson.dumps({'params': [1, 18446744073709551616L], 'method': ''})
'{"params":[1,],"method":""}'

> ujson.dumps({'params': [1, 18446744073709551616L], 'test': ''})
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
OverflowError: long too big to convert

as you can see in first example encoder silently ignore LONG and return incorrect data, but in second example it raise error

[isakruas]

When I try to convert a dictionary to json, I get this error.

I'm using

ecc==0.0.1.dev8
ujson ==5.1.0

Python 3.8.10 (tags/v3.8.10:3d8993a, May 3 2021, 11:48:03) [MSC v.1928 64 bit (AMD64)] on win32

from uuid import uuid4
from ecc.ecdh import ECDH

client = ECDH(private_key=int(uuid4().hex, 16), curve='secp521r1')

data = {
    'private_key': client.private_key,
    'public_key': client.public_key
}
print(ujson.dumps(data))

Traceback (most recent call last):
File "try.py", line 53, in
print(ujson.dumps(data))
OverflowError: int too big to convert