ujson dumps NaN but doesn't load it

[stinos]

The mod_ujson_load function doesn't handle nan though it is a valid representation:

>>> import ujson
>>> ujson.dumps(float('nan'))
'nan'
>>> ujson.loads(ujson.dumps(float('nan')))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: syntax error in JSON

Quick hack to solve this (sorry couldn't immediately figure out how to get the nan constant), line 115 in modujson.c:

                } else if (S_CUR(s) == 'a' && S_NEXT(s) == 'n') {
                    S_NEXT(s);
                    next = mp_parse_num_decimal("nan", 3, false, false, NULL);

[dpgeorge]

Reading the JSON spec, it doesn't actually support encoding of nan or inf (or -inf). These values should be encoded in the JSON stream as "null".

OTOH, Python implements a common extension to JSON that encodes the values as NaN, Infinity and -Infinity, which are valid JavaScript expressions. And then Python accepts an argument to dumps to select whether it outputs strict JSON or the extended JSON.

So the question is what uPy should do.

[stinos]

Yes this is a 'standard' discussion of micro vs CPython-compatibility vs usability. In this case I'd lean towards the principle of least surprise, which is likely also the reason why e.g. CPython defaults to True for the allow_nan keyword on the json functions. I might be biased though because we need this functionality since we've got 2 uPy applications talking to each other and apart from cooking our own serialization, ujson has proven to be the best option (or micropython-lib's pickle, but it also doesn't handle nan)

[martijnthe]

I also just ran into this.

OTOH, Python implements a common extension to JSON that encodes the values as NaN, Infinity and -Infinity, which are valid JavaScript expressions.

Hmm, but in practice, one will most likely use JSON.parse() in JavaScript to parse JSON (not every JSON string is valid JS code).

JSON.parse('Infinity') will actually throw a syntax error (it's not a valid JSON as per the spec):

$ node
> JSON.parse('Infinity')
SyntaxError: Unexpected token I in JSON at position 0

So, I tend to agree with @dpgeorge, even though it deviates from what Python does (which is wrong IMO):

These values should be encoded in the JSON stream as "null".

[stinos]

Personally I still like the current situation more than using 'null'. At least now you know it when something goes wrong.. Whereas using 'null' would mean the parser returns a different value than what was originally there, without the ability to figure out if the value parsed is the same as what was dumped or whether it was an unsupported value. Distinct values with very different meanings like None, float('nan'), float('inf') etc would all result in the same encoded value.

[martijnthe]

At least now you know it when something goes wrong..

How about raising a ValueError when a nan, inf or -inf (or any other value that can't be JSON-serialized) is passed? (So basically what would happen in Python if you'd add allow_nan=False.)

Deviating from standard specs in ways that breaks the format is a no-no, IMO... I'm actually really (unpleasantly) surprised Python defaults to allow_nan=True and spits out non-spec-compliant "JSON".

[pfalcon]

How about raising a ValueError when a nan, inf or -inf (or any other value that can't be JSON-serialized) is passed?

You can go over your data and raise arbitrary exception on arbitrary value you see there.

[stinos]

I'm actually really (unpleasantly) surprised

Well I was also unpleasantly surprised to learn that the spec chose to leave out perfectly valid values for a widely used datatype :)

[aalexandrov]

Why not add the allow_nan flag and serialize NaN and inf as string literals ("nan" and "inf"). This is JSON-spec compliant and allows readers to recover the original value.