Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sanely handle Infinity/-Infinity values in JSON using ?_json_infinity=1 #332

Closed
simonw opened this issue Jun 29, 2018 · 12 comments
Closed
Labels

Comments

@simonw
Copy link
Owner

simonw commented Jun 29, 2018

It turns out if you load this CSV using csvs-to-sqlite you get an Infinity value in SQLite:

name,num
sasha,10
terry,Inf
cathy,0.5

csvs-to-sqlite infinity-bug.csv infinity-bug.db

I deployed this using:

datasette publish now infinity-bug.db --name=datasette-infinity-bug --install=datasette-vega

Datasette outputs that as Infinity in the JSON format, which causes JavaScript errors.

Demo

[
    {
        "rowid": 1,
        "name": "sasha",
        "num": 10.0
    },
    {
        "rowid": 2,
        "name": "terry",
        "num": Infinity
    },
    {
        "rowid": 3,
        "name": "cathy",
        "num": 0.5
    }
]

But... in Firefox that gets rendered like this:

2018-06-29 at 4 20 pm

And if you click the "Show charting options" button you get this error in the console:

SyntaxError: JSON.parse: unexpected character at line 1 column 83 of the JSON data
@simonw simonw added the bug label Jun 29, 2018
@simonw
Copy link
Owner Author

simonw commented Jun 29, 2018

https://docs.python.org/3/library/json.html#json.dump

json.dump(obj, fp, *, skipkeys=False, ensure_ascii=True, check_circular=True, allow_nan=True, cls=None, indent=None, separators=None, default=None, sort_keys=False, **kw)¶
If allow_nan is false (default: True), then it will be a ValueError to serialize out of range float values (nan, inf, -inf) in strict compliance of the JSON specification. If allow_nan is true, their JavaScript equivalents (NaN, Infinity, -Infinity) will be used.

@simonw
Copy link
Owner Author

simonw commented Jun 29, 2018

I'm not sure what the correct thing to do here is. I don't want to throw a ValueError when trying to render that data as JSON, but I also want to produce JSON that doesn't break when fetched by JavaScript.

@simonw
Copy link
Owner Author

simonw commented Jul 3, 2018

I think I'm going to return null in the JSON for infinity/nan values by default, but if you send _nan=1 I will instead return invalid JSON with Infinity or NaN in it (since you have opted in to getting those and hence should be able to handle them).

@simonw simonw added this to the Next release milestone Jul 10, 2018
@simonw simonw changed the title Sanely handle Infinity values Sanely handle Infinity values in JSON Jul 12, 2018
@simonw
Copy link
Owner Author

simonw commented Jul 12, 2018

@simonw
Copy link
Owner Author

simonw commented Jul 12, 2018

And here's how django-rest-framework did it: https://github.com/encode/django-rest-framework/pull/4918/files

@simonw
Copy link
Owner Author

simonw commented Jul 12, 2018

Since my data is all flat lists of values I don't think I need to customize the JSON encoder itself (no need to deal with nested values). I'll fix the data on its way into the encoder instead. This will also help if I decide to move to uJSON for better performance #48

@simonw
Copy link
Owner Author

simonw commented Jul 18, 2018

Maybe argument should be ?_json_nan=1 since that makes it more explicitly obvious what is going on here.

@simonw simonw changed the title Sanely handle Infinity values in JSON Sanely handle Infinity/NaN values in JSON using ?_json_nan=1 Jul 18, 2018
@simonw
Copy link
Owner Author

simonw commented Jul 24, 2018

Actually SQLite doesn't handle NaN at all (it treats it as null) so I'm going to change this ticket to just deal with Infinity and -Infinity.

@simonw simonw changed the title Sanely handle Infinity/NaN values in JSON using ?_json_nan=1 Sanely handle Infinity/-Infinity values in JSON using ?_json_infinity=1 Jul 24, 2018
@simonw
Copy link
Owner Author

simonw commented Jul 24, 2018

I'm going with _json_infinity=1 as the querystring argument.

@simonw
Copy link
Owner Author

simonw commented Jul 24, 2018

According to https://www.mail-archive.com/sqlite-users@mailinglists.sqlite.org/msg110573.html you can insert Infinity/-Infinity in raw SQL (as used by our fixtures) using 1e999 and -1e999.

@simonw simonw closed this as completed in 700d83d Jul 24, 2018
@simonw
Copy link
Owner Author

simonw commented Jul 24, 2018

Demo:

@simonw
Copy link
Owner Author

simonw commented Jul 24, 2018

@simonw simonw removed this from the Next release milestone Jul 24, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

1 participant