Breaking on 64bit ids?

[pablomendes]

If one parses a JSON object with a large number represented as int and as string (e.g. Twitter IDs), jq seems to break.

$ curl https://gist.github.com/gnip/764239/raw/6c6a2297f3e4e29a626f07db0c57b45af7d7e5d7/Twitter+%28json+format%29.js | \
 jq -r '"\(.id)\t\(.id_str)"'

Works fine.

28039652140     28039652140

But with 64bit Twitter IDs:

curl https://gist.github.com/gnip/764239/raw/6c6a2297f3e4e29a626f07db0c57b45af7d7e5d7/Twitter+%28json+format%29.js | \
 sed -r 's|28039652140|308125194256523266|' | \
 jq -r '"\(.id)\t\(.id_str)"'

It does not:

308125194256523>>260<<      308125194256523>>266<<

(Emphasis added.) A bug?

$ jq --version
jq version 1.3             

PS: just in case you arrived here searching for a solution for the specific case of Twitter JSON, just using id_str instead of id is probably the way to go. Just thought it would be worth mentioning the issue.

[lluchs]

See #143

[pablomendes]

Thanks for the reference. In this particular case I would have loved to at least get a warning, if not an error message, so that the caller knows that the processing of that JSON piece was "not safe."

[nicowilliams]

Well, you should see the recent 1,000+ mail threads in the IETF JSON WG
mailing list about: what JSON strings may contain, whether there may be
duplicate names (keys) in objects and what should happen if there are, the
range and precision of JSON numbers, whether there may be values other than
arrays or objects at the top-level, and what a JSON text is. Suffice it to
say that there's mostly only consensus about explaining what has been
implemented and giving some idea as to what can be expected to interop.

There are implementations that don't handle decimals very well at all (from
they only deal with C ints, to 32-bit floats, to non-IEEE 754 doubles,
....). This is just one of those things.

In general you really must read the docs for any JSON implementations you
intend to use. It's quite obnoxious that this is so. But it is so.

[stedolan]

Yes, this is annoying. I don't really want to add 64-bit ints, because that just moves the screwup further away (64bit int overflow still exists). I would quite like to add proper bigint support at some stage, though.

I know you don't have control over this data format, but in general IDs should be strings, not numbers.

[nicowilliams]

Close as dup of #218.