DataContractJsonSerializer: I-JSON Option Request

[cyberphone]

Related: https://github.com/javaee/jsonb-spec/issues/80

[karelz]

@cyberphone what is the bug / feature ask? Can you please describe it?

[cyberphone]

@karelz Sure, I-JSON (https://tools.ietf.org/html/rfc7493) is a scheme which uses the part of JSON supported by ECMAScript where the treatment of the JSON Number is the most important one. The purpose is simply to be able to interop with other JSON implementations. As can be seen in the Java bug report, Oracle indeed have an I-JSON option, but unfortunately it doesn't work.

Serialization of a long would using I-JSON, be "value" rather than value.

[huanwu]

@yujayee Can you look at this issue with @mconnew?

[cyberphone]

@karelz @mconnew @yujayee I wouldn't bother about this request because a recent investigation of mine indicates that the JSON community is unable to fix this in a meaningful way:
https://github.com/cyberphone/I-JSON-Number-System#existing-solutions

[karelz]

Closing as Won't Fix per @cyberphone's suggestion. @yujayee feel free to reopen if you disagree.

[mconnew]

Here's the relevant paragraph from the I-JSON RFC:

An I-JSON sender cannot expect a receiver to treat an integer whose
absolute value is greater than 9007199254740991 (i.e., that is
outside the range [-(253)+1, (253)-1]) as an exact value.

For applications that require the exact interchange of numbers with
greater magnitude or precision, it is RECOMMENDED to encode them in
JSON string values. This requires that the receiving program
understand the intended semantic of the value. An example would be
64-bit integers, even though modern hardware can deal with them,
because of the limited scope of JavaScript numbers.

I'm not convinced by the document that @cypherphone wrote. Java JSON-B predates I-JSON but was created as a part of the Java spec so can't really be changed now. I suspect there will be a defacto third party library (if there isn't already) which handles this, just like JodaTime is pretty universally used for datetime handling as the Java built in one is insufficient. JSON.NET puts everything in as a number but will lose precision when read by JavaScript and maybe other platforms. Not ideal, but easily worked around with a custom formatter. bignumber.js is a JavaScript library so has the limitations of JavaScript so they use the format that I-JSON specifies. The next two aren't relevant. "W3C Payment Request" is encoding a monetary value and not an integer. As the JSON number format is just an IEEE-754 64 bit floating point number, it is ill suited to use with money as it can't precisely represent even low precision numbers. Basically, it's a really bad idea to use a floating point for money. So that doesn't conflict here. "Open Banking UK" also falls into that same bucket. So you are left with Twitter, which is being pragmatic and duplicating the number in 2 fields, one of which is a JSON number and will lose precision on some platforms (including javascript), and the other is in the I-JSON format as a string.

Also, saying that because there are multiple conflicting ways of doing it out there which resulted in the creation of a standard to try and fix the issue would seem to me to be justification for implementing the standard.

[cyberphone]

@mconnew That there are workarounds is true, but the absence of any kind of standard makes it (IMO) rather pointless providing an option. JSON-B is only compatible with itself and so is every other implementation out there. JSON-B is in its latest incarnation fully I-JSON compatible by default but I think their solution is pretty awful (small BigInteger numbers are written as Number and large as String).

My document indeed contained a proposal: https://github.com/cyberphone/I-JSON-Number-System#extended-numeric-data-types but it has currently no support from the nowadays concluded JSON WG. If you would consider an I-JSON option, I would first try to get an opinion by MSFT people involved in the IETF. The primary Json.NET author wrote that they do not consider Node.js/Browser/ES6 compatibility important, "you just have to write a converter".

BTW, I have implemented Java, C#/.NET, ES6, and Python3 support for another (by me) proposed JSON standard, building on I-JSON:
https://tools.ietf.org/html/draft-rundgren-json-canonicalization-scheme-01
https://github.com/cyberphone/json-canonicalization#json-canonicalization

[mconnew]

@cyberphone, the reason I think something needs to be done here is because DataContractJsonSerializer adds it's own proprietary mechanism for longs which isn't listed in your document. Although direct support for signing JSON isn't something which I believe would ever become a goal for DCJS, I would like to get it to the point where it can generate a JSON document which can be easily signed. I.E. ordered fields, correct representations of the data etc. So maybe the title of this bug needs to be changed so it's not specifically about I-JSON, but I would like to improve the output and parseability of JSON containing longs and big integers. I see two options, pick a mechanism which there's good support for in popular JWS stacks or hold off until there's more consensus. The thing with consensus is it never happens if everyone is sat waiting for others to go first. gRPC makes use of JWS to register for callbacks with HTTP/2, so whatever they do might be a good wagon to hitch this cart to.

[cyberphone]

Hi @mconnew,
Of course I'm not opposed to any initiative from your side. Although I indeed started with this from a signature context, at the lowest level it is really about JS compatibility. Ordered fields may even be something certain applications do not want since some people [unfortunately] do assumptions about the order and thus specify their own ordering. Maintaining JS compatibility may require the latter as well.

Now to signatures. JWS does not really sign JSON but arbitrary binary data encoded in Base64Url which was motivation for my signature project. I have published an Internet-Draft (https://tools.ietf.org/html/draft-erdtman-jose-cleartext-jws-00) together with MSFT's Mike Jones who is the primary author of JWS. However, I have later found that the predictive parsing scheme used in ECMAScript will unlikely ever get vendor support outside of the JS world, so I recently started with another approach which does not need any vendor support at all to work. It would though be trivial to add built-in support with the exception of Numbers. I ported 2000 lines of Java to C# to get this part right. It is painfully slow but correct. I believe this part belongs more to the double.ToString() method than to DCJS.

Just to make things even more fun I have combined JWS with this scheme:
https://github.com/cyberphone/jws-jcs#combining-detached-jws-with-jcs-json-canonicalization-scheme

Cheers,
Anders

[cyberphone]

This topic has now reached an unprecedented level of weirdness:
tc39/proposal-bigint#24 (comment)

[mconnew]

@cyberphone, it seems like there's still a lot up in the air around this issue. Any changes would require a new API to opt in (which itself isn't a problem), but this could have problems if the wind changes again and things normalize differently. So I'm going to close this issue but feel free to reopen/open a new issue once the state of things has settled on a "correct" solution.