-
-
Notifications
You must be signed in to change notification settings - Fork 6.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Accounting for arbitrary precision numerical literals #1849
Comments
The SAX parser could help. A SAX parser does not create an in-memory representation of the parsed input, but only calls certain functions each time a parse event is encountered. The interface is documented here: https://nlohmann.github.io/json/structnlohmann_1_1json__sax.html For you, function A simple implementation of a SAX parser can be found here: https://github.com/nlohmann/json/blob/develop/include/nlohmann/detail/input/json_sax.hpp#L631. It is the code used for Let me know if you need further assistance. |
Hi, thanks for your response! Looking through the SAX api and example in json_sax, its clear to me how to implement the functions defined by the sax api to determine if a float round trips. However, its not clear to me how to actually use the json_sax class in order to construct a json object - there is a lot of logic that goes into that, and I'm not sure how to easily do it. It seems like I would have to rewrite most of basic_json, but surely I'm wrong on that? |
You can copy/paste the |
aha! this looks promising, thanks so much :)) As a newbie, I would never be able to figure this out on my own. Is there a way that documentation for this could be added? I'd offer to do it but I do not believe I am qualified or should be trusted lol. Specifically for being able to keep the default json parsing mechanism, but being able to "override" the default functionality |
The documentation for Maybe its because I'm new to c++, or that I'm unfamiliar with this library, but giving an example of using The rest of the documentation was really intuitive and easy to understand, but this seems like the sort of thing one has to dig through code or ask the author to know how to do without documentation for this |
This is a rather specific usecase, and I would be happy for any proposal (PRs welcome) how to extend https://github.com/nlohmann/json#sax-interface. |
I would be happy to think about an alternative API or a canonical example I could come up with. I feel more comfortable contributing to your library because I believe you have testing infrastructure in place to prevent bugs introduced from C++ beginners like me (only somewhat joking). I'll think on the matter more once I implement a solution to my current use case. |
Actually, I have a better idea than trying to revise the SAX api or case-specific documentation. My use case can more generally be stated as follows: When trying to get (or set!) the value of a numerical json field, instead of getting the value as a particular c++ type, such as with Why would someone This causes issues if there isn't a way to unable users to address this discrepancy when they need to. In my case, it manifests itself as me wanting to read the raw string literal of the number to ensure that the user can't input a number of such high precision that it won't round-trip to a double losslessly. In #1421 , it was that the user cares about maintaining the representation of the original floating point number without tacking on any extra zeros. Looking through the issue history of the repo, there were several other issues on round tripping floats, although I don't know if the proposed fixes applied to all the use cases. Not accounting for this discrepancy between c++ primitives and JSON primitives makes this library unable to allow users to handle lossless serialization and de-serialization of the subset of valid JSON files that have numerical literals of a higher precision than that of c++ primitives. I think this is not a niche use case but rather functionality that users would appreciate. Think about the vast quantities of people that use JSON for scientific computing, or financial data, or (in my case) just want a way to sanitize user inputted floats so that they will serialize back to the same decimal representation. The good news is that there is probably an easy API fix for all of this (and its not SAX :P )
I think the following usage fulfills these requirements, inspired by #1421 (comment): json j = R"({
"too_precise_for_double": 123456789.123456789
})"_json;
// Accessing JSON fields
json::numerical d_literal = j.at("too_precise_for_double").get<json::numerical>();
double d_double = j.at("too_precise_for_double").get<double>();
// prints `123456789.123456789` (note no string quotes, because its not a string)
std::cout << std::fixed << d_literal << std::endl;
// prints `123456789.123457` (note truncation due to limited precision double)
std::cout << std::fixed << d_double << std::endl;
try {
// Throws an exception, because the value isn't a string
string d_string = j.at("too_precise_for_double").get<string>();
} catch (...) {}
// Setting JSON fields
json j2;
j2['new_numerical_literal'] = json::numerical("987654321.987654321");
// prints `{"new_numerical_literal":987654321.987654321}`
cout << std::fixed << j2 << endl;
Would this be something fairly feasible for me as a c++ novice to implement? Keep in mind that even looking through the codebase is very overwhelming for me, neverless trying to modify it. |
Just checking in to see what you think about this proposal. Is this a good solution? Want to check in with you before I go and try to get a pull request working |
So you would store an additional string with each number? |
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. |
Hi, I'm also encountering this problem of parsing the numeric value when it is not representable as If the original string cannot be stored along the numeric interpretation, is there any way to use e.g. GMP's |
@fbrausse Can you work around the trivially destructable issue with wrapping your But in general I think there should be a way of retrieving the string representation at parse time when fetching a value. |
Hi, if I manage the lifetime somehow myself, probably. At the moment, it is not clear to me what the lifetime of I understand the defaults of Indeed, somehow accessing the string representation in the source would be very helpful; it might also open up the possibility to use different interpretations for the user - e.g. if accuracy is required I could imagine plugging in some decimal float type depending on the use case. |
A slightly different approach might also work: Internally store "JSON numbers" as you store "JSON strings", but the Do you know whether that would imply an API change or whether it would be an acceptable modification? |
Describe what you want to achieve.
I have a config struct that gets initialized via a json file. I want to make sure that any floats in the json file will round trip (convert from string-float-string) and remain identical. To enforce this, I need a way of seeing if the provided JSON-float (which is represented on the computer as a string) exceeds a certain number of digits of precision (in the case of string-float-string roundtrips, this precision is 15 decimal digits on my machine). Once I have a way to read the JSON-float as a string before it gets converted to a C++-float, I can throw a runtime error if the user tries to provide a JSON-float of too high a precision.
Describe what you tried.
I can control JSON serialization via
std::setprecision()
, but I cannot control JSON deserialization. I know that there is a SAX interface that looks like I might be able to get an event hook on when the JSON-float gets parsed into a string before conversion to a C++-float, but I don't know how to use it as the SAX documentation said that it doesn't handle the actual serialization and deserialization (I also don't know what SAX is).Describe which system (OS, compiler) you are using.
MacOS, Gcc-8
Describe which version of the library you are using (release version, develop branch).
master, v3.7.3
P.S: Very new to C++, your library is making me hate the language a little less :)
The text was updated successfully, but these errors were encountered: