Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Already on GitHub? Sign in to your account

RJSONIO::toJSON: issues with large numbers #141

Open
reinholdsson opened this Issue Jul 1, 2013 · 16 comments

Comments

Projects
None yet
8 participants
Contributor

reinholdsson commented Jul 1, 2013

I saw today that there are some limitations in RJSONIO when dealing with large numbers, e.g. datetimes.

Some examples:

fromJSON(toJSON(123456789))
# [1] 123460000

toJSON(1.35432e+12)
# [1] "[ 1.3543e+12 ]"

It's possible to extend the number of digits, but neither that is perfect:

toJSON(1234567890123456789, digits = 20)
# [1] "[   1234567890123456768 ]"  # see the last digits

toJSON(152.4, digits = 50)
[1] "[   152.400000000000005684341886080801486968994140625 ]"

@ramnathv @timelyportfolio do you have any suggestions on how to deal with this?

Joe Cheng (@jcheng5) created a pull request duncantl/RJSONIO#3 quite recently, so perhaps there is a solution on its way.

Owner

ramnathv commented Jul 1, 2013

Yes, the digits idea was just a monkey patch, and will not work in the long run. It would be nice if RJSONIO is patched based on the pull request made by @jcheng5. In the absence of that, we can see if we can extend the toJSON method in RJSONIO directly so that we have more control.

Contributor

reinholdsson commented Jul 1, 2013

Numerics are passed to the formatC function in base R, see https://github.com/duncantl/RJSONIO/blob/master/R/json.R#L165

Changing its attributes or use another formatter would probably help, e.g.

n <- c(123456789012345.12345, 123456789012345, 1234567890.123)
formatC(n, format = "fg", digits = 25)
# [1] "       123456789012345.125" "           123456789012345"
# [3] "1234567890.122999906539917"

formatC(n, format = "f", digits = 50, drop0trailing = T)
# [1] "123456789012345.125"              
# [2] "123456789012345"                  
# [3] "1234567890.1229999065399169921875"

But, I am not completely following on how R works with large numbers, and thus not sure on the optimal defaults.

jcheng5 commented Jul 2, 2013

There are precision limits on the JS side as well. From Chrome:

image

Why isn't the digits mechanism the right solution for the long run?

Owner

ramnathv commented Jul 2, 2013

Ah. Interesting. I wasn't aware of that. So using digits might after all be a long term solution.

ramnathv added a commit that referenced this issue Jul 4, 2013

Merge pull request #141 from edgester/fix2
Issue #133: fix blank metadata error

Hi, I've me the same problem. Can you please tell the right way to overcome such problem?
Do I have to translate long number into string before using toJson?

Contributor

reinholdsson commented Aug 10, 2013

@seregasheypak I convert them to strings (using "#! ... !#") as a temporary fix, see #145 (comment) or http://rcharts.io/viewer/?5911880. At least until we have implemented a better solution, e.g. overriding the S4 method as suggested by jcheng5.

I ve used impala as backend. I've explicitly casted long field to string in
sql query and rjsonio treated it as string. The problem is solved.
10.08.2013 22:04 ÐÏÌØÚÏ×ÁÔÅÌØ "Thomas Reinholdsson" <
notifications@github.com> ÎÁÐÉÓÁÌ:

@seregasheypak https://github.com/seregasheypak I convert them to
strings (using "#! ... !#") as a temporary fix, see #145#145 (comment)
http://rcharts.io/viewer/?5911880. At least until we have implemented a
better solution, e.g. overriding the S4 method as suggested by jcheng5.

Reply to this email directly or view it on GitHubhttps://github.com/ramnathv/rCharts/issues/141#issuecomment-22446370
.

@rohithv rohithv referenced this issue in wactbprot/R4CouchDB Dec 30, 2013

Closed

Large Number formatting #39

rohithv commented Dec 30, 2013

I don't know if I am the only one facing this issue. Googling got me here. Below is, not such a large number. But whenever it ends with a 5 preceded by 4 zeros , toJSON fails to convert it properly. Below is a sample code. Am I missing something?

library(RJSONIO)
a = 700005
b = toJSON(a)
b
[1] "[ 7.e+05 ]"
a = 700004
b = toJSON(a)
b
[1] "[ 7e+05 ]"

Owner

ramnathv commented Dec 30, 2013

@rohithv It is a question of precision and number of digits. Try toJSON(a, digits = 13) an you will get what is expected.

rohithv commented Dec 30, 2013

Yes. You are right. I posted this question here as the discussion thread was very appropriate.

Unfortunately I encounter this issue not directly but when I use R4CouchDB. I have posted this issue there as well. So hoping that this gets fixed in the core package itself.

Thanks for replying.

rohithv commented Dec 30, 2013

@ramnathv And this is also a weird case which seems to happen only the number is preceded with 4 zeros followed by a five. Replace five with 4 and you would get a valid JSON. In the previous case the JSON itself is invalid due to the decimal in the exponent form.

Contributor

timelyportfolio commented Dec 30, 2013

this StackOverflow discussion might also be helpful

rohithv commented Dec 31, 2013

@timelyportfolio Thank for the reply. However what I meant to say was, why was there as dot when the number was 700005 and no dot when the number was 700004. This conversion with dot was causing the subsequent JSON formatter to fail. Anyways, apparently this issue has been resolved in RJSONIO-1.0-3 build.

ttriche commented Dec 31, 2013

why not switch to the jsonlite fork in the future? Just curious.

--t

On Mon, Dec 30, 2013 at 9:06 PM, Rohith V notifications@github.com wrote:

@timelyportfolio https://github.com/timelyportfolio Thank for the
reply. However what I meant to say was, why was there as dot when the
number was 700005 and no dot when the number was 700004. This conversion
with dot was causing the subsequent JSON formatter to fail. Anyways,
apparently this issue has been resolved in RJSONIO-1.0-3 build.


Reply to this email directly or view it on GitHubhttps://github.com/ramnathv/rCharts/issues/141#issuecomment-31384137
.

Owner

ramnathv commented Dec 31, 2013

Yes. rCharts does plan to switch to jsonlite in the future. We are waiting for some performance issues to be resolved with jsonlite.

This is an embugerance. I can't change my input data so I'm having to correct it with perl -0777 -i -pe 's/: ([0-9]+),/: "$1",/igm' *.json. Would be really helpful if we could specify character format in fromJSON

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment