New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

No control on when float numbers are written with exponent #427

Closed
hagai-shatz opened this Issue Jan 21, 2017 · 22 comments

Comments

6 participants
@hagai-shatz

hagai-shatz commented Jan 21, 2017

When serialising JSON with float/double values, there is control on how many digits to write after the decimal point, with default of 2.
However, if the number is bigger than 1000 or smaller than 0.001 it will always be serialized with exponent: 1234.5 to 1.23e7

I can track this hard coded behaviour to: https://github.com/bblanchon/ArduinoJson/blob/master/include/ArduinoJson/Serialization/JsonWriter.hpp#L95

This is not in line with Arduino Serial.print that does not add exponent.

@bblanchon

This comment has been minimized.

Show comment
Hide comment
@bblanchon

bblanchon Jan 21, 2017

Owner

Hello Hagai,

What is the behavior that you expect?
Can you give concrete examples?

Owner

bblanchon commented Jan 21, 2017

Hello Hagai,

What is the behavior that you expect?
Can you give concrete examples?

@hagai-shatz

This comment has been minimized.

Show comment
Hide comment
@hagai-shatz

hagai-shatz Jan 21, 2017

I expect to have the same behaviour as Serial.print by default: write all the digits of the integer part. For example: the float number 123456.789 will be written as 123456.79 by default (since we have 2 digits after the decimal point by default).
It should be an option to provide the exponent threshold, not have it fixed to 3 (write with exponent if number is bigger than 10^3 or smaller than 10^-3). And the default threshold should be undefined to print as many digits as possible.

hagai-shatz commented Jan 21, 2017

I expect to have the same behaviour as Serial.print by default: write all the digits of the integer part. For example: the float number 123456.789 will be written as 123456.79 by default (since we have 2 digits after the decimal point by default).
It should be an option to provide the exponent threshold, not have it fixed to 3 (write with exponent if number is bigger than 10^3 or smaller than 10^-3). And the default threshold should be undefined to print as many digits as possible.

@bblanchon

This comment has been minimized.

Show comment
Hide comment
@bblanchon

bblanchon Jan 22, 2017

Owner

I'm sorry Hagai, but I disagree.
I think that ArduinoJson does a much better job that Serial.print()
Here are a few examples:

Input Serial ArduinoJson
3.14e10 ovf 3.14e10
3.14e-10 0.00 3.14e-10

Disabling the scientific notation by default is a bad idea.
Remember that a double goes up to 1.7976931348623157E+308.
Without scientific notation, this gives 179769313486232000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000.00

Not only this is a huge amount of wasted memory, but it can also be a security issue if the developer hadn't anticipated such a long string.

Making the threshold configurable would increase the code size, which is only acceptable if the benefit is real.

Can you tell me why 123456.79 is better than 1.2345679e5?

Owner

bblanchon commented Jan 22, 2017

I'm sorry Hagai, but I disagree.
I think that ArduinoJson does a much better job that Serial.print()
Here are a few examples:

Input Serial ArduinoJson
3.14e10 ovf 3.14e10
3.14e-10 0.00 3.14e-10

Disabling the scientific notation by default is a bad idea.
Remember that a double goes up to 1.7976931348623157E+308.
Without scientific notation, this gives 179769313486232000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000.00

Not only this is a huge amount of wasted memory, but it can also be a security issue if the developer hadn't anticipated such a long string.

Making the threshold configurable would increase the code size, which is only acceptable if the benefit is real.

Can you tell me why 123456.79 is better than 1.2345679e5?

@hagai-shatz

This comment has been minimized.

Show comment
Hide comment
@hagai-shatz

hagai-shatz Jan 22, 2017

Well... you are absolutely right.
My initial thought was to add precision parameter with a default value of 3 to keep the current behaviour (almost).
The code will look something like this, assuming another parameter uint8_t precision = 3:

    if (value > pow(10, precision) || value < pow(10, -precision) {
      powersOf10 = Polyfills::normalize(value);
      digits = precision - 1;
    } else {
      powersOf10 = 0;
    }

With default precision of 3 and digits set to 2 we get the same behaviour:

Input Output
12.1234 12.12
234.2 234.2
1234.56 1.23e3
0.0001234 1.23e-4

When you want higher precision you can specify it, for example precision of 6:

Input Output Exceptions
1234.56 1234.56
123456789.123 1.23456e8
0.00012345678 1.23456e-4
12345.1234 12345.12 Precision of 7 - digits is 2 by default
12.1234 12.12 Precision of 4 - digits is 2 by default

hagai-shatz commented Jan 22, 2017

Well... you are absolutely right.
My initial thought was to add precision parameter with a default value of 3 to keep the current behaviour (almost).
The code will look something like this, assuming another parameter uint8_t precision = 3:

    if (value > pow(10, precision) || value < pow(10, -precision) {
      powersOf10 = Polyfills::normalize(value);
      digits = precision - 1;
    } else {
      powersOf10 = 0;
    }

With default precision of 3 and digits set to 2 we get the same behaviour:

Input Output
12.1234 12.12
234.2 234.2
1234.56 1.23e3
0.0001234 1.23e-4

When you want higher precision you can specify it, for example precision of 6:

Input Output Exceptions
1234.56 1234.56
123456789.123 1.23456e8
0.00012345678 1.23456e-4
12345.1234 12345.12 Precision of 7 - digits is 2 by default
12.1234 12.12 Precision of 4 - digits is 2 by default
@hagai-shatz

This comment has been minimized.

Show comment
Hide comment
@hagai-shatz

hagai-shatz Jan 22, 2017

Can you tell me why 123456.79 is better than 1.2345679e5?

They both have the same precision so I don't really care, although the scientific representation consume additional 2 bytes: 'e5'.
The current code will output the number 123456.79 as 1.23e5 unless I set digits to a higher value.
Maybe I will take this workaround for the time being and set digits to 12.

hagai-shatz commented Jan 22, 2017

Can you tell me why 123456.79 is better than 1.2345679e5?

They both have the same precision so I don't really care, although the scientific representation consume additional 2 bytes: 'e5'.
The current code will output the number 123456.79 as 1.23e5 unless I set digits to a higher value.
Maybe I will take this workaround for the time being and set digits to 12.

@bblanchon

This comment has been minimized.

Show comment
Hide comment
@bblanchon

bblanchon Jan 22, 2017

Owner

That's true that those two additional bytes are annoying.
I'll see what I can do.

Meanwhile, you can use a RawJson() as a workaround.

Owner

bblanchon commented Jan 22, 2017

That's true that those two additional bytes are annoying.
I'll see what I can do.

Meanwhile, you can use a RawJson() as a workaround.

@hagai-shatz

This comment has been minimized.

Show comment
Hide comment
@hagai-shatz

hagai-shatz Jan 31, 2017

Specifying 12 digits is a good workaround for now: item.set("val", value, 12);

I had difficulties using RawJson() with the ESPAsyncWebServer since I had no good way to keep the String/char* in memory once outside the handler, and by the time the server was serializing the Json, the String/char* was released.

hagai-shatz commented Jan 31, 2017

Specifying 12 digits is a good workaround for now: item.set("val", value, 12);

I had difficulties using RawJson() with the ESPAsyncWebServer since I had no good way to keep the String/char* in memory once outside the handler, and by the time the server was serializing the Json, the String/char* was released.

@andrei-ivanov

This comment has been minimized.

Show comment
Hide comment
@andrei-ivanov

andrei-ivanov May 1, 2017

Any chance to get this fixed? :-)

andrei-ivanov commented May 1, 2017

Any chance to get this fixed? :-)

@bblanchon

This comment has been minimized.

Show comment
Hide comment
@bblanchon

bblanchon May 1, 2017

Owner

@andrei-ivanov Yes and No.

Yes, I'm going to change the current behavior.
No, I'll not add a third parameter to set(float,int).

Indeed, my plan is to remove the second parameter that specifies the number of decimal places.
This is an heritage from Print but it's not required anymore.
I want to remove this as I believe it's more a nuisance than a feature.

Owner

bblanchon commented May 1, 2017

@andrei-ivanov Yes and No.

Yes, I'm going to change the current behavior.
No, I'll not add a third parameter to set(float,int).

Indeed, my plan is to remove the second parameter that specifies the number of decimal places.
This is an heritage from Print but it's not required anymore.
I want to remove this as I believe it's more a nuisance than a feature.

@andrei-ivanov

This comment has been minimized.

Show comment
Hide comment
@andrei-ivanov

andrei-ivanov May 1, 2017

I just thought it would be nice to be able to have correct numbers serialized by default, no matter the notation and not having to use the RawJson workaround.

andrei-ivanov commented May 1, 2017

I just thought it would be nice to be able to have correct numbers serialized by default, no matter the notation and not having to use the RawJson workaround.

@bblanchon

This comment has been minimized.

Show comment
Hide comment
@bblanchon

bblanchon May 1, 2017

Owner

My goals is to get the right number of decimals, without asking the user to specify it.
Unfortunately, it's a notoriously hard problem, and I need to keep the code small.

Owner

bblanchon commented May 1, 2017

My goals is to get the right number of decimals, without asking the user to specify it.
Unfortunately, it's a notoriously hard problem, and I need to keep the code small.

@bblanchon

This comment has been minimized.

Show comment
Hide comment
@bblanchon

bblanchon May 20, 2017

Owner

Float serialization was completely rewritten in v5.10.0

Owner

bblanchon commented May 20, 2017

Float serialization was completely rewritten in v5.10.0

@bblanchon bblanchon closed this May 20, 2017

@andrewjaykeller

This comment has been minimized.

Show comment
Hide comment
@andrewjaykeller

andrewjaykeller Jun 17, 2017

On ArduinoJSON 5.8.4:

I have no ability to set the precision on a double that is an NTP time stamp in microseconds.

I want 14977139448720123.0 but am getting 1e51. How can i force this to be expanded.

Likewise I'm using this for the new [OpenBCI Wifi Shield](www.github.com/openbci/openbci_wifi] (to put brainwaves on the internet) and the high precision 24bit ADS values converted into floats are on the microvolt level so numbers are 0.00001234567 and these are coming over like 0.00.

I had ARDUINOJSON_USE_DOUBLE set as well.

Just upgraded to 5.10.1 so I'll post back on what this changes! Love this library! Want to make it work!

andrewjaykeller commented Jun 17, 2017

On ArduinoJSON 5.8.4:

I have no ability to set the precision on a double that is an NTP time stamp in microseconds.

I want 14977139448720123.0 but am getting 1e51. How can i force this to be expanded.

Likewise I'm using this for the new [OpenBCI Wifi Shield](www.github.com/openbci/openbci_wifi] (to put brainwaves on the internet) and the high precision 24bit ADS values converted into floats are on the microvolt level so numbers are 0.00001234567 and these are coming over like 0.00.

I had ARDUINOJSON_USE_DOUBLE set as well.

Just upgraded to 5.10.1 so I'll post back on what this changes! Love this library! Want to make it work!

@devyte

This comment has been minimized.

Show comment
Hide comment
@devyte

devyte Jun 17, 2017

devyte commented Jun 17, 2017

@andrewjaykeller

This comment has been minimized.

Show comment
Hide comment
@andrewjaykeller

andrewjaykeller Jun 20, 2017

We want to be able to stream decimal values of a 24bit analog to digital converter. So we for example send a JSON object directly to the browser over web sockets. See #13 for the required decimal...

Snipit from issue:

{
"data":[ 7.056745022195285, 4.754953953750924, ...],
"timestamp": 1497479774194733.0
}

andrewjaykeller commented Jun 20, 2017

We want to be able to stream decimal values of a 24bit analog to digital converter. So we for example send a JSON object directly to the browser over web sockets. See #13 for the required decimal...

Snipit from issue:

{
"data":[ 7.056745022195285, 4.754953953750924, ...],
"timestamp": 1497479774194733.0
}
@devyte

This comment has been minimized.

Show comment
Hide comment
@devyte

devyte Jun 20, 2017

devyte commented Jun 20, 2017

@andrewjaykeller

This comment has been minimized.

Show comment
Hide comment
@andrewjaykeller

andrewjaykeller Jun 20, 2017

so you think convert to nano volts (we are measuring micro volts) and then converting to longs?

andrewjaykeller commented Jun 20, 2017

so you think convert to nano volts (we are measuring micro volts) and then converting to longs?

@andrewjaykeller

This comment has been minimized.

Show comment
Hide comment
@andrewjaykeller

andrewjaykeller Jun 20, 2017

i understand how floats/doubles work, using a text transfer protocol I accept the loss of precision, but how can I say got out 10 places?

andrewjaykeller commented Jun 20, 2017

i understand how floats/doubles work, using a text transfer protocol I accept the loss of precision, but how can I say got out 10 places?

@bblanchon

This comment has been minimized.

Show comment
Hide comment
@bblanchon

bblanchon Jun 25, 2017

Owner

ArduinoJson 5.10 supports up to 9 digits after the decimal point.
For that you need to enable support for doubles with ARDUINOJSON_USE_DOUBLE

Owner

bblanchon commented Jun 25, 2017

ArduinoJson 5.10 supports up to 9 digits after the decimal point.
For that you need to enable support for doubles with ARDUINOJSON_USE_DOUBLE

@andrewjaykeller

This comment has been minimized.

Show comment
Hide comment
@andrewjaykeller

andrewjaykeller Jun 25, 2017

Thank you! I am using longs where my voltages are in nano volts so I move the precision to the other side which should work just fine. I love ArduinoJson! Keep up the great work!

andrewjaykeller commented Jun 25, 2017

Thank you! I am using longs where my voltages are in nano volts so I move the precision to the other side which should work just fine. I love ArduinoJson! Keep up the great work!

@tsaitsai

This comment has been minimized.

Show comment
Hide comment
@tsaitsai

tsaitsai Oct 8, 2017

Just wanted to chime in to say that the higher precision support for ArduinoJson makes GPS coordinates much easier to work with. Thanks for the update.

tsaitsai commented Oct 8, 2017

Just wanted to chime in to say that the higher precision support for ArduinoJson makes GPS coordinates much easier to work with. Thanks for the update.

@bblanchon

This comment has been minimized.

Show comment
Hide comment
@bblanchon

bblanchon Oct 9, 2017

Owner

Thanks, @tsaitsai! That kind of comment is scarce.
It's good to get positive feedback, especially since it's been a lot of work to implement this feature correctly.
By the way, I documented the process in this blog post:
https://blog.benoitblanchon.fr/lightweight-float-to-string/

Owner

bblanchon commented Oct 9, 2017

Thanks, @tsaitsai! That kind of comment is scarce.
It's good to get positive feedback, especially since it's been a lot of work to implement this feature correctly.
By the way, I documented the process in this blog post:
https://blog.benoitblanchon.fr/lightweight-float-to-string/

Repository owner locked and limited conversation to collaborators Sep 21, 2018

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.