Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Trailing 0s are lost when typecasting is done to float #42246

Open
gosavi-as opened this issue Oct 28, 2020 · 9 comments
Open

Trailing 0s are lost when typecasting is done to float #42246

gosavi-as opened this issue Oct 28, 2020 · 9 comments

Comments

@gosavi-as
Copy link

@gosavi-as gosavi-as commented Oct 28, 2020

Go Version: go version go1.14.2 darwin/amd64

Does this issue reproduce with the latest release? Yes

For float64 values like 0.0 or 1.0 etc. values loose trailing 0s when declared as interface and typecasting is done.

Eg.

var x interface{}
x = 1.00

fmt.Println(float64(x.(float64)))

Expected output here is 1.00 however, output is 1. Missing trailing 0s for such numbers.

When such data is Json marshalled same thing happens. When this marshalled data is consumed, unmarshalled and loaded in Parquet which is a schema on write format there is an inconsistency because of this issue.

Expectation is when float64 typecasting / json marshalling is done it should return float value without stripping decimal precision even if it's 0.

@gosavi-as gosavi-as changed the title Trailing 0s are lost Trailing 0s are lost when typecasting is done to float Oct 28, 2020
@gosavi-as
Copy link
Author

@gosavi-as gosavi-as commented Oct 28, 2020

Languages like python do maintain such precision even though there are 0s after decimal.

@mpx
Copy link
Contributor

@mpx mpx commented Oct 28, 2020

This is how floating point values are converted to text by default - it is unrelated to interfaces.

Please check https://pkg.go.dev/fmt for details on formating floating point values.

You can try this example to experiment with formating: https://play.golang.org/p/q0aWVhkWtlb

f := 1.0                // float64
fmt.Println(f)          // 1
fmt.Printf("%v\n", f)   // 1
fmt.Printf("%f\n", f)   // 1.000000
fmt.Printf("%.2f\n", f) // 1.00

This is not a bug with Go. If you have any questions, please follow up on the golang-nuts mailing list.

@gosavi-as
Copy link
Author

@gosavi-as gosavi-as commented Oct 28, 2020

Primary issue is when json marshalling is done. This loses the precision and when unmarshalling is done there is no way to determine if such value is int or float.

@davecheney
Copy link
Contributor

@davecheney davecheney commented Oct 28, 2020

In json, all numbers are floating point

@gosavi-as
Copy link
Author

@gosavi-as gosavi-as commented Oct 28, 2020

Agreed on json part. However, why can't this maintain 1.0 in json.Marshal?

This is the case in Python. As mentioned this kind of stuff creates issue in Parquet which is a schema on write format. It can not differentiate between floats and ints. If value is maintained as 1.0 Parquet can determine it's a float value.

@davecheney
Copy link
Contributor

@davecheney davecheney commented Oct 28, 2020

However, why can't this maintain 1.0 in json.Marshal?

can you write a program that demonstrates the problem?

@mpx
Copy link
Contributor

@mpx mpx commented Oct 29, 2020

why can't this maintain 1.0

Floating point 1, 1.0, and 1.00 are all identical. The number of decimal digits provided is not stored in floating point - there is nothing to maintain.

As pointed out above, all JSON numbers are floating point, 1 and 1.0 must be parsed into the same floating point value. Relying on anything else is not compatible with JSON.

I'd recommend asking about the underlying problem you are trying to solve on a more appropriate forum. Please see https://github.com/golang/go/wiki/Questions

@gosavi-as
Copy link
Author

@gosavi-as gosavi-as commented Nov 2, 2020

Golang

package main

import (
	"encoding/json"
	"fmt"
)

type employee struct {
	Name string
	Stocks float64
}

func main() {
	
	e := employee{"name1", 30.0}

	fmt.Printf("%+v\n", e)
	data, _ := json.Marshal(e)
	fmt.Println(string(data))
}

Output

{Name:name1 Stocks:30}
{"Name":"name1","Stocks":30}

Python

import json

dict = {
    "Name": "name1",
    "Stocks": 30.0
}

print(dict)

d = json.dumps(dict).encode('utf-8')
dec = json.loads(d.decode('utf-8'))
print(dec)

Output

{'Stocks': 30.0, 'Name': 'name1'}
{u'Stocks': 30.0, u'Name': u'name1'}

Sample program to illustrate this. Json marshal of float value is not printing the decimal value if it is whole number like show in python and that's the case in most of the languages.

Because of this up stream big data system like in case of parquet which are deriving the schema from Json will look at the first value and will start thinking it as int rather than float. When it receives the subsequent Json with float values like 30.001, then it will convert that as float; which creates inconsistency in the schema.

@mpx
Copy link
Contributor

@mpx mpx commented Nov 3, 2020

I'm not sure "most languages" keep trailing period for floating point numbers which represent integers. Javascript is a notable exception..

> console.log(1.0)
1

Go JSON marshalling has previously been changed to encode floating point numbers similar to Javascript (#6384, #14135).

Interpreting JSON numbers as different types depending on formatting is fundamentally fragile (or really, anything other an a float64). JSON is a limited serialisation format - it does not offer an "integer" type. I would recommend more robust solutions:

  • Using a more expressive serialisation format (eg, Protobuf)
  • Using a fixed JSON schema (eg, field F should always decode as float64 - no auto-detection), or
  • Encoding the type separately in JSON. Eg: {"my_field": 1, "my_field_type": "float"}. json.Number can also be useful to represent numbers as a string. This way a field can represent the full range of float64/int64 values (or larger).

However, you can override Go formatting with a custom type if you need to interact with an (arguably buggy) JSON consumer that guesses the type based on formatting:

type foo struct {
    Num  floatFrac
}

type floatFrac float64

func (f floatFrac) MarshalJSON() ([]byte, error) {
    n := float64(f)
    if math.IsInf(n, 0) || math.IsNaN(n) {
        return nil, errors.New("unsupported number")
    }
    prec := -1
    if math.Trunc(n) == n {
        prec = 1 // Force ".0" for integers.
    }
    return strconv.AppendFloat(nil, n, 'f', prec, 64), nil
}

func main() {
    buf, _ := json.Marshal(&foo{Num: 1})
    fmt.Printf("%s\n", buf)
    // Output: {"Num":1.0}
}
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Linked pull requests

Successfully merging a pull request may close this issue.

None yet
4 participants