Review / fix behavior of json schema inference for numbers with trailing decimal point (e.g. 2.)

[alamb]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
This is a follow on to this conversation #831 (comment) on #831 . Specifically, the question is what to do with numbers like 2.

Today the JSON parser schema inference treats numbers like 2. as Utf8 (aka as strings).

As @brianrackle and @jimexist noted, this could be treated as either Int or a Float and it is not clear exactly what the behavior should be

Describe the solution you'd like
Propose a behavior and implement any code / test changes necessary

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

[brianrackle]

Investigated JSON further and 2. and .2 are not valid tokens in JSON. Since this isn't valid JSON I don't think these values need any special handling to interpret them as numbers. In these cases, serde rightfully returns an error.

json spec https://www.json.org/json-en.html

[brianrackle]

But I think this issue should be about CSV handling of these values and was mistakenly worded to refer to JSON.

For CSV there is not a formal spec, so the expected behavior is undefined. My thoughts are that since 2. can be interpreted by an Int then it should be and the usual schema inference rules will upgrade from Int to Float only if there are other values in the column which can only be represented by Float.

[brianrackle]

But I think this issue should be about CSV handling of these values and was mistakenly worded to refer to JSON.

For CSV there is not a formal spec, so the expected behavior is undefined. My thoughts are that since 2. can be interpreted by an Int then it should be and the usual schema inference rules will upgrade from Int to Float only if there are other values in the column which can only be represented by Float.

After playing with the solution some more inferring as an Int would mean changing the native parse logic which currently does not support converting "2." to an Int. So I followed the path of least resistance and went ahead with doing a conversion which is already supported and infer "2." as a Float so that the existing native parsing logic can be used.

[alamb]

So I followed the path of least resistance and went ahead with doing a conversion which is already supported and infer "2." as a Float so that the existing native parsing logic can be used.

I think that makes sense 👍