-
Notifications
You must be signed in to change notification settings - Fork 220
JSON parser has issues with small f64 vales using scientific notation #1426
Comments
Seems that the issue is possibly in the underlying crate used So either
fn infer_number(n: &Number) -> DataType {
match n {
Number::Float(..) => DataType::Float64,
Number::Integer(_, b) => {
if !b.is_empty() && b[0] == b'-' {
DataType::Float64
} else {
DataType::Int64
}
}
} |
Will the sign check be enough though? That is, an IEEE 754 floating point is valid to 10^308 or something, which is larger than a 64 bit integer. |
Actually, fixing this in I've worked out a fix in |
fixed via jorgecarleitao/json-deserializer#20 |
I am cross referencing my original
polars
issue pola-rs/polars#7312 as some digging around trying to find the source has led to finding this.In https://github.com/jorgecarleitao/arrow2/blob/main/src/io/json/read/infer_schema.rs#L112-L117 it would seem that if the source json contains (small) floating point values like
2e-06
, then this will cause problems as it attempts to set the dtype asInt64
and failure occurs. This problem only seems to occur for values of the form2e-06
as opposed to2.0e-06
as the presence of the decimal point keeps things legit.The text was updated successfully, but these errors were encountered: