.describe() doesn't give expected output instead gives nan

[ninjakx]

Expected output (using pandas):

	bill_length_mm	bill_depth_mm	flipper_length_mm	body_mass_g	year
count	342.000000	342.000000	342.000000	342.000000	344.000000
mean	43.921930	17.151170	200.915205	4201.754386	2008.029070
std	5.459584	1.974793	14.061714	801.954536	0.818356
min	32.100000	13.100000	172.000000	2700.000000	2007.000000
25%	39.225000	15.600000	190.000000	3550.000000	2007.000000
50%	44.450000	17.300000	197.000000	4050.000000	2008.000000
75%	48.500000	18.700000	213.000000	4750.000000	2009.000000
max	59.600000	21.500000	231.000000	6300.000000	2009.000000

Danfo.js is giving me:

	bill_length_mm	bill_depth_mm	flipper_length_mm	body_mass_g	year
count	342	342	342	342	344
mean	43.921928	17.151169	200.915207	4201.754395	2008.028809
std	NaN	NaN	NaN	NaN	0.818356
min	32.099998	13.1	172	2700	2007
median	44.45	18	202	4387.5	2008
max	59.599998	21.5	231	6300	2009
variance	NaN	NaN	NaN	NaN	0.669706
max	59.600000	21.500000	231.000000	6300.000000	2009.000000

Even median and 50% have some difference.

You can run the same csv here:
https://observablehq.com/@visnup/hello-danfo-js

[ninjakx]

So pandas std uses ddof as 1 and is taking care of nan values.

ddof int, default 1
Delta Degrees of Freedom. The divisor used in calculations is N - ddof, where N represents the number of elements.

I have to look at math.js to fix the cases containing nan values.
Simple solution is to remove all the nan values then it will give the same value.

[risenW]

@ninjakx Thanks for raising this issue.

• First, you need to update your version of danfojs to the latest (0.2.3).
• Secondly, as of 0.2.0 upward, we already take care of NaN values, by removing them before making calculations.
• Thirdly, where I think we can make updates is the inferring of dtypes. At the moment we use the first 10 values to infer dtype, and if there are missing values in those first 10 fields, the dtype is automatically set to string.
• Finally, there's definitely a slight bug in the median function. At the moment we use a custom median function. And I'm surprised why we didn't just use the median function in mathjs 🙂

At the moment, you can do the following:

So if you look at the dtypes of your data:

async function load_process_data() {
let df = await dfd.read_csv("https://static.observableusercontent.com/files/7a8dbde13ff5fc1b27de1fcd824956a65c4029682927f8728f625cccdb804dea33e248c9dd96660f5e33f308a0bae5e9067db3ba53dcead095aed8990b3262a1?response-content-disposition=attachment%3Bfilename*%3DUTF-8%27%27penguins%25401.csv")
  df.ctypes.print()
  let float_cols = [
    "bill_length_mm",
    "bill_depth_mm",
    "flipper_length_mm",
    "body_mass_g",
  ]

  float_cols.forEach((col)=>{
    df = df.astype({ column: col, dtype: "float32" });
  })
     
  df.ctypes.print()
  df.describe().print()
}

And yes, we still have the median bug, which I'll fix immediately.

Thanks once again for this!

[ninjakx]

@risenW :

Thirdly, where I think we can make updates is the inferring of dtypes. At the moment we use the first 10 values to infer dtype, and if there are missing values in those first 10 fields, the dtype is automatically set to string.

If there is nan,None, NaN or blank rather than treating them as string they should be treated as float and yes the dtype would become float then instead of int but that will solve the inference type thing.

If any string other than those NaN are present they should make the column as object(string) type.

[risenW]

@risenW :

Thirdly, where I think we can make updates is the inferring of dtypes. At the moment we use the first 10 values to infer dtype, and if there are missing values in those first 10 fields, the dtype is automatically set to string.

If there is nan,None, NaN or blank rather than treating them as string they should be treated as float and yes the dtype would become float then instead of int but that will solve the inference type thing.

If any string other than those NaN are present they should make the column as object(string) type.

Is there a specific reason why NaN types should be set to float instead of strings? I would genuinely love to know, any pointers

[risenW]

Also just to mention, in this fix #147 I updated the code to properly detect dtypes with NaN:

So the following holds:

 [ [ true, false, true ], [ "boy", "girl", "boy" ], [ NaN, undefined, NaN ] ] ==> [ 'boolean', 'string', 'string' ];

[ [18.7, 17.4, 18, NaN, 19.3],[ 20, NaN, 19, 18, 20 ]] ==> [ 'float32', 'int32' ];

[ninjakx]

@risenW : because of some IEEE standard.
In pandas NaN are treated as float because of the same. Don't know about the other language.

source

[risenW]

@risenW : because of some IEEE standard.
In pandas NaN are treated as float because of the same. Don't know about the other language.

source

Thanks for the pointer. Will definitely update this.

[ninjakx]

@risenW : I see some issues with the functions like reset_index(). I will raise it.
I love pandas actually so I would like to contribute. I know javascript (basic level) so If there are some bugs left related to pandas functionality, I can try to resolve it during weekends.