Frequencies should show doubles not ints

[mchav]

Is your feature request related to a problem? Please describe.
The frequencies function has its numerical fields as integers. This means we don't see accurate percentages.

ghci> df <- D.readCsv "./data/housing.csv"
ghci> D.frequencies "ocean_proximity" df
----------------------------------------------------------------------------
index |   Statistic    | <1H OCEAN | INLAND | ISLAND | NEAR BAY | NEAR OCEAN
------|----------------|-----------|--------|--------|----------|-----------
 Int  |      Text      |    Int    |  Int   |  Int   |   Int    |    Int
------|----------------|-----------|--------|--------|----------|-----------
0     | Count          | 9136      | 6551   | 5      | 2290     | 2658
1     | Percentage (%) | 44        | 31     | 0      | 11       | 12

In this example, island is clearly not 0%.

Describe the solution you'd like
We should change these to doubles.

[kayvank]

I want to work on this, please.
I did a bit of digging. The fix has 2 parts:

1. convert from Int to Double
2. correctly render,show the Double

For part one, the fix is in:
module DataFrame.Operations.Statistics function frequencies
which will result in the not-so-nice presentation:

---------------------------------------------------------------------------------------------------------------------------------
index |   Statistic    |     <1H OCEAN      |       INLAND       |        ISLAND        |      NEAR BAY      |     NEAR OCEAN
------|----------------|--------------------|--------------------|----------------------|--------------------|-------------------
 Int  |      Text      |       Double       |       Double       |        Double        |       Double       |       Double
------|----------------|--------------------|--------------------|----------------------|--------------------|-------------------
0     | Count          | 9136.0             | 6551.0             | 5.0                  | 2290.0             | 2658.0
1     | Percentage (%) | 44.263565891472865 | 31.739341085271317 | 2.422480620155039e-2 | 11.094961240310077 | 12.877906976744185

This brings up a questions for me:

1. It is a bit confusing to have a column heading of Double for Count. And less than ideal to have a heading of Double for Percentages . What should be the correct behavior?

[mchav]

I think Double is passable for percentages but you're right it kinda sucks for count. Maybe the idea would be:

---------------------------------------------------------------------------------------------------------------------------------
index |   Statistic    |     <1H OCEAN      |       INLAND       |        ISLAND        |      NEAR BAY      |     NEAR OCEAN
------|----------------|--------------------|--------------------|----------------------|--------------------|-------------------
 Int  |      Text      |       Value        |       Value        |        Value         |        Value       |        Value
------|----------------|--------------------|--------------------|----------------------|--------------------|-------------------
0     | Count          | 9136               | 6551               | 5                    | 2290               | 2658
1     | Percentage     | 44.26%             | 31.73%             | 0.02%                | 11.09%             | 12.88% 

Where Value is actually RowValue

[kayvank]

I am afraid I couldn't find a way to use RowValue as you pointed out above for the frequency calculation, and will need a bit of guidance there, please.
I have the draft PR #44 that converts the frequency to Double as pointed out above the output doesn't look as nice

[mchav]

So the problem is that we effectively want two different types for the column. An int for count and a rounded double/text for percentage.

Since lists are homogenous on their types we can't do: [2000 :: Int, 0.5 :: Double]

But that's essentially what we want to put into the insertVector function.

We can use a type like:

data Value = IntValue i | DoubleValue d

[IntValue 2000, DoubleValue 0.5]

And that's essentially what RowValue is.l and it provides a toRowValue function is a effectively a constructor for this type using GADTs.

I learnt this approach here: https://hengchu.github.io/posts/2018-05-09-type-lists-and-type-classes.lhs.html

Of course we could also make a heterogeneous list for this use case but maybe that's overkill.

What's your preference?

[kayvank]

Thank you for the link; it was very informative indeed. I think I need to read it a couple more times to get a better appreciation for it.
I also tend to think the HList is an overkill for this one use case.
I'm pretty busy this weekend with personal stuff, but I plan on getting back at it late Sunday or Monday.

[kayvank]

Using the RowValue as suggested above and implementing the Read for the RowValue resulted:

---------------------------------------------------------------------------------------------------------------------------------
index |   Statistic    |     <1H OCEAN      |       INLAND       |        ISLAND        |      NEAR BAY      |     NEAR OCEAN
------|----------------|--------------------|--------------------|----------------------|--------------------|-------------------
 Int  |      Text      |      RowValue      |      RowValue      |       RowValue       |      RowValue      |      RowValue
------|----------------|--------------------|--------------------|----------------------|--------------------|-------------------
0     | Count          | 9136.0             | 6551.0             | 5.0                  | 2290.0             | 2658.0
1     | Percentage (%) | 44.263565891472865 | 31.739341085271317 | 2.422480620155039e-2 | 11.094961240310077 | 12.877906976744185

It's less than ideal that the Count row is in Double, but I think this will solve this issue as it is described.

[kayvank]

@mchav since #44 is merged, may we close the issue, please?