NaN values don't show in categorical column

[F1nnM]

Hi,

I've come across a weird bug with the parallel plot:
One of my float columns/axis contains NaN-values.
When plotted with Hiplot only sometimes the entry "nan/inf/null" appears on the axis; more exactly it only appears if there are at least 6 unique values other than NaN in that column.
For example, if all entries in that column contain only the values [nan 3. 5. 7. 9. 15. 30. ] the "nan/inf/null" entry shows correctly.
If however I replace all 30's with 15's without changing anything else, the entry doesn't show up.

Is that a bug, or am I overseeing something on my side?

[danthe3rd]

Hi @F1nnM - and thank you for reporting this bug!

It looks weird indeed - one possible explanation that comes to my mind: when your column does not have a lot of different values, it might no longer be considered as a numerical value, but as a categorical value (you can check/modify that by right clicking a column's name above the parallel plot). Changing back to numerical could be a workaround - you can also force the column type programmatically
Now there might be a bug with how we display NaN values for categorical columns.

[F1nnM]

Indeed, that seems to be the issue; it's considered categorical.
Thank you for that info and the workaround!

[F1nnM]

So the NaN handling seems weird in general, I tried to fix it, but it's to complex to understand for me.
I might try again somewhen, should it not be fixed by then.

What I found out so far:
In the standalone version, just running the Hiplot command, the NaN values are shown in categorical columns, but numerical columns with nans appear completely empty.

params_max_columns categorical, standalone version

params_max_columns numerical, standalone version

In the streamlit component, categorical columns don't show NaNs, only the other values. But it correctly handles NaNs in numerical columns.

params_max_columns categorical, Streamlit version

params_max_columns numerical, Streamlit version

This time I can even supply the dataset I used.
Random-Forest-Example.csv.txt

For the Hiplot in Streamlit with this dataset you can also check here

[F1nnM]

Half-asleep idea: Probably the default csv handler already fills the NaNs with strings "NaN", does it?
That would explain it.
If that's the case, maybe the change in my linked PR is actually correct.

[danthe3rd]

Since it's a problem that only happens with streamlit, I'm wondering if the experimental _compress flag might cause it (https://facebookresearch.github.io/hiplot/tuto_streamlit.html#improving-performance-with-streamlit-caching-experimental).
Otherwise, the code in both cases should be exactly the same...
I'll try to have a look at it later this week.

[danthe3rd]

I've started this PR (still WIP) that should fix it, but I don't have time to extensively test it now - hopefully next week

[F1nnM]

Ah, awesome!

[danthe3rd]

should be fixed - I just merged #180
I'm pushing version 0.1.25