Unsupported Category dtype as_type

[dorisjlee]

There is a bug when using Lux with this example from Datashader.

import pandas as pd
import numpy as np
from collections import OrderedDict as odict

num=10000
np.random.seed(1)

dists = {cat: pd.DataFrame(odict([('x',np.random.normal(x,s,num)), 
                                  ('y',np.random.normal(y,s,num)), 
                                  ('val',val), 
                                  ('cat',cat)]))      
         for x,  y,  s,  val, cat in 
         [(  2,  2, 0.03, 10, "d1"), 
          (  2, -2, 0.10, 20, "d2"), 
          ( -2, -2, 0.50, 30, "d3"), 
          ( -2,  2, 1.00, 40, "d4"), 
          (  0,  0, 3.00, 50, "d5")] }

df = pd.concat(dists,ignore_index=True)
df["cat"]=df["cat"].astype("category") #  If commented, the df.intent=vis line doesn't break

df  #Select and export the scatterplot vis (big circular blob)

vis = df.exported[0]

df.intent = vis 
df # This errors on the key 'cat' which is a categorical

We should look into supporting Categorical data types in Pandas. More importantly, we should look into whether bugs show up when we perform astype operations for other data types.

[westernguy2]

This is another bug where Lux isn't supporting CategoricalDtype. Specifically, both cut and qcut output a column of CategoricalDtype.

import pandas as pd
import numpy as np
import lux

url = 'https://github.com/lux-org/lux-datasets/blob/master/data/cars.csv?raw=true'
df = pd.read_csv(url)
df['Year'] = pd.to_datetime(df['Year'], format='%Y')

df["Weight"] = pd.qcut(df["Weight"], q = 3)

import pandas as pd
import numpy as np
import lux

url = 'https://github.com/lux-org/lux-datasets/blob/master/data/cars.csv?raw=true'
df = pd.read_csv(url)
df['Year'] = pd.to_datetime(df['Year'], format='%Y')

df["Weight"] = pd.cut(df["Weight"], bins = [0, 2500, 7500, 10000], labels = ["small", "medium", "large"])

Both code blocks have the same error:

[westernguy2]

Another bug for Category dtype is for this example:

import pandas as pd
import numpy as np
import lux

url = 'https://github.com/lux-org/lux-datasets/blob/master/data/cars.csv?raw=true'
df = pd.read_csv(url)
df['Year'] = pd.to_datetime(df['Year'], format='%Y')

new_df = df.drop([0, 1, 2], axis = "rows")
pd.merge(df, new_df, how = "left", indicator = True)

The indicator argument creates a column of dtype category called _merge, resulting in this bug:

This happens because Lux can't handle category dtype, so it doesn't assign it a data type.

[dorisjlee]

This is resolved with @westernguy2's new PRs!