Error gracefully when on categorical matrices with missing data (segfault is bad).

[jtilly]

This produces a segfault:

import numpy as np
import pandas as pd

from quantcore.matrix import from_pandas
from quantcore.glm import GeneralizedLinearRegressor

df = pd.DataFrame(
    {
        "x1": [4, 4, 5, 6, 0, np.nan],
    }
)

y = np.array(
    [
        1.1,
        0.8,
        1.2,
        1.0,
        2.7,
        0.5,
    ]
)

split_matrix = from_pandas(df.astype("category"))
model = GeneralizedLinearRegressor(family="gamma", alpha=1.0)
model.fit(X=split_matrix, y=y)
model.coef_

This works:

split_matrix = from_pandas(df.fillna("missing").astype("category"))
model = GeneralizedLinearRegressor(family="gamma", alpha=1.0)
model.fit(X=split_matrix, y=y)
model.coef_
# array([ 0.15917267, -0.05053825,  0.00268371, -0.02201278, -0.08930535])

I think it's okay not to support this, but this should fail more graciously.

The underlying problem is that astype(category) doesn't represent missings as a separate level:

split_matrix
# [4.0, 4.0, 5.0, 6.0, 0.0, NaN]
# Categories (4, float64): [0.0, 4.0, 5.0, 6.0]  # <- 4 elements no 5
split_matrix.matvec(np.array([1, 2, 3, 4]))
array([2, 2, 3, 4, 1, 0])

[tbenthompson]

Closing because this fails gracefully now:

Traceback (most recent call last):
  File "/home/tbent/Dropbox/active/quantco/quantcore.glm/fail.py", line 24, in <module>
    split_matrix = from_pandas(df.astype("category"))
  File "/home/tbent/.miniconda3/envs/quantcore.glm/lib/python3.9/site-packages/quantcore/matrix/constructor.py", line 95, in from_pandas
    cat = CategoricalMatrix(coldata, dtype=dtype)
  File "/home/tbent/.miniconda3/envs/quantcore.glm/lib/python3.9/site-packages/quantcore/matrix/categorical_matrix.py", line 40, in __init__
    raise ValueError("Categorical data can't have missing values.")
ValueError: Categorical data can't have missing values.