Skip to content

fix(transformers): ContinuousOrdinalTransformer breaks on DataFrames #14

Description

@ChrisW09

Summary

ContinuousOrdinalTransformer produces silently wrong output when given a pandas
DataFrame: the wrong number of mappings, the wrong output shape, and all-zero codes. No
exception is raised.

fit iterates for col in X.T, which for a DataFrame yields the column labels of the
transpose
— i.e. the original row index — not the column data. transform iterates
for row in X, which for a DataFrame yields column names, not rows.

Reproduction

import numpy as np, pandas as pd
from pretab.transformers import ContinuousOrdinalTransformer

df = pd.DataFrame({"c1": ["a", "b", "a", "c"], "c2": ["x", "y", "x", "y"]})
t = ContinuousOrdinalTransformer().fit(df)
print("len(mapping_):", len(t.mapping_), "(expected 2)")
print("mapping_:", t.mapping_)
print("transform(df):\n", t.transform(df), " <- expected shape (4, 2)")
len(mapping_): 4 (expected 2)
mapping_: [{np.int64(0): 1, None: 0}, {np.int64(1): 1, None: 0}, {np.int64(2): 1, None: 0}, {np.int64(3): 1, None: 0}]
transform(df): 
 [[0 0]
 [0 0]]  <- expected shape (4, 2)

mapping_ has one entry per row (keyed by the row index 0..3) instead of one per
column, and transform returns 2 rows of zeros for a 4-row, 2-column frame.

A related shape bug on empty input, which affects ndarray callers too:

import numpy as np
from pretab.transformers import ContinuousOrdinalTransformer

arr = np.array([["a", "x"], ["b", "y"]], dtype=object)
t = ContinuousOrdinalTransformer().fit(arr)
print("transform(empty).shape:", t.transform(np.empty((0, 2), dtype=object)).shape,
      "(expected (0, 2))")
transform(empty).shape: (0,) (expected (0, 2))

np.array([]) from the empty list comprehension is 1-D, which breaks any downstream
hstack.

Expected

DataFrame and ndarray inputs behave identically: one mapping per column, output shape
(n_samples, n_features), and (0, n_features) for zero rows.

Actual

DataFrame input yields n_rows mappings and an (n_columns, n_columns) all-zero result;
empty input yields a 1-D array.

Root cause

pretab/transformers/encoders/continuous_ordinal.py:49-52:

self.mapping_ = [
    {category: i + 1 for i, category in enumerate(np.unique(col))}
    for col in X.T                                  # <- DataFrame.T iterates labels
]

pretab/transformers/encoders/continuous_ordinal.py:73-78:

X_transformed = np.array([
    [self.mapping_[col].get(value, 0) for col, value in enumerate(row)]
    for row in X                                    # <- DataFrame iterates column names
])

Inside Preprocessor the defect is masked, because the SimpleImputer that precedes this
step in get_categorical_transformer_steps always hands it an ndarray. It bites anyone
using the transformer directly — and it is public, exported from pretab.transformers, and
documented in the API reference.

Suggested fix

Coerce to a 2-D ndarray at the top of both methods and build the output with an explicit
shape:

def fit(self, X, y=None):
    X = np.asarray(X, dtype=object)
    if X.ndim == 1:
        X = X.reshape(-1, 1)
    self.mapping_ = [
        {category: i + 1 for i, category in enumerate(np.unique(X[:, j]))}
        for j in range(X.shape[1])
    ]
    ...

def transform(self, X):
    check_is_fitted(self, "mapping_")
    X = np.asarray(X, dtype=object)
    if X.ndim == 1:
        X = X.reshape(-1, 1)
    out = np.zeros(X.shape, dtype=int)
    for j, mapping in enumerate(self.mapping_):
        out[:, j] = [mapping.get(v, 0) for v in X[:, j]]
    return out

The explicit np.zeros(X.shape, dtype=int) also fixes the empty-input shape for free.

While in there, two adjacent points worth a look:

  • transform does not validate that X.shape[1] == len(self.mapping_), unlike the other
    transformers in the package, which raise PretabDataError via
    pretab.core.validation.validate_2d_allow_nan.
  • The docstring says "Unknown or missing categories are mapped to 0", and fit sets
    mapping[None] = 0, but NaN is not None — a float("nan") category is treated as an
    ordinary category at fit and (usually) as unknown at transform. Worth stating explicitly
    or handling.

Impact

Public transformer, exported from pretab.transformers and listed in
docs/api/index.rst. Direct DataFrame use produces plausible-looking but entirely wrong
codes with no error. The Preprocessor path is not affected.

Environment

  • pretab 0.1.0 (main @ 51c3043)
  • Python 3.11.15, numpy 2.4.6, pandas 2.3.3, scikit-learn 1.9.0, scipy 1.17.1
  • macOS (darwin 25.5.0)

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions