Numerical & datetime transformers crash if the entire column is null

[npatki]

• RDT version: 1.4.1 (latest)

Error Description

Sometimes, I may have a numerical or datetime column that is entirely null. This can happen for a number of reasons - eg. I may only have access to a subset of data for training and it just happens to be all null.

Currently, the numerical and datetime transformers crash if they encounter this. Instead, we can gracefully fall back and allow the remaining flow to proceed.

Steps to reproduce

Reproduce this by creating data with all null values. Note that this is replicable for both datetime and numerical sdtypes.

import pandas as pd
import numpy as np
from rdt import HyperTransformer
from rdt.transformers.datetime import UnixTimestampEncoder

data = pd.DataFrame(data={
  'null_datetime_column': ['']*5,
  'null_numerical_column': [np.nan]*5
})

ht = HyperTransformer()
ht.detect_initial_config(data=data)

ht.update_sdtypes({ 'null_datetime_column': 'datetime' })
ht.update_transformers({ 'null_datetime_column': UnixTimestampEncoder(datetime_format='%Y-%m-%d') })
ht.fit(data)

Output:

TransformerInputError: 'missing_value_replacement' cannot be set to 'mean' when the provided data
only contains NaNs.

Fix

If the entire column is null, then use some pre-determined values as the missing value replacement

• For numerical, we can use 0
• For datetime, we can use the Unix equivalent of 0: January 1st, 1970 at UTC

Then we can convert the TransformerInputError into an logger.INFO message insetad:

logger.INFO: 'missing_value_replacement' cannot be set to 'mean' when the provided data
only contains NaNs. Using 0 instead.