Range constraint does not produce cases of missing values & may create violative data

[npatki]

• SDV version: 1.0.1

Error Description

In my dataset, I have a Range constraint such that column A < B < C. Any of the values may be null in different scenarios.

Expected:

• When present: A < B, B < C and A < C
• There may be missing values in any of the columns in any combination

Observed:

• Similar to Inequality constraint does not produce all possibilities of missing values #1392, not all combinations of missing values are created
• Additionally, sometimes the C > A (if B is missing)

Fix

The fix is also similar to #1392:

1. Keep the lowest column A and then replace each other column with the diffs (Replace B with (B-A) and replace C with (C-B))
2. Create a new categorical column that stores which columns are missing
3. Replace any originally missing values with the average

On the reversal, we can compute the values in a cascading way using the diffs (B = A + diff1, C = B + diff2), and then set the missing values using the categorical column.

Steps to reproduce

import pandas as pd
import numpy as np
from sdv.metadata import SingleTableMetadata
from sdv.single_table import GaussianCopulaSynthesizer

data = pd.DataFrame(data={
    'low':    [1, 4, np.nan, 0,      4,      np.nan, np.nan, 5,      np.nan],
    'middle': [2, 5, 3,      np.nan, 5,      np.nan, 5,      np.nan, np.nan],
    'high':   [3, 7, 8,      4,      np.nan, 9,      np.nan, np.nan, np.nan]
})

metadata_dict = {
    'columns': {
        'low': { 'sdtype': 'numerical' },
        'middle': { 'sdtype': 'numerical'},
        'high': { 'sdtype': 'numerical'}
    }
}

metadata = SingleTableMetadata.load_from_dict(metadata_dict)
synth = GaussianCopulaSynthesizer(metadata)

my_constraint = {
    'constraint_class': 'Range',
    'constraint_parameters': {
        'low_column_name': 'low',
        'middle_column_name': 'middle',
        'high_column_name': 'high'
    }
}

synth.add_constraints(constraints=[my_constraint])
synth.fit(data)

synth_data = synth.sample(2000)
synth_data[synth_data['low'] > synth_data['high']]