HMA1 KeyError with `model.fit()`

[SijRa]

Environment Details

• SDV version: 0.17.1
• Pandas version: 1.3.5
• Python version: Python 3.7.14
• Operating System: Windows 10

Error Description

I'm trying to model multi-tabular data using the HMA1 algorithm and with the CTGAN model. After fitting the data, model.fit() returns a pandas KeyError.

Here's the metadata visualisation:

Epoch 1, Loss G:  0.7057,Loss D: -0.0040
/usr/local/lib/python3.7/dist-packages/ctgan/data_transformer.py:112: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  data[column_name] = data[column_name].to_numpy().flatten()
/usr/local/lib/python3.7/dist-packages/ctgan/data_transformer.py:112: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  data[column_name] = data[column_name].to_numpy().flatten()
/usr/local/lib/python3.7/dist-packages/ctgan/data_transformer.py:112: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  data[column_name] = data[column_name].to_numpy().flatten()
/usr/local/lib/python3.7/dist-packages/ctgan/data_transformer.py:112: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  data[column_name] = data[column_name].to_numpy().flatten()
/usr/local/lib/python3.7/dist-packages/ctgan/data_transformer.py:112: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  data[column_name] = data[column_name].to_numpy().flatten()
/usr/local/lib/python3.7/dist-packages/ctgan/data_transformer.py:112: SettingWithCopyWarning: 
A value is trying to be set on a copy of a slice from a DataFrame.
Try using .loc[row_indexer,col_indexer] = value instead

See the caveats in the documentation: https://pandas.pydata.org/pandas-docs/stable/user_guide/indexing.html#returning-a-view-versus-a-copy
  data[column_name] = data[column_name].to_numpy().flatten()
Epoch 2, Loss G:  0.6715,Loss D: -0.0245
Epoch 3, Loss G:  0.6671,Loss D: -0.0493
Epoch 4, Loss G:  0.7102,Loss D: -0.0643
...
Epoch 300, Loss G: -0.2943,Loss D:  0.0184
---------------------------------------------------------------------------
KeyError                                  Traceback (most recent call last)
[/usr/local/lib/python3.7/dist-packages/pandas/core/indexes/base.py](https://localhost:8080/#) in get_loc(self, key, method, tolerance)
   3360             try:
-> 3361                 return self._engine.get_loc(casted_key)
   3362             except KeyError as err:

8 frames
pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: '__admissions__subject_id__num_rows'

The above exception was the direct cause of the following exception:

KeyError                                  Traceback (most recent call last)
[/usr/local/lib/python3.7/dist-packages/pandas/core/indexes/base.py](https://localhost:8080/#) in get_loc(self, key, method, tolerance)
   3361                 return self._engine.get_loc(casted_key)
   3362             except KeyError as err:
-> 3363                 raise KeyError(key) from err
   3364 
   3365         if is_scalar(key) and isna(key) and not self.hasnans:

KeyError: '__admissions__subject_id__num_rows'

[npatki]

Hi @SijRa, thanks for filing. Appreciate you putting in the details about your metadata and the stack trace -- very helpful!

From the logs you have provided, it appears that you are trying to use the CTGAN technique? Currently, the HMA1 model is only compatible with copulas modeling (see API spec). Since CTGAN is not based on copulas, it cannot be used.

We are using #457 to keep track of the feature request for using CTGAN for multi-table synthetic data.

To help us prioritize, it would be great if you could describe your use case. After you create your synthetic data, how are you planning to use it? In what ways is the current functionality (using GaussianCopula) insufficient for your project?

[npatki]

Hi @SijRa, I'll close off this issue since the HMA1 is currently working as intended (cannot be used with CTGAN). It would be great if you can add to #457 for continued discussion in supporting this new feature.

Feel free to reply here if you have any other thoughts or questions about the KeyError you were encountering.