New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ValueError: Cannot convert float NaN to integer #1730
Comments
Hi @ThomasK1018, thanks for the detailed description. We were able to replicate your error. For future reference, I'm attaching the stack trace below. Root CauseWe will investigate more but in all likelihood, this is probably related to #1691. The known bug is that HMA sometimes creates null values even when there aren't any in the real data. Usually this happens for float64 or string types in the child table. In this particular case, you have int64 columns (that are categorical). The int64 storage type does not support null values, hence the crash. As a next step, the team can investigate more to confirm that the root cause is, indeed, #1691. If so, we will dedupe the issue. We can also do a better job at preventing the crash and instead just returning the null value (even though this is not quite correct). I've filed RDT issue 747 to keep track of this. WorkaroundsChanging the default distribution for each table to synthesizer = HMASynthesizer(metadata)
for table_name in ['unique_id', 'ads_dataset', 'feeds_dataset']:
synthesizer.set_table_parameters(
table_name=table_name,
table_parameters={
'enforce_min_max_values': True,
'default_distribution': 'norm'
}
)
synthesizer.fit(datasets)
synthesizer.sample() Do note that the HMA is only meant for smaller datasets though. So even with this workaround in place, you'll still see the performance alert:
Ultimately, I'd recommend using the HSASynthesizer as it's designed to handle larger datasets more robustly. I've confirmed that it works without issue on this dataset (fitting in a few seconds). |
Hi Neha, Thanks for your reply. That change on default distribution has certainly helped. May I also ask what are the available choices to be filled in for this section? I have also tried inputting truncnorm and the output is different from leaving in blank. So I would like to know if there are more choices for the distribution. Thanks. Best Regards, |
Hi @ThomasK1018, no problem. For more info, I'd recommend checking our docs here. Note that the |
Hi @ThomasK1018, I'm closing off this issue as a duplicate since we do have now have more evidence that it is caused by #1691. We are actively looking into this root cause and hope to have fix up in an upcoming release. Please feel free to reply directly to #1691 if there is anything more to discuss and we can always continue the conversation there. |
Environment Details
Please indicate the following details about the environment in which you found the bug:
Error Description
I used the following code and the following dataset. Everything was alright yesterday. I tried to amend some of the contents of the datasets and suddenly this error keeps appearing. I reverted everything back to yesterday's version and this error keeps appearing. But there was not a single missing value in the existing dataset. Datasets are also attached for your testing purposes. Thanks.
ads_dataset.csv
feeds_dataset.csv
unique_id.csv
Steps to reproduce
The text was updated successfully, but these errors were encountered: