DataValidator class method obliterates memory

[ChrisSoderberg-ONS]

For reproduction:

from datachecker import check_and_export
import json
import pandas as pd
from sklearn.datasets import load_wine

test_df = load_wine(as_frame=True).data
test_df = test_df.rename(columns={"od280/od315_of_diluted_wines": "od280_od315_of_diluted_wines"})

schema = {
    "check_duplicates": True,
    "check_completeness": True,
    "columns": {},
}

for name, value in test_df.items():
    schema["columns"].update({name: {
            "type": str(value.dtype), 
            "optional": False, 
            "allow_na": True,
    }})

with open("schema.json", "w", encoding="utf-8") as f:
    json.dump(schema, f, ensure_ascii=False, indent=4)

check_and_export(
    data=test_df,
    schema="schema.json",
    format="html",
    file="data_checker_default_log.html"
)

We get a MemoryError when the DataValidator._check_completeness method is called, specifically caused by this line:

datachecker/datachecker/data_checkers/pandas_validator.py

Line 42 in c7bb5cb

combinations = set(product(*unique_values))

unique_values in this case will be an array of 13 arrays, with the size of the sub arrays shown below:

Then we are calling a cartesian product of this nested array e.g, set(product([13 x 126], [13 x 133], [13 x 79], ... , [13 x 121])) so we very quickly get into a memory explosion

A quick screengrab shows the rapid exponential growth in output when we start creating cartesian products of entire columns:

After just 4 columns we're pretty much cooked for memory, and these data size are tiny because it's just a toy data - with production size data we'd be done even sooner

I can't replicate the issue on the R version because I get a failure point happening somewhere else but from looking at the implementation of the completeness checks I believe we have the same issue
https://github.com/ONSdigital/data.checker/blob/a0113e4604ee3763499cdb96b4423dcebdabd090/R/checks.R#L43

[ChrisSoderberg-ONS]

@shilohd @Jday7879 would I be able to get a second pair of eyes on this?

[Jday7879]

@ChrisSoderberg-ONS I think you are right that this runs the risk of a memory issue as its checking for every possible combination. I was thinking is it worth not letting the check run unless they give specific columns to cross check in completness_columns ? That way users wont cross check every combination by accident. There might be better ways to perform the check its self to limit memory usage that could be looked into after this quicker fix

[ChrisSoderberg-ONS]

Is there a reason completeness checks are being done this way instead of a missing/total calculation per column?

[Jday7879]

@AntonZogk and I had a crack at this looking at the pandas implementation. Because we don't use the exact combinations that are missed, we have tried to use counts see branch for more details. https://github.com/ONSdigital/datachecker/tree/potential-completeness-fix
This seems to run for the example code you gave, but might need a bit more work / testing with larger data sets