Great Expectations failing with "MemoryError" during large dataset profiling and suite creation

[aayostem]

We're encountering MemoryError when running great_expectations suite new with the automated profiler on large datasets (5M+ rows, 50+ columns). The process gets killed when memory usage exceeds 16GB:

2024-01-15 16:45:33.789 INFO  [great_expectations] - Starting automated profiling for dataset...
2024-01-15 16:45:45.123 ERROR [great_expectations] - MemoryError: Unable to allocate 2.3 GiB for an array with shape (5000000, 62) and data type object
Killed

Our environment:

• Great Expectations 0.18.0
• Python 3.9
• Pandas backend
• 16GB RAM machine
• Dataset: 5M rows × 62 columns (mixed data types)
• Automated profiling with great_expectations suite new

What we've tried:

• Used sample_size parameter in profiler config
• Tried different profilers (BasicSuiteBuilderProfiler, OnboardingDataAssistant)
• Increased swap space to 8GB

Error patterns:

• Occurs during column type inference for large string columns
• Fails when calculating value counts for high-cardinality columns
• Memory spikes during histogram calculations for numeric columns

How can we profile large datasets with Great Expectations without running into memory issues?

[joshua-stauffer]

hey @aayostem , thanks for the question. The profiler that shipped with GX 0.18 had numerous issues, so we removed it in the 1.0 release. As of yet, there isn't a replacement in GX 1.0+, but I'm happy to share that it's on the roadmap.