pyspark.DataFrameSchema
doesn't have data syntheis method(s) implemented
#1479
Labels
enhancement
New feature or request
Is your feature request related to a problem? Please describe.
Currently, the
pyspark.DataFrameSchema
class does not support synthetic data generation. This feature is extremely useful for testing and development purposes, as it allows developers to quickly generate example data that adheres to a specific schema.Describe the solution you'd like
I would like to see a feature similar to the one provided by the
pandera
library for synthetic data generation onpandas
based data modules. In theory I guess it could look like:This would then generates a DataFrame with 3 rows, where each column adheres to the constraints defined in the schema.
Additional context
It's worth noting that the synthetic data generation doesn't necessarily need to use PySpark to generate the data. In most development scenarios, the volume of synthetic data required wouldn't be so large as to necessitate a distributed backend.
The text was updated successfully, but these errors were encountered: