Generating data 100x faster with Rust

Example scripts to demonstrate the ability of Rust to accelerate data generation. Performance improvement can vary, initial measurements currently show 100-150x gain for a typical dataset/ configuration in v1.

v2 version is still a work in progress, although most functionality is working it may have some rough edges. Performance with Rayon threading is still something I am exploring, when scaling up it does not saturate all the cores perfectly. If you know why feel free to send me a PR.

PRs on any other performance improvement are welcome, as the goal of this project is to get the fastest data generation possible. While we are currently at 100x -- have a sense we can push this much higher over time.

Versions

v1 (Python / Rust)

• generate dataset

v2 (Python / Rust)

• ability to pass parameters via CLI
• dynamic schema loading (Rust-only)
• enable threading (multi-core)
• convert to dataframe
• export to Parquet

Run examples

Python

# enter directory
cd examples/pyfake-v1

# v1 requires faker
pip install faker
# v2 requires faker and polars
pip install faker polars

# run
python pyfake/generate.py

# benchmark row (average over 10 runs)
python -c 'import pyfake; pyfake.benchmark_row()'

# benchmark column (average over 10 runs)
python -c 'import pyfake; pyfake.benchmark_column()'

Python with Poetry

# enter directory
cd examples/pyfake-v1

# install dependencies
poetry update

# run script
poetry run pyfake

Rust

# enter directory
cd examples/rsfake-v1

cargo build --release
target/release/rsfake

Docker

For convenience a Dockerfile is included with both Python and Rust dependencies pre-installed.

# build
docker build -t fakeroo .

# run interactive shell
docker run -ti  --rm fakeroo bash

# run Python example
cd /examples/pyfake-v1
python pyfake/generate.py

# run Rust example
cd /examples/rsfake-v1
bin/rsfake