Skip to content
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
30 changes: 30 additions & 0 deletions docs/source/user-guide/crate-configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -92,6 +92,36 @@ lto = true
codegen-units = 1
```

### Profile Guided Optimization (PGO)

Profile Guided Optimization can improve DataFusion performance by up to 25%. It works by compiling with instrumentation, running representative workloads to collect profile data, then recompiling with optimizations based on that data.

Build with instrumentation:

```shell
RUSTFLAGS="-C profile-generate=/tmp/pgo-data" cargo build --release
```

Run your workloads to collect profile data. Use benchmarks like TPCH or Clickbench, or your actual production queries:

```shell
./target/release/your-datafusion-app --benchmark
```

Rebuild using the collected profile:

```shell
RUSTFLAGS="-C profile-use=/tmp/pgo-data" cargo build --release
```

Tips:

- Use workloads that match your production patterns
- Run multiple iterations during profiling for better coverage
- Combine with LTO and CPU-specific optimizations for best results

See the [Rust compiler guide](https://rustc-dev-guide.rust-lang.org/building/optimized-build.html#profile-guided-optimization) for more details. Discussion and results in [issue #9507](https://github.com/apache/datafusion/issues/9507).

### Alternate Allocator: `snmalloc`

You can also use [snmalloc-rs](https://crates.io/crates/snmalloc-rs) crate as
Expand Down