Evaluate using Profile-Guided Optimization (PGO) and Post-Link Optimization (PLO)

[zamazan4ik]

Hi!

Recently I checked many optimizations like PGO and PLO (mostly with LLVM BOLT) improvements on multiple projects. The results are available here. According to the tests, these optimizations can help to achieve better performance in many cases like databases. I think trying to optimize this project with them will be an interesting idea to achieve more performance.

I already did some (very basic!) benchmarks and want to share my results here.

Test environment

• Fedora 39
• Linux kernel 6.8.7
• AMD Ryzen 9 5900x
• 48 Gib RAM
• SSD Samsung 980 Pro 2 Tib
• Compiler - Rustc 1.77.2
• bbolt-rs version: the latest for now from the main branch on commit 980b96b81768c4c8c78034a99aa7004dc4672674
• Disabled Turbo boost

Benchmark

For benchmark purposes, I used these benchmarks. The PGO training workload was bench command run. The release and PGO-optimized results are generated with bench -c 10000000.

All PGO and PLO optimizations are done with cargo-pgo. All tests are done on the same machine, done multiple times, with the same background "noise" (as much as I can guarantee of course) - the results are consistent enough across runs. taskset -c 0 is used for reducing the OS scheduler result interference.

Results

Let's start with the results.

Release:

taskset -c 0 bench -c 10000000
# Write  8.914817311s  (891ns/op)  (1122334 op/sec)
# Read  1.11635498s  (16ns/op)  (62499999 op/sec)

Release + PGO optimization:

taskset -c 0 bench -c 10000000
# Write  6.502718524s  (650ns/op)  (1538461 op/sec)
# Read  1.06592023s  (13ns/op)  (76923076 op/sec)

Release + PGO optimization + BOLT optimization:

taskset -c 0 bench -c 10000000
# Write  6.50363167s  (650ns/op)  (1538461 op/sec)
# Read  1.0854954s  (14ns/op)  (71428571 op/sec)

(just for reference) Release + PGO instrumentation:

taskset -c 0 bench -c 10000000
# Write	14.626009666s	(1.463µs/op)	(683526 op/sec)
# Read	1.131095569s	(28ns/op)	(35714285 op/sec)

(just for reference again) Release + PGO optimized + BOLT instrumented:

taskset -c 0 bench -c 10000000
# Write  8.641237686s  (864ns/op)  (1157407 op/sec)
# Read  1.102478063s  (12ns/op)  (83333333 op/sec)

According to the tests above, I see measurable improvements from enabling PGO in performance. However, enabling PLO with LLVM BOLT didn't show measurable improvements at least in the simple test above.

For anyone interested in binary sizes, I collected some statistics too (without debug symbols stripping):

• Release: 1.3 Mib
• Release + PGO instrumentation: 3.4 Mib
• Release + PGO optimized: 1.3 Mib
• Release + PGO optimized + BOLT instrumented: 14 Mib
• Release + PGO optimized + BOLT optimized: 4.3 Mib

The only interesting case here is the last one - "Release + PGO optimized + BOLT optimized". I don't know why the binary size was increased so much. I guess some "magic" BOLT's option should be involved here and "fix" the situation. However, it's just a guess for now, no more.

Further steps

I can suggest the following action points:

• Perform more PGO and PLO benchmarks on the database in more scenarios. If it shows improvements - add a note to the documentation about possible improvements in the project performance with PGO.
• Providing an easier way (e.g. a build option) to build scripts with PGO can be helpful for the end-users and maintainers since they will be able to optimize bbolt-rs according to their workloads.

Here are some examples of how PGO optimization is integrated into other projects:

• Rustc: a CI script for the multi-stage build
• GCC:
  • Official docs, section "Building with profile feedback"
  • A part in a "wonderful" configure script
• Clang: Docs
• Python:
  • CPython: README
  • Pyston: README
• Go: Bash script
• V8: Bazel flag
• ChakraCore: Scripts
• Chromium: Script
• Firefox: Docs
  • Thunderbird has PGO support too
• PHP - Makefile command and old Centminmod scripts
• MySQL: CMake script
• YugabyteDB: GitHub commit
• FoundationDB: Script
• Zstd: Makefile
• Foot: Scripts
• Windows Terminal: GitHub PR
• Pydantic-core: GitHub PR
• file.d: GitHub PR
• OceanBase: CMake flag

I would be happy to answer your questions about all the optimizations above. Please do not treat the issue as a bug or something like that - it's just an idea of how the project performance can be improved.

[ambaxter]

Hi @zamazan4ik ,

I played with PGO about a month ago, but decided to wait until after I had completed the first release to take a deeper look.

Do you have an updated link to the Rust ci script? It currently points to a 404.

Also, would you mind describing the 2nd step a bit more? I'm not sure what you mean.

[zamazan4ik]

Do you have an updated link to the Rust ci script? It currently points to a 404.

Oh, sorry for that - the tool is changed in the upstream and now it's rewritten in Rust from Python: https://github.com/rust-lang/rust/tree/master/src/tools/opt-dist . Updated the original post as well.

Also, would you mind describing the 2nd step a bit more? I'm not sure what you mean.

Sure! By "Providing an easier way (e.g. a build option) to build scripts with PGO" I mean extending existing bbolt-rs's build infra (scripts) with an additional option - building the database with PGO. It can look like make build_with_pgo (just an example!). With one simple command, it can be easier for users to build their own bbolt-rs version and tweak it accordingly to their workloads. Is it worth it or not - it's up to you. My 0.5$ - it's not worth it on the current project lifecycle stage.