Open
Description
Feature Request
Profile-Guided Optimization (PGO) allows gaining additional performance for the software since it uses runtime profile information to perform more advanced optimization during the compilation process. Since Firecracker is a large project I guess PGO could help here to gain even more performance.
Describe the desired solution
Possible steps:
- Prepare firecracker build with PGO and bench it compared to the non-PGO firecracker.
- At least consider adding PGO to CI. Yes, it has a LOT of caveats like a huge bump in a build time, good profile preparation, profile stability between releases, and much more other stuff but in my opinion, it could be worth it.
Maybe would be a good idea to try to use BOLT too, but FMPOV "usual" PGO is much more stable.
Describe possible alternatives
Just leave things as is.
Additional context
Probably the most helpful link would be this one: vectordotdev/vector#15631
Checks
- Have you searched the Firecracker Issues database for similar requests?Have you read all the existing relevant Firecracker documentation?Have you read and understood Firecracker's core tenets?
Activity
JonathanWoollett-Light commentedon Feb 28, 2023
Great idea. There are some concerns we have with PGO optimization for specific workloads, but feel free to post a PR on the issue.
zamazan4ik commentedon Jun 21, 2023
Yeah, I understand. I think would be a good idea to start with just a testing PGO approach on some workloads with Firecracker. And then, if the results would be interesting, think about different workloads, their intersection in profiles, etc.
Also, recently I found results of applying PGO on similar projects:
More PGO results on different projects could be found here.
xmarcalx commentedon Aug 21, 2023
Hi @zamazan4ik ,
Thank you very much for your pointer and ideas.
I think it will be interesting to see a prototype of this functionality working so we can really quantify the benefits.
If you or anyway in the community is interested in take this task and showcase the data, it will be amazing and we will may have more datapoint on the best course of actions.
ambersongliu commentedon Apr 11, 2025
Hello,
We are students from UT Austin and are looking to work on this issue for our virtualization class. If no one is working on it, we would love to. Thanks!
zamazan4ik commentedon Apr 11, 2025
I am pretty sure that no one is working on this right now, so feel free to work on it. I would be happy to see your results on this!
ambersongliu commentedon Apr 15, 2025
Amazing thank you! Do you mind assigning me to this issue? Thank you!
valeriepieger commentedon Apr 27, 2025
Hi all,
Thanks so much for the details on this feature request and for outlining the concerns regarding PGO. I am in the group at UT Austin with Amber (currently assigned to this issue) and wanted to propose our current plan for this issue.
Based on the previous discussions on this issue and after reading the QEMU and CrosVM PGO workflows, we would like to start with a small, isolated prototype for PGO, without affecting the current build process or CI.
Here's what we are thinking:
1. Documentation
We will add a README-pgo.md with documentation on how to manually perform PGO builds:
RUSTFLAGS="-Cprofile-generate=/tmp/firecracker-profdata" cargo build --release
to be able to generate .profraw files during executionllvm-profdata
) and rebuilding Firecracker usingRUSTFLAGS="-Cprofile-use=/tmp/firecracker.profdata -Cllvm-args=-pgo-warn-missing-function" cargo build --release
2. Prototype Developer Tooling
We will add a new script under tools/devtool called
pgo-build
that will allow the following subcommands:instrument
: build Firecracker with profiling instrumentationprofile
: Run example workloads to generate profile data; will be optionalmerge
: merge .profraw files into a .profdata fileoptimize
: build Firecracker using the profiling dataThis should make local experimentation easier while being opt-in.
3. Optional Example Workload Runner
We may add a simple script to launch basic microVMs and exercise typical Firecracker functionality to help collect realistic profile data, but it will be purely for convenience. Users can always run their custom workloads instead.
4. Benchmarking
Please let us know if this approach sounds reasonable or if you have any questions or concerns! It will be isolated so it won't change the default build behavior.
Thank you again for your time!
xmarcalx commentedon Apr 30, 2025
Hi @valeriepieger ,
Thank you very much for the previous message.
The plan overall seems good 🙂
Yes as you said I think this is the right first step. Let's focus on building a prototype which shows the effectiveness of a PGO approach with Firecracker. So feel free to develop a prototype in a feature branch so you can move faster than publishing PRs.
The documentation proposal is fine, let's make sure you write in your README how you perform the build and profile Firecracker such that we can enable everyone to reproduce the builds (and ourselves too, in case you will need a bit of help along the way).
Regarding benchmark to use, i think a good starting point is to actually run the performance tests that you already find into our repository and that you should be able to run locally too.
Will be good for example to see what will be the difference in those baselines between mainline Firecracker and the PGO-enabled one.