Skip to content

[Feature Request] PGO applicability #3456

Open
@zamazan4ik

Description

@zamazan4ik

Feature Request

Profile-Guided Optimization (PGO) allows gaining additional performance for the software since it uses runtime profile information to perform more advanced optimization during the compilation process. Since Firecracker is a large project I guess PGO could help here to gain even more performance.

Describe the desired solution

Possible steps:

  • Prepare firecracker build with PGO and bench it compared to the non-PGO firecracker.
  • At least consider adding PGO to CI. Yes, it has a LOT of caveats like a huge bump in a build time, good profile preparation, profile stability between releases, and much more other stuff but in my opinion, it could be worth it.

Maybe would be a good idea to try to use BOLT too, but FMPOV "usual" PGO is much more stable.

Describe possible alternatives

Just leave things as is.

Additional context

Probably the most helpful link would be this one: vectordotdev/vector#15631

Checks

  • Have you searched the Firecracker Issues database for similar requests?
    Have you read all the existing relevant Firecracker documentation?
    Have you read and understood Firecracker's core tenets?

Activity

JonathanWoollett-Light

JonathanWoollett-Light commented on Feb 28, 2023

@JonathanWoollett-Light
Contributor

Great idea. There are some concerns we have with PGO optimization for specific workloads, but feel free to post a PR on the issue.

zamazan4ik

zamazan4ik commented on Jun 21, 2023

@zamazan4ik
Author

There are some concerns we have with PGO optimization for specific workloads

Yeah, I understand. I think would be a good idea to start with just a testing PGO approach on some workloads with Firecracker. And then, if the results would be interesting, think about different workloads, their intersection in profiles, etc.

Also, recently I found results of applying PGO on similar projects:

More PGO results on different projects could be found here.

added
Priority: LowIndicates that an issue or pull request should be resolved behind issues or pull requests labelled `
on Aug 21, 2023
xmarcalx

xmarcalx commented on Aug 21, 2023

@xmarcalx
Contributor

Hi @zamazan4ik ,

Thank you very much for your pointer and ideas.
I think it will be interesting to see a prototype of this functionality working so we can really quantify the benefits.
If you or anyway in the community is interested in take this task and showcase the data, it will be amazing and we will may have more datapoint on the best course of actions.

added
Status: ParkedIndicates that an issues or pull request will be revisited later
on Aug 12, 2024
ambersongliu

ambersongliu commented on Apr 11, 2025

@ambersongliu

Hello,

We are students from UT Austin and are looking to work on this issue for our virtualization class. If no one is working on it, we would love to. Thanks!

zamazan4ik

zamazan4ik commented on Apr 11, 2025

@zamazan4ik
Author

I am pretty sure that no one is working on this right now, so feel free to work on it. I would be happy to see your results on this!

ambersongliu

ambersongliu commented on Apr 15, 2025

@ambersongliu

Amazing thank you! Do you mind assigning me to this issue? Thank you!

valeriepieger

valeriepieger commented on Apr 27, 2025

@valeriepieger

Hi all,

Thanks so much for the details on this feature request and for outlining the concerns regarding PGO. I am in the group at UT Austin with Amber (currently assigned to this issue) and wanted to propose our current plan for this issue.

Based on the previous discussions on this issue and after reading the QEMU and CrosVM PGO workflows, we would like to start with a small, isolated prototype for PGO, without affecting the current build process or CI.

Here's what we are thinking:

1. Documentation

We will add a README-pgo.md with documentation on how to manually perform PGO builds:

  • Instrumentation: Build Firecracker with RUSTFLAGS="-Cprofile-generate=/tmp/firecracker-profdata" cargo build --release to be able to generate .profraw files during execution
  • Profiling: Running realistic, typical workloads (e.g., booting microVMs, basic I/O, networking)
  • Optimization: merging profiling data (using llvm-profdata) and rebuilding Firecracker using RUSTFLAGS="-Cprofile-use=/tmp/firecracker.profdata -Cllvm-args=-pgo-warn-missing-function" cargo build --release

2. Prototype Developer Tooling

We will add a new script under tools/devtool called pgo-build that will allow the following subcommands:

  • instrument: build Firecracker with profiling instrumentation
  • profile: Run example workloads to generate profile data; will be optional
  • merge: merge .profraw files into a .profdata file
  • optimize: build Firecracker using the profiling data
    This should make local experimentation easier while being opt-in.

3. Optional Example Workload Runner

We may add a simple script to launch basic microVMs and exercise typical Firecracker functionality to help collect realistic profile data, but it will be purely for convenience. Users can always run their custom workloads instead.

4. Benchmarking

  • If possible, we want to benchmark the normal vs optimized/PGO build to hopefully prove that PGO will be beneficial! We'll measure metrics like binary size, CPU usage, boot latency, etc.

Please let us know if this approach sounds reasonable or if you have any questions or concerns! It will be isolated so it won't change the default build behavior.

Thank you again for your time!

xmarcalx

xmarcalx commented on Apr 30, 2025

@xmarcalx
Contributor

Hi @valeriepieger ,

Thank you very much for the previous message.

The plan overall seems good 🙂

we would like to start with a small, isolated prototype for PGO, without affecting the current build process or CI.

Yes as you said I think this is the right first step. Let's focus on building a prototype which shows the effectiveness of a PGO approach with Firecracker. So feel free to develop a prototype in a feature branch so you can move faster than publishing PRs.

The documentation proposal is fine, let's make sure you write in your README how you perform the build and profile Firecracker such that we can enable everyone to reproduce the builds (and ourselves too, in case you will need a bit of help along the way).

Regarding benchmark to use, i think a good starting point is to actually run the performance tests that you already find into our repository and that you should be able to run locally too.
Will be good for example to see what will be the difference in those baselines between mainline Firecracker and the PGO-enabled one.

linked a pull request that will close this issueAdd PGO Applicability Feature (Prototype) #5193on May 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

Priority: LowIndicates that an issue or pull request should be resolved behind issues or pull requests labelled `Status: ParkedIndicates that an issues or pull request will be revisited laterType: EnhancementIndicates new feature requests

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

    Development

    Participants

    @zamazan4ik@JonathanWoollett-Light@xmarcalx@ambersongliu@roypat

    Issue actions

      [Feature Request] PGO applicability · Issue #3456 · firecracker-microvm/firecracker