Anyway to isolate Dynamic PGO impact?

[davepcallan]

Great lib and great idea to have discussions open here on GH 👍🏻

I see Dynamic PGO is enabled by default in .NET 8.

Is Dynamic PGO handled / How is Dynamic PGO handled in BenchmarkDotNet?

If I'm running a benchmark for say some kind string concat in .NET 7 v .NET 8 and I get certain results does that include Dynamic PGO improvement or is the PGO impact removed from displayed results?

Would love to know more about how (and if) DPGO is handled in BDN

[aalmada]

You can turn on the PGO flags using a configuration. To turn on for all benchmarks, use the following config as a parameter for your BenchmarkSwitcher:

 var config = DefaultConfig.Instance
   .AddJob(Job.Default
        .WithRuntime(CoreRuntime.Core60)
        .WithEnvironmentVariables(
            new EnvironmentVariable("DOTNET_ReadyToRun", "0"), // Disable AOT
            new EnvironmentVariable("DOTNET_TC_QuickJitForLoops", "1"), // Enable Quick Jit for loop
            new EnvironmentVariable("DOTNET_TieredPGO", "1")) // Turn on layered PGO
        .WithId(".NET 6 PGO");

The flags are set as specified at https://devblogs.microsoft.com/dotnet/performance-improvements-in-net-6/

[davepcallan]

Thanks Antão,
Do you know if these env flags are honored in .NET 8 given that DPGO is enabled by default ...

var config = DefaultConfig.Instance
   .AddJob(Job.Default
        .WithRuntime(CoreRuntime.Core80)
        .WithEnvironmentVariables(
            new EnvironmentVariable("DOTNET_ReadyToRun", "0"), // Disable AOT
            new EnvironmentVariable("DOTNET_TC_QuickJitForLoops", "1"), // Enable Quick Jit for loop
            new EnvironmentVariable("DOTNET_TieredPGO", "1")) // Turn on layered PGO
        .WithId(".NET 8 PGO");
    .AddJob(Job.Default
        .WithRuntime(CoreRuntime.Core80)
        .WithEnvironmentVariables(
            new EnvironmentVariable("DOTNET_ReadyToRun", "0"), // Disable AOT
            new EnvironmentVariable("DOTNET_TC_QuickJitForLoops", "0"), // Disable Quick Jit for loop
            new EnvironmentVariable("DOTNET_TieredPGO", "0")) // Turn off layered PGO
        .WithId(".NET 8 WITHOUT PGO");

Would something like above work to just show PGO impact?
Haven't tried it yet, will try later, thanks for your response.

[aalmada]

Sorry but I don't know.
I only know how to turn it on on previous versions.

[AndyAyersMS]

By default, BDN results for .NET 8 (preview 5 and later) will include Dynamic PGO.

To disable this you can set DOTNET_TieredPGO=0 in the environment or via --envVars DOTNET_TieredPGO:0 if you enable BDN's command line parsing.

If your [Benchmark] method is very long running (say 100's of ms) then BDN's strategy for measuring optimized code may not be effective. You will either need to shorten the benchmark or else force BDN to do more invocations/iterations. We have a few tests like that in https://github.com/dotnet/performance. See for instance dotnet/runtime#84264 (comment).

[AndyAyersMS]

I should add, you no longer need to modify DOTNET_ReadyToRun or DOTNET_TC_QuickJitForLoops, they can be left as-is.

In .NET 7 we enabled OSR which removes the need to changeDOTNET_TC_QuickJitForLoops.
In .NET 8 we made changes to the interactions of tiering and PGO which removes the need to change DOTNET_ReadyToRun

[davepcallan]

Thanks Andy ...

Can DPGO make performance worse, or am I doing something stupid here ...

Benchmark ->
https://gist.github.com/davepcallan/c33482551f3d5bd1598866c661bf80af

[AndyAyersMS]

PGO can occasionally make things slower. If you look at the issue I linked above there were some examples, and dotnet/runtime#87194 is an active issue tracking more cases. There are a variety of reasons why this can happen.

When you run across cases like this please open an issue in this repo so we can dig in and see what's up.

[davepcallan]

Great, thanks a lot for your help Andy, I just wasn't sure if it was my with my setup ...

 private class Config : ManualConfig
        {
            public Config()
            {
                AddJob(Job.Default.WithId("Non DPGO")
                    .WithBaseline(true)
                    .WithRuntime(CoreRuntime.Core80)
                    .WithEnvironmentVariables(
                        new EnvironmentVariable("DOTNET_TieredPGO", "0")));

                AddJob(Job.Default.WithId("DPGO")
                    .WithRuntime(CoreRuntime.Core80)
                    .WithEnvironmentVariables(
                        new EnvironmentVariable("DOTNET_TieredPGO", "1")));

                SummaryStyle =
                    SummaryStyle.Default.WithRatioStyle(RatioStyle.Percentage)
                    .WithTimeUnit(Perfolizer.Horology.TimeUnit.Millisecond);
            }
        }

or I was successfully isolating DPGO affect and seeing some regressions ..
I will keep an eye out for regressions and post them here for further investigations ...