New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Runtime knobs #262

Closed
AndreyAkinshin opened this Issue Sep 21, 2016 · 5 comments

Comments

Projects
None yet
2 participants
@AndreyAkinshin
Member

AndreyAkinshin commented Sep 21, 2016

In CoreCLR, we have a list of knobs which can be specified.

In Mono, we can set some command line arguments (like --optimize=OPT or --gc=[sgen,boehm]) and environment variables (like GC_DONT_GC or MONO_GAC_PREFIX, see also https://linux.die.net/man/1/mono).

I think, it would be cool to have an ability to specify some of knobs/arguments as a part of Job.

@adamsitnik

This comment has been minimized.

Show comment
Hide comment
@adamsitnik

adamsitnik Sep 21, 2016

Member

Good idea!

It could be a list of process arguments and a dictionary of environment variables.

Hint for potential contributors: we have an Executor class that is responsible for creating and starting the process. Currently the list of extra arguments for mono is set here

Member

adamsitnik commented Sep 21, 2016

Good idea!

It could be a list of process arguments and a dictionary of environment variables.

Hint for potential contributors: we have an Executor class that is responsible for creating and starting the process. Currently the list of extra arguments for mono is set here

@adamsitnik

This comment has been minimized.

Show comment
Hide comment
@adamsitnik

adamsitnik Sep 3, 2017

Member

Done ;)

You can specify env variables or mono arguments per job now.

Example:

public class IntroCustomMonoArguments
{
    public class ConfigWithCustomArguments : ManualConfig
    {
        public ConfigWithCustomArguments()
        {
            // --optimize=MODE , -O=mode
            // MODE is a comma separated list of optimizations. They also allow
            // optimizations to be turned off by prefixing the optimization name with a minus sign.

            Add(Job.Mono.With(new[] { new MonoArgument("--optimize=inline") }).WithId("Inlining enabled"));
            Add(Job.Mono.With(new[] { new MonoArgument("--optimize=-inline") }).WithId("Inlining disabled"));
        }
    }

    [Benchmark]
    public void Sample()
    {
        ShouldGetInlined(); ShouldGetInlined(); ShouldGetInlined();
        ShouldGetInlined(); ShouldGetInlined(); ShouldGetInlined();
        ShouldGetInlined(); ShouldGetInlined(); ShouldGetInlined();
    }

    void ShouldGetInlined() { }
}
BenchmarkDotNet=v0.10.9.20170903-develop, OS=Windows 10 Redstone 1 (10.0.14393)
Processor=Intel Core i7-6600U CPU 2.60GHz (Skylake), ProcessorCount=4
Frequency=2742190 Hz, Resolution=364.6720 ns, Timer=TSC
  [Host]            : .NET Framework 4.7 (CLR 4.0.30319.42000), 64bit RyuJIT-v4.7.2101.1
  Inlining disabled : Mono 5.0.1 (Visual Studio), 32bit 
  Inlining enabled  : Mono 5.0.1 (Visual Studio), 32bit 

Runtime=Mono  
Method Job Arguments Mean Error StdDev
Sample Inlining disabled --optimize=-inline 19.4252 ns 0.4406 ns 0.4525 ns
Sample Inlining enabled --optimize=inline 0.0000 ns 0.0000 ns 0.0000 ns
Member

adamsitnik commented Sep 3, 2017

Done ;)

You can specify env variables or mono arguments per job now.

Example:

public class IntroCustomMonoArguments
{
    public class ConfigWithCustomArguments : ManualConfig
    {
        public ConfigWithCustomArguments()
        {
            // --optimize=MODE , -O=mode
            // MODE is a comma separated list of optimizations. They also allow
            // optimizations to be turned off by prefixing the optimization name with a minus sign.

            Add(Job.Mono.With(new[] { new MonoArgument("--optimize=inline") }).WithId("Inlining enabled"));
            Add(Job.Mono.With(new[] { new MonoArgument("--optimize=-inline") }).WithId("Inlining disabled"));
        }
    }

    [Benchmark]
    public void Sample()
    {
        ShouldGetInlined(); ShouldGetInlined(); ShouldGetInlined();
        ShouldGetInlined(); ShouldGetInlined(); ShouldGetInlined();
        ShouldGetInlined(); ShouldGetInlined(); ShouldGetInlined();
    }

    void ShouldGetInlined() { }
}
BenchmarkDotNet=v0.10.9.20170903-develop, OS=Windows 10 Redstone 1 (10.0.14393)
Processor=Intel Core i7-6600U CPU 2.60GHz (Skylake), ProcessorCount=4
Frequency=2742190 Hz, Resolution=364.6720 ns, Timer=TSC
  [Host]            : .NET Framework 4.7 (CLR 4.0.30319.42000), 64bit RyuJIT-v4.7.2101.1
  Inlining disabled : Mono 5.0.1 (Visual Studio), 32bit 
  Inlining enabled  : Mono 5.0.1 (Visual Studio), 32bit 

Runtime=Mono  
Method Job Arguments Mean Error StdDev
Sample Inlining disabled --optimize=-inline 19.4252 ns 0.4406 ns 0.4525 ns
Sample Inlining enabled --optimize=inline 0.0000 ns 0.0000 ns 0.0000 ns
@adamsitnik

This comment has been minimized.

Show comment
Hide comment
@adamsitnik

adamsitnik Sep 3, 2017

Member

For variables it's very similar:

var variables = new [] { new EnvironmentVariable("VeryNiceKey", "VeryNiceValue") }
Add(job.With(variables));
BenchmarkDotNet=v0.10.9.20170903-develop, OS=Windows 10 Redstone 1 (10.0.14393)
Processor=Intel Core i7-6600U CPU 2.60GHz (Skylake), ProcessorCount=4
Frequency=2742190 Hz, Resolution=364.6720 ns, Timer=TSC
  [Host]     : .NET Framework 4.7 (CLR 4.0.30319.42000), 32bit LegacyJIT-v4.7.2101.1
  Job-HXYYII : .NET Framework 4.7 (CLR 4.0.30319.42000), 32bit LegacyJIT-v4.7.2101.1

EnvironmentVariables=VeryNiceKey=VeryNiceValue  LaunchCount=1  RunStrategy=ColdStart  
TargetCount=1  UnrollFactor=1  WarmupCount=1  
Member

adamsitnik commented Sep 3, 2017

For variables it's very similar:

var variables = new [] { new EnvironmentVariable("VeryNiceKey", "VeryNiceValue") }
Add(job.With(variables));
BenchmarkDotNet=v0.10.9.20170903-develop, OS=Windows 10 Redstone 1 (10.0.14393)
Processor=Intel Core i7-6600U CPU 2.60GHz (Skylake), ProcessorCount=4
Frequency=2742190 Hz, Resolution=364.6720 ns, Timer=TSC
  [Host]     : .NET Framework 4.7 (CLR 4.0.30319.42000), 32bit LegacyJIT-v4.7.2101.1
  Job-HXYYII : .NET Framework 4.7 (CLR 4.0.30319.42000), 32bit LegacyJIT-v4.7.2101.1

EnvironmentVariables=VeryNiceKey=VeryNiceValue  LaunchCount=1  RunStrategy=ColdStart  
TargetCount=1  UnrollFactor=1  WarmupCount=1  
@adamsitnik

This comment has been minimized.

Show comment
Hide comment
@adamsitnik

adamsitnik Sep 3, 2017

Member

@AndreyAkinshin I will add the docs tomorrow. And I will most probably also add .NET Core 2.0 target to BenchmarkDotNet. The problem is that ProcessStartInfo.EnvironmentVariables is available for .NET Core 2.0+. Our goal is to set runtime knobs for .NET Core, without support for .NET Core it makes less sense ;)

Member

adamsitnik commented Sep 3, 2017

@AndreyAkinshin I will add the docs tomorrow. And I will most probably also add .NET Core 2.0 target to BenchmarkDotNet. The problem is that ProcessStartInfo.EnvironmentVariables is available for .NET Core 2.0+. Our goal is to set runtime knobs for .NET Core, without support for .NET Core it makes less sense ;)

@AndreyAkinshin

This comment has been minimized.

Show comment
Hide comment
@AndreyAkinshin

AndreyAkinshin Sep 4, 2017

Member

Awesome!

Member

AndreyAkinshin commented Sep 4, 2017

Awesome!

@adamsitnik adamsitnik closed this in 73a30a2 Sep 6, 2017

alinasmirnova added a commit to alinasmirnova/BenchmarkDotNet that referenced this issue Sep 22, 2018

alinasmirnova added a commit to alinasmirnova/BenchmarkDotNet that referenced this issue Sep 22, 2018

alinasmirnova added a commit to alinasmirnova/BenchmarkDotNet that referenced this issue Sep 22, 2018

alinasmirnova added a commit to alinasmirnova/BenchmarkDotNet that referenced this issue Sep 22, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment