New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow the users to compare 32 vs 64 RyuJit for .NET Core #534

Closed
adamsitnik opened this Issue Aug 26, 2017 · 5 comments

Comments

Projects
None yet
2 participants
@adamsitnik
Member

adamsitnik commented Aug 26, 2017

As of now if the default .NET Sdk is configured to x86, then the user can run only x86 benchmarks. So no 32 vs 64 with single config ;(

I have reported the issue at dotnet cli repo.

The only idea I have now is to ask the user to provide the custom path for dotnet cli. But I don't want to do it that way..

@adamsitnik adamsitnik self-assigned this Aug 26, 2017

@AndreyAkinshin

This comment has been minimized.

Show comment
Hide comment
@AndreyAkinshin

AndreyAkinshin Aug 28, 2017

Member

An idea about the "x86 vs x32" .NET Core problem: we can detect all available dotnet.exe (e.g. via where dotnet, it should be available on all modern versions of Windows). Next, we can check the architecture of each dotnext.exe (e.g. like this, it works perfectly for C:\Program Files\dotnet\dotnet.exe and C:\Program Files (x86)\dotnet\dotnet.exe on my Windows 10). Finally, we can use dotnet.exe which is matched to the user config (or print a message that says something like "sorry, we didn't find appropriate runtime").
@adamsitnik, what do you think?

Member

AndreyAkinshin commented Aug 28, 2017

An idea about the "x86 vs x32" .NET Core problem: we can detect all available dotnet.exe (e.g. via where dotnet, it should be available on all modern versions of Windows). Next, we can check the architecture of each dotnext.exe (e.g. like this, it works perfectly for C:\Program Files\dotnet\dotnet.exe and C:\Program Files (x86)\dotnet\dotnet.exe on my Windows 10). Finally, we can use dotnet.exe which is matched to the user config (or print a message that says something like "sorry, we didn't find appropriate runtime").
@adamsitnik, what do you think?

@adamsitnik

This comment has been minimized.

Show comment
Hide comment
@adamsitnik

adamsitnik Aug 28, 2017

Member

I would like to hear more from the dotnet cli team first. I created issue for that. If this problem can not be solved on dotnet cli level then we can try your idea or we could introduce similar param to what we did for Mono (custom path).

I had exactly the same problem with dnx, but with dnx it was possible to use dvvm to specify the architecture.

Member

adamsitnik commented Aug 28, 2017

I would like to hear more from the dotnet cli team first. I created issue for that. If this problem can not be solved on dotnet cli level then we can try your idea or we could introduce similar param to what we did for Mono (custom path).

I had exactly the same problem with dnx, but with dnx it was possible to use dvvm to specify the architecture.

@adamsitnik

This comment has been minimized.

Show comment
Hide comment
@adamsitnik

adamsitnik Sep 10, 2017

Member

Done.

Users are supposed to provide custom dotnet cli path. I did not implement any discovery because it would be hard to do it right and the gain would be very small.

Sample config:

[Config(typeof(CustomPathsConfig))]
[DisassemblyDiagnoser]
public class Jit_RyuJitX86
{
    public class CustomPathsConfig : ManualConfig
    {
        public CustomPathsConfig() 
        {
            var dotnetCli32bit = NetCoreAppSettings
                .NetCoreApp20
                .WithCustomDotNetCliPath(@"C:\Program Files (x86)\dotnet\dotnet.exe", "32 bit cli");

            var dotnetCli64bit = NetCoreAppSettings
                .NetCoreApp20
                .WithCustomDotNetCliPath(@"C:\Program Files\dotnet\dotnet.exe", "64 bit cli");

            Add(Job.RyuJitX86.With(CsProjCoreToolchain.From(dotnetCli32bit)).WithId("32 bit cli"));
            Add(Job.RyuJitX64.With(CsProjCoreToolchain.From(dotnetCli64bit)).WithId("64 bit cli"));
        }
    }

    [Params(false, true)]
    public bool CallStopwatchTimestamp { get; set; }

    [GlobalSetup]
    public void GlobalSetup()
    {
        if (CallStopwatchTimestamp)
            Stopwatch.GetTimestamp();
    }

    private const int IterationCount = 10001;

    [Benchmark(OperationsPerInvoke = IterationCount)]
    public string WithStopwatch()
    {
        double a = 1, b = 1;
        var sw = new Stopwatch();
        for (int i = 0; i < IterationCount; i++)
        {
            // fld1
            // fadd        qword ptr [ebp-0Ch]
            // fstp        qword ptr [ebp-0Ch]
            a = a + b;
        }
        return string.Format("{0}{1}", a, sw.ElapsedMilliseconds);
    }

    [Benchmark(OperationsPerInvoke = IterationCount)]
    public string WithoutStopwatch()
    {
        double a = 1, b = 1;
        for (int i = 0; i < IterationCount; i++)
        {
            // fld1
            // faddp       st(1),st
            a = a + b;
        }
        return string.Format("{0}", a);
    }
}

Sample results:

BenchmarkDotNet=v0.10.9.20170910-develop, OS=Windows 10 Redstone 1 (10.0.14393)
Processor=Intel Core i7-6600U CPU 2.60GHz (Skylake), ProcessorCount=4
Frequency=2742185 Hz, Resolution=364.6727 ns, Timer=TSC
.NET Core SDK=2.1.0-preview1-007074
  [Host]     : .NET Core 2.0.0 (Framework 4.6.00001.0), 64bit RyuJIT
  32 bit cli : .NET Core 2.0.0 (Framework 4.6.00001.0), 32bit RyuJIT
  64 bit cli : .NET Core 2.0.0 (Framework 4.6.00001.0), 64bit RyuJIT

Jit=RyuJit  
Method Job Platform Toolchain CallStopwatchTimestamp Mean Error StdDev
WithStopwatch 32 bit cli X86 32 bit cli False 2.862 ns 0.0552 ns 0.0657 ns
WithoutStopwatch 32 bit cli X86 32 bit cli False 2.771 ns 0.0435 ns 0.0407 ns
WithStopwatch 64 bit cli X64 64 bit cli False 1.314 ns 0.0194 ns 0.0172 ns
WithoutStopwatch 64 bit cli X64 64 bit cli False 1.256 ns 0.0187 ns 0.0175 ns
WithStopwatch 32 bit cli X86 32 bit cli True 2.841 ns 0.0568 ns 0.0995 ns
WithoutStopwatch 32 bit cli X86 32 bit cli True 2.765 ns 0.0361 ns 0.0338 ns
WithStopwatch 64 bit cli X64 64 bit cli True 1.298 ns 0.0256 ns 0.0314 ns
WithoutStopwatch 64 bit cli X64 64 bit cli True 1.309 ns 0.0179 ns 0.0168 ns
Member

adamsitnik commented Sep 10, 2017

Done.

Users are supposed to provide custom dotnet cli path. I did not implement any discovery because it would be hard to do it right and the gain would be very small.

Sample config:

[Config(typeof(CustomPathsConfig))]
[DisassemblyDiagnoser]
public class Jit_RyuJitX86
{
    public class CustomPathsConfig : ManualConfig
    {
        public CustomPathsConfig() 
        {
            var dotnetCli32bit = NetCoreAppSettings
                .NetCoreApp20
                .WithCustomDotNetCliPath(@"C:\Program Files (x86)\dotnet\dotnet.exe", "32 bit cli");

            var dotnetCli64bit = NetCoreAppSettings
                .NetCoreApp20
                .WithCustomDotNetCliPath(@"C:\Program Files\dotnet\dotnet.exe", "64 bit cli");

            Add(Job.RyuJitX86.With(CsProjCoreToolchain.From(dotnetCli32bit)).WithId("32 bit cli"));
            Add(Job.RyuJitX64.With(CsProjCoreToolchain.From(dotnetCli64bit)).WithId("64 bit cli"));
        }
    }

    [Params(false, true)]
    public bool CallStopwatchTimestamp { get; set; }

    [GlobalSetup]
    public void GlobalSetup()
    {
        if (CallStopwatchTimestamp)
            Stopwatch.GetTimestamp();
    }

    private const int IterationCount = 10001;

    [Benchmark(OperationsPerInvoke = IterationCount)]
    public string WithStopwatch()
    {
        double a = 1, b = 1;
        var sw = new Stopwatch();
        for (int i = 0; i < IterationCount; i++)
        {
            // fld1
            // fadd        qword ptr [ebp-0Ch]
            // fstp        qword ptr [ebp-0Ch]
            a = a + b;
        }
        return string.Format("{0}{1}", a, sw.ElapsedMilliseconds);
    }

    [Benchmark(OperationsPerInvoke = IterationCount)]
    public string WithoutStopwatch()
    {
        double a = 1, b = 1;
        for (int i = 0; i < IterationCount; i++)
        {
            // fld1
            // faddp       st(1),st
            a = a + b;
        }
        return string.Format("{0}", a);
    }
}

Sample results:

BenchmarkDotNet=v0.10.9.20170910-develop, OS=Windows 10 Redstone 1 (10.0.14393)
Processor=Intel Core i7-6600U CPU 2.60GHz (Skylake), ProcessorCount=4
Frequency=2742185 Hz, Resolution=364.6727 ns, Timer=TSC
.NET Core SDK=2.1.0-preview1-007074
  [Host]     : .NET Core 2.0.0 (Framework 4.6.00001.0), 64bit RyuJIT
  32 bit cli : .NET Core 2.0.0 (Framework 4.6.00001.0), 32bit RyuJIT
  64 bit cli : .NET Core 2.0.0 (Framework 4.6.00001.0), 64bit RyuJIT

Jit=RyuJit  
Method Job Platform Toolchain CallStopwatchTimestamp Mean Error StdDev
WithStopwatch 32 bit cli X86 32 bit cli False 2.862 ns 0.0552 ns 0.0657 ns
WithoutStopwatch 32 bit cli X86 32 bit cli False 2.771 ns 0.0435 ns 0.0407 ns
WithStopwatch 64 bit cli X64 64 bit cli False 1.314 ns 0.0194 ns 0.0172 ns
WithoutStopwatch 64 bit cli X64 64 bit cli False 1.256 ns 0.0187 ns 0.0175 ns
WithStopwatch 32 bit cli X86 32 bit cli True 2.841 ns 0.0568 ns 0.0995 ns
WithoutStopwatch 32 bit cli X86 32 bit cli True 2.765 ns 0.0361 ns 0.0338 ns
WithStopwatch 64 bit cli X64 64 bit cli True 1.298 ns 0.0256 ns 0.0314 ns
WithoutStopwatch 64 bit cli X64 64 bit cli True 1.309 ns 0.0179 ns 0.0168 ns

@adamsitnik adamsitnik added this to the v0.10.10 milestone Sep 10, 2017

@AndreyAkinshin

This comment has been minimized.

Show comment
Hide comment
@AndreyAkinshin

AndreyAkinshin Sep 11, 2017

Member

Great, thanks!
However, discovering still can be a great feature, it will allow making benchmarks more portable. Maybe we can create another up-for-grabs issue?

Member

AndreyAkinshin commented Sep 11, 2017

Great, thanks!
However, discovering still can be a great feature, it will allow making benchmarks more portable. Maybe we can create another up-for-grabs issue?

@adamsitnik

This comment has been minimized.

Show comment
Hide comment
@adamsitnik

adamsitnik Sep 11, 2017

Member

@AndreyAkinshin it would require handling Windows/Mac/Linux and custom paths. What if user has few dotnet cli installed? Which should we choose then?

It's possible to handle all of these situations but so far I was the only person that wanted to use it, the chance for having some contributor willing to implement it (and do it correctly) is very low to me.

On the other hand, now we can allow people to experiment with different versions of dotnet cli.

Member

adamsitnik commented Sep 11, 2017

@AndreyAkinshin it would require handling Windows/Mac/Linux and custom paths. What if user has few dotnet cli installed? Which should we choose then?

It's possible to handle all of these situations but so far I was the only person that wanted to use it, the chance for having some contributor willing to implement it (and do it correctly) is very low to me.

On the other hand, now we can allow people to experiment with different versions of dotnet cli.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment