adjust thread count based on hardware and scenario

[nathana1]

Changes some configuration so we should get near optimal perf against both the Azure environment and the physical environment.

/cc @DamianEdwards @mikeharder

[knewmanTE]

@nathana1 do you want to wait for feedback from @DamianEdwards and @mikeharder, or are you okay with us looking at this/merging it in?

[DamianEdwards]

Wondering whether we should mark these as "NETCore" rather than "NET". I could see us eventually having "NETCore", "NETFramework", and "Mono" as distinct platform categories. Thoughts?

[DamianEdwards]

And even further out "NETCoreAOT". Not sure if these are inline with other pivots of "platform" though, looking for guidance here.

[msmith-techempower]

"Mono" is not a platform; it is a language implementation which we either are or have implemented a fix for in the way of adding "flavor" to the benchmark_config.json files. In this case, the language would be "C#" and the flavor would be "Microsoft" (or something similar).

Unless "NET" and "NETCore" are distinct platforms for writing web applications, I do not see the need for the differentiation. It sounds like they are, and as such that should be reflected here.

I am not sure what "NETCoreAOT" is versus "NETCore".

[DamianEdwards]

As always, this stuff is nuanced 😄

.NET is a platform in the same way Java is. However I see in many other places that platform is used to indicate an application stack, independent of the language or runtime, e.g. Netty is used as a platform, although it's written in and runs on Java, similarly "Servlet" is a platform.

Also, the official description of platform seems to indicate it's the thing that provides the HTTP primitives the application is built on, rather than the underlying runtime or execution environment:

"platform" is the low-level software or API used to host web applications for the framework; the platform provides an implementation of the HTTP fundamentals.

In this vein, it could be viewed that the platform in our case is "ASP.NET Core", vs. "ASP.NET" or some other .NET-based web framework, e.g. Nancy, as ASP.NET Core defines the primitives on which the web application is written. Another .NET web application platform, like Nancy, has its own primitives.

"NETCoreAOT" is a future pivot where the application has been run through the .NET native compiler (AOT == Ahead of Time vs. JIT == Just in Time, in .NET parlance) and hence has a different .NET runtime (CoreRT) embedded in it, rather than running on the CLR or CoreCLR with JIT compilation, etc.

All that said, does it sound more like perhaps we should be classifying these along these lines:

• language = C#
• platform = aspnetcore for our tests. If Nancy submits tests, they'd be "nancy"
• classification = "Platform" for when we use Kestrel (our server) directly without middleware at all (we don't have a test for this right now), "Micro" for raw middleware or routing, "FullStack" for MVC controllers
• front-end-server = None in our case, as the framework includes a server in-process (Kestrel)
• flavor = CoreCLR or CLR or CoreRT or Mono

?

[DamianEdwards]

Reading more of the existing results, I totally left out "framework" (doh).

With that in mind, perhaps the following is better:

• language = C#
• platform = NET
• framework = aspnetcore for our tests. If Nancy submits tests, they'd be "nancy"
• classification = "Platform" for when we use Kestrel (our server) directly without middleware at all (we don't have a test for this right now), "Micro" for raw middleware or routing, "FullStack" for MVC controllers
• front-end-server = None in our case, as the framework includes a server in-process (Kestrel)
• flavor = CoreCLR or CLR or CoreRT or Mono

[msmith-techempower]

.NET is a platform in the same way Java is
Java should not be classified as a platform. I realize that this classification has been murky in the past, and we are actively trying to clean it up with this branch.

Otherwise, I agree with your proposed setup of language/platform/framework/etc.

[DamianEdwards]

I only have some unrelated comments really, about classification, not the actual substance of this PR.

[ashawnbandy-te-tfb]

[EDIT]
@DamianEdwards @msmith-techempower What remains to be done for this PR?

[nathana1]

Chatted with @DamianEdwards offline and finalized on our classification. We should be fine merging with the last commit.