Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
0.10.4: Timing Oddities #433
I have gone down another timing rabbit hole, I am afraid. I am beginning to think that my machine is fubared and a reinstall will do me good. What makes me say this is that I recently created a performance VM, which has 6 of my 8 cores assigned to it, and it seems to deliver far better and more consistent results.
Upon upgrading to 0.10.4, my experience on my (possibly-fubar'd) machine seems much more chunky and slow. The results are themselves are now slower (but probably more accurate).
However, I am struggling to understand this, which is a concise example of the pain I have been enduring of late:
Notice that the main target jumps ~.03us from the warmup. This is surprisingly (and maddeningly) consistent, and by consistent I mean the amount, not the direction. Here is another example:
There is .03us again, but in the different direction.
Now take an example from my newly installed VM (that is indeed/ironically hosted on my possibly horked machine that produces the dubious results above):
This seems more appropriate and expected to me.
So I am beginning to wonder if there is a setting/configuration somewhere that I do not have set properly on my primary host, or perhaps a bad install. For reference, here are the stats from my host:
As you can see, I just upgraded to Creator Update which installs .NET 4.7 and the latest RyuJIT.
Here's the stats from my VM:
I noticed that the Timer is
Finally, I will also state that I still encounter wild discrepancies on my host machine. For instance from the results above I've noticed that it starts out at around ~1.10us and then after successive tries (about 10 or so) it will work its way down to ~1.00us (or even to 986ns at one point). As you can imagine, determining if a change I made an impact is nearly impossible from this process (or at the very least extremely time-consuming).
Conversely, the VM hangs steady at around 1.10-1.12us each and every time.
So, is it possible I have a bad install/configuration somewhere? One thought that occurs to me is that I do own a Rampage Extreme IV and there is a bunch of overclocking on it that I (read: a friend
So, still lost here, unfortunately. Any insight/assistance would be greatly appreciated.
This was referenced
Apr 24, 2017
Wanted to post an update here and report that indeed this appears to be related to two underlying issues of my own making:
So I have basically spent the week engaged in some applied spring cleaning, whether I liked it or not.
Now I am on a totally brand new environment, with QVL-d memory, and things run so much better. Additionally, I was pleasantly surprised to find that I could run BDN from Safe Mode.
That's the good news. The bad news is that I still see some weird behavior with my executions.
I didn't want to get into this earlier (or even now) with a lot of detail, but the oddity that I am seeing appears to stem from GC. It seems that running code that involves the use of a
Anyways, I am going to attempt to make a very simple project that hopefully outlines and captures this for you. I will update here once I do.
Ahhh to clarify here @adamsitnik I was running benchmarks in:
The VM and SP3 were showing vastly different results and behavior than my host. I have since refreshed my entire environment and everything seems so much better now, especially with the expected timings where the
FWIW I am running on Windows Server 2016 as a Hyper-V host. I switched to Server when I tried going into Safe Mode with Windows 10 (explicitly for benchmarking) and noticed that even there, Cortana and other services were still enabled. Didn't seem so "safe" to me and figured Server would be much more streamlined (and was right).
With that said, one final thought (for now): with all of this work of setting up services and a new machine (well SUITE of machines), I was thinking that this project could totally innovate this space (more than it already has!) by offering it as a service: Benchmarking-as-a-Service (Like a B-a-a-S! Hehe),
I have seen more and more articles and blogs using Benchmark.NET as its authoritative benchmark reference (and rightfully so). However, all the numbers are relative to the developers machine. If anything with all of my pain here it would seem that benchmarking is the antithesis of "works on my machine" as there are an incredible amount of variables at play that can impact results. How great would it be to have a service we could simply send our code to crunch and spit out an authoritative, 99.9999-percentile consistent/accurate number that can be referenced and known/understood by others?
Additionally, being able to scale it out and run it concurrently on numerous/multiple machines and instances would be invaluable in scenarios such as this to track down WTF is going on and/or if oddities are showing up elsewhere, too. :)
If I am not mistaken @AndreyAkinshin you work for JetBrains, correct? This would be something they should be all over, IMO. Additionally, it would be priceless to be able to integrate something like this with dotMemory/dotTrace to really get at the root of what is going on by having it also generate associated files, regardless.
@adamsitnik yeah it was just a thought and I am in agreement with your assessment. I am sure it will get built one way or another, in any case.
An update on this issue: my effort to refresh my digital life has spilled into my real life, so that's set me back a few weeks here. I have this issue flagged in my TODO list once I return and will aim to get you a simple, sample project when I can.
ALRIGHT. More than two months later...
Really could not have anticipated the sheer amount of offline time this would take. So much accomplished, however. I have completely refreshed my environment (both virtual and real life haha) and now things run much better and (more importantly) as expected.
I am not sure what exactly happened, and I am apt to not care at the point, only that it forced me to commit the significant undertaking that has just transpired. Now I am on the latest Windows 10 install along with Visual Studio 2017 Preview 3 and I am seeing the results that I expect with the same code.
So my best guess on what happened was the tooling along with some wonky OS glitchery (yes that's a word). I was probably running up against something lurking within the VS2015 .NET Core 1.0 tooling.
Anyways, everything is running MUCH better now. Even building projects is blazing fast compared to '15. Night and day difference. I will be closing this issue now as this problem has fixed itself. Thanks again all for your awesome assistance and (more importantly) patience!