HashedWheelTimerScheduler is spending 35% of execution time in sleep

[IgorFedchenko]

I used Akka.IO TcpOperationBenchmark as a sample to apply profiling, and it turns out that it is spending ~35% of time in sleep (Sleep function is my wrapper over Thread.Sleep that is used in scheduler). See the picture:

So, 1/3 of execution time scheduler is waiting for the next tick. This may slow down actors messaging pretty much. I am going to open PR for it.

[IgorFedchenko]

I have added simple static public fields to track how much scheduler should wait (in ticks, ideally), and how much is looses because of inaccurate windows clock.
Like this:

                    var stopWatch = Stopwatch.StartNew();
                    
#if UNSAFE_THREADING
                    try
                    {
                        Thread.Sleep(TimeSpan.FromMilliseconds(sleepMs));
                    }
                    catch (ThreadInterruptedException)
                    {
                        if (_workerState == WORKER_STATE_SHUTDOWN)
                            return long.MinValue;
                    }
#else
                    Thread.Sleep(TimeSpan.FromMilliseconds(sleepMs));
#endif
                    
                    stopWatch.Stop();
                    _totalTicksRequiredToWaitStrict += deadline - currentTime;
                    _totalTicksRequiredToWait += sleepMs * TimeSpan.TicksPerMillisecond;
                    _totalTicksLost += stopWatch.ElapsedTicks - sleepMs * TimeSpan.TicksPerMillisecond;

It turned out, that:

1. _totalTicksRequiredToWait is about 1.01 * _totalTicksRequiredToWaitStrict to 1.1 * _totalTicksRequiredToWaitStrict , so we are loosing ~1-10% of sleep time because we have to round ticks to milliseconds to call Thread.Sleep. And _totalTicksRequiredToWaitStrict is 3+ seconds in my test, which means that it is just load loaded with work, that's why it is sleeping so much.
2. _totalTicksLost is almost always about 1.1 * _totalTicksRequiredToWait, so we are constantly adding 10% of time due to windows clock delays.

Scala is doing special rounding for Windows:

val sleepMs = if (Helpers.isWindows) (nanos + 4 999 999) / 10 000 000 * 10 else (nanos + 999 999) / 1 000 000

I will try this and some other ideas to reduce useless waiting.

Update: did not help. Still +10% to the expected waiting time.

[Aaronontheweb]

This is great work @IgorFedchenko - keep going with it.

[IgorFedchenko]

I think I have found partial solution for this problem, which may give some performance boost for long running actively scheduled scenarios. See my PR attached.

[Aaronontheweb]

I suspect this issue is responsible for some of the high idle CPU numbers we're seeing here: #5400 (comment)

[Aaronontheweb]

Marking this as "won't fix" - there isn't a cheaper way to do this. We've tried. Plus it barely registers on our large scale tests.