Improve the adaptation speed of polling algorithm

[SzymonPobiega]

• Added a bunch of simulation tests that are meant to verify various scenarios of adapting to load
• Refactored BackOff class to be testable
• Decouple the receiver and sender from the BehaviorContext in order to test them easier
• Centralized all the important parameters of the adaptive algorithm in AdaptivePollingReceiver class
• Added code for starting additional threads when the current thread is processing a slow message (> 5 seconds of processing time) so that the adaptive algorithm can keep adapting

[SzymonPobiega]

@Particular/engineers @johnsimons can you review?

[mikeminutillo]

Is the use of DateTime going to be problematic around Daylight Savings change-overs (i.e. Suddenly everything claims to be long running). You could do this with 2 flags. First check for "is long running" and "hasn't triggered another task" and trigger new tasks. Then mark everything as "long running". If they don't get reset before you get back around to them then they're long running.

[johnsimons]

Is the use of DateTime going to be problematic around Daylight Savings change-overs

Not if we use UTC

[mikeminutillo]

This works too :)

class TaskState
{
    int seen;

    public void Reset()
    {
        seen = 0;
    }

    public void TriggerIfFirstDetectedSlow(Action action)
    {
        seen +=1;
        if(seen == 2)
            action();
    }
}

[SzymonPobiega]

I like ❤️ ❤️ ❤️ the seen version. Thanks @johnsimons and @WolfByte !

[danielmarbach]

You little @andreasohlund you!

[andreasohlund]

Huh? :)

[danielmarbach]

See the unnecessary line break?

[SzymonPobiega]

Fixed. Ufff. Disaster averted.

[danielmarbach]

There is much complexity involved here. I wonder if it is really needed. The default TaskScheduler itself already has ramp up strategies built in. I wonder if we could just use that and do some smart connection pooling instead.

[SzymonPobiega]

Applied the suggestions.

[adamralph]

@SzymonPobiega I reviewed the PR on my flight on Friday and it all seemed to make sense.

BTW - I did some minor refactorings as I was reading the code, if you want to cherry pick them they are on https://github.com/adamralph/NServiceBus.SqlServer/commits/make-scheduler-adapt-faster-refactor

Before we merge, I'd like to second @danielmarbach's comment. Do we have evidence that this isn't already a solved problem .NET's default TaskScheduler?

[adamralph]

Woot! My first commits to core 😛

[SzymonPobiega]

@adamralph What about Rabbit?

[adamralph]

not core?

[adamralph]

Oops, this is sql... And I already have 3 commits in core.

Ignore me. As you were. 😊

[SzymonPobiega]

Be quick, merge this one: Particular/NServiceBus#2884

[yta1]

@ramonsmits and all, it looks like 2.1.4 fix[https://github.com//issues/90] was not merged into 2.2.0? If we upgrade we'll reintroduce the memory leak of 2.1.3: https://github.com/Particular/NServiceBus.SqlServer/blob/2.2.0/src/NServiceBus.SqlServer/AdaptiveExecutor.cs

[yta1]

My bad, it is merged in master.,,