Add common model for sync and async processing

[odinserj]

This PR brings common model for executing both synchronous and asynchronous methods in background. This is the result of pain and frustration that accompanied me for several months, while I was trying hard to make this model robust even case of ThreadAbortException and ThreadInterruptException, simple to understand, maintain and operate, fully testable.

Today I'm committing the first part of this work, related to basic primitives: IBackgroundExecution is to provide execution loops, exception handling and proper logging. IBackgroundDispatcher to schedule work on thread pools, regardless of its nature, and BackgroundTaskScheduler to be able to execute asynchronous work on dedicated threads that is close as possible to the ThreadPoolTaskScheduler.

Soon I will create more commits that mix these abstractions with background processes and servers. This will result in the following improvements in the background processing:

• Much better logging. you will not get thousands of exceptions in case of a transient failures, thresholds are applied when messages are logged, starting from the info level. This will allow to have a built-in logger for Event Log, and have fewer users who disable the logging.
• Reliability even in case of ThreadAbortException or ThreadInterruptException. I know they shouldn't be used from user code, you know they shouldn't be called from user code. But bad things happen, and they will almost silently shutdown the processing. Now they are caught, aborts are reset when possible (but not during AppDomain unloads), and fatal message is logged when this isn't possible.
• Better handling of AppDomain unloads and process exits. We are listening now for these events to stop the processing earlier, giving some time for workers and other processes to complete their work before thread aborts are thrown to not to cause delays in processing.
• Server restart with the newly generated identifier. When Server A decided that Server B is dead, it (B) should remain dead and not to process anything else, regardless of any reasons (for example, B was disconnected for a long period of time, there was a time blip, etc). To keep the underlying machine busy, we will restart the Server B with the newly generated identifier to conduct further processing. Currently such servers are "resurrected" or acting as spies, processing jobs without any mention on the servers page.

I didn't mention any features related to asynchronous execution, because full support for it requires storage interfaces to be asynchronous as well, and this will only be possible in 2.0 due to the breaking nature of these changes. However, custom processes can be made asynchronous now, so this will be useful anyway, and this is great step forward.

• Same programming model for sync and async. This means you can have mix of synchronous and asynchronous processes at the same time. In V2 it will be possible to move your processes to the async world step-by-step (or not to move them at all).
• Simple and testable model for asynchronous processes. I've seen a lot of implementations of async processing pipelines with a bunch of TaskScheduler implementations, or limited only to ThreadPool (more on this in next item). This model will be based on piece of code you want to execute, and a configured concurrency. Dedicated threads or default thread pool – doesn't matter.
• Cooperative and preemptive execution can be mixed with no work item priorities or other complications. Since there's a task scheduler that can be run on dedicated threads, you can have multiple independent thread pools.

Relates to #150.

• Implement background processes and servers on the top of this API.
• Add a lot of unit tests.

[codecov-io]

Codecov Report

Merging #1082 into dev will decrease coverage by 4.15%.
The diff coverage is 43.07%.

Impacted Files	Coverage Δ
...gfire.Core/Processing/BackgroundDispatcherAsync.cs	0% <0%> (ø)
src/Hangfire.SqlServer/SqlServerStorage.cs	72.54% <0%> (+0.47%)	⬆️
src/Hangfire.Core/BackgroundJobServer.cs	0% <0%> (ø)	⬆️
...fire.Core/Server/ServerProcessDispatcherBuilder.cs	0% <0%> (ø)
.../Server/BackgroundProcessDispatcherBuilderAsync.cs	0% <0%> (ø)
...angfire.Core/Processing/BackgroundTaskScheduler.cs	1.14% <1.14%> (ø)
src/Hangfire.Core/Server/Worker.cs	87.77% <100%> (+0.82%)	⬆️
src/Hangfire.Core/BackgroundJobServerOptions.cs	95.91% <100%> (ø)	⬆️
src/Hangfire.Core/Processing/TaskExtensions.cs	19.35% <19.35%> (ø)
...angfire.Core/Server/BackgroundProcessExtensions.cs	33.33% <33.33%> (ø)
... and 23 more

[pieceofsummer]

Need more awaits! :)

[odinserj]

Yeah, more awaits is more reliable 🤡