[feat] Updates and improvements to the poll control mechanism

[vkarak]

This PR updates and improves the poll control mechanism in different ways:

1. The poll interval (sleep duration) is not controlled directly anymore but rather through the poll rate.
2. The poll rate starts at a maximum value and decays towards a predefined minimum. At every step the desired poll rate is calculated and from this the required poll interval is derived.
3. The desired poll rate is reset every time a monitored job finishes, so as to capture quickly other jobs that have also finished (this is the same as before).
4. Support is added for adding random noise to the poll interval (disabled by default), by allowing the random decrease or increase of the poll interval. This is useful to avoid aggregate poll rate spikes when multiple parallel reframe processes execute the same test suite.
5. All the polling control parameters are exposed to the users and can be tuned
6. A script is added to the tools/ that allows to generate the plots for debugging polling issues.

Todos

• Add reference docs
• Add article explaining the polling mechanism
• Document the plot script
• Add requirements file in the tools/ directory
• Add unit tests for checking the validation of poll control parameters

Closes #211.

[codecov]

Codecov Report

❌ Patch coverage is 98.36066% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 91.28%. Comparing base (3d1ba5c) to head (37533e5).
⚠️ Report is 6 commits behind head on develop.

Files with missing lines	Patch %	Lines
reframe/frontend/executors/policies.py	98.24%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff             @@
##           develop    #3575      +/-   ##
===========================================
+ Coverage    91.26%   91.28%   +0.01%     
===========================================
  Files           62       62              
  Lines        13542    13581      +39     
===========================================
+ Hits         12359    12397      +38     
- Misses        1183     1184       +1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[victorusu]

lgtm