Refactor health check service to be more responsive

[JamesNK]

Description

Fixes #6112

The health check service immediately checks health when a resource moves to a running state. However, if the intial report is unhealthy then it waits 5 seconds before checking again. This PR refactors the health check service to be more responsive.

• If a health report isn't healthy then the delay interval starts at 1 second and increases by 1 second per failure up to the existing 5 sec. Healthy delay is still 30 seconds.
• Refactored the health check loop to use an instant interrupt instead of polling the state once per second. The DelayAsync method is complicated, but the actual health monitoring loop is much simpler.
• Allow setting the health delays from tests so time based tests run a lot faster. No more 30 second tests.
• No more blocking in tests: AutoResetEvent 👎

Checklist

• Is this feature complete?
  • Yes. Ready to ship.
  • No. Follow-up changes expected.
• Are you including unit tests for the changes and scenario tests if relevant?
  • Yes
  • No
• Did you add public API?
  • Yes
    • If yes, did you have an API Review for it?
      • Yes
      • No
    • Did you add <remarks /> and <code /> elements on your triple slash comments?
      • Yes
      • No
  • No
• Does the change make any security assumptions or guarantees?
  • Yes
    • If yes, have you done a threat model and had a security review?
      • Yes
      • No
  • No
• Does the change require an update in our Aspire docs?
  • Yes
    • Is this introducing a breaking change?
      • Yes
        • Link to aspire-docs issue (please use this breaking-change template):
      • No
        • Link to aspire-docs issue (please use this doc-idea template):
  • No

[davidfowl]

Nit about adding comments but LGTM

[eerhardt]

This code isn't multi-threaded, right? There is just 1 thread/loop processing these events. So is there a need to lock here?

[eerhardt]

I guess there is an internal GetResourceMonitorState that is used for tests that could be on a different thread.

[JamesNK]

Yes. I didn't want tests to fail because the dictionary was in a bad state during a get.

The lock will never be contended in the real world, and this isn't performance critical code, so it seemed better to add some safety in the cause of test stability.