Krator Does not use Notify Properly

[kflansburg]

In krator/src/runtime.rs, tokio::sync::Notify is expected to always return instantly after it has been notified, rather than its actual behavior of waiting for a new notification. This results in the task getting stuck in some cases waiting for events to happen when they already have. Change to using a RwLock<bool>, or something similar.

[kflansburg]

Confirmed still an issue.

To reproduce:

1. Switch to Nightly Rust (required by Tokio console)

2. Checkout branch kflansburg/tokio-console, then run:

RUSTFLAGS="--cfg tokio_unstable" RUST_LOG=tokio::task=trace cargo run --example moose --features=derive -- --console

3. Clone and run console in another shell.

4. Use examples/assets/generate_random_meese.py 10 to create 10+ resources managed by the controller. (there is a requirements file to install Python deps).

5. Delete all resources: kubectl delete mooses --all.

You should observe a tasks related to individual resources stick around forever. There are two main task entrypoints:

• src/runtime.rs:222
• src/runtime.rs:181

The one at line 222 is the one getting stuck, and I believe it is because of the notification behavior mentioned above.