Clean-up phase in krator fixed where a channel was closed prematurely

[siegfriedweber]

Fixes #556

[kflansburg]

Thanks for catching this. It sounds like keeping the manifest object around until the end of run_object_task has fixed the issue for you, but I would expect there to still be a race condition here. Do you think that we should await at the end of run_object_task until the Deleted event arrives?

[siegfriedweber]

I am fairly new to Rust so treat my proposals with caution.

We have the following options:

1. Do not send a manifest after the resource was deleted, i.e. remove krator/src/runtime.rs#L207-L213.
2. Add another Notify channel, e.g. resource_task_terminated besides deleted, which is awaited at the end of run_object_task. The deleted channel is actually notified very often:
   • in the Running phase when the deletion_timestamp is applied
   • in the Terminated phase when the new status is applied
   • when the run_object_task deletes the resource and the deletion_timestamp without a grace period is applied
   • when the Deleted event is handled

I would opt for option 1 because the resource is already deregistered and nobody listens anymore to manifest updates.

[kflansburg]

I think its important that Deleted events get sent to the channel in case one arrives when Krator was not already shutting down. I think this happens when someone runs kubectl delete --force --grace-period=0.

I think my suggestion was that Manifest would have a new method on it that allows you to await for it to be deleted.

[kflansburg]

@siegfriedweber Let me know what you think of my commit. I think your solution makes sense, I just want to make sure that the Manifest isn't dropped until the Deleted event is received.

We now have a lot of Notifies, but I think that can be refactored later.

[siegfriedweber]

I fixed a copy&paste error and moved the notification of deleted_event after the manifest was sent.

[kflansburg]

LGTM