[Core feature] Decorate manually deleted k8s resources with error message

[hamersaw]

Motivation: Why do you think this is important?

Flyte Pods (or other k8s resources depending on the plugin) may be delete by an external process in the background. Flyte can detect this in two scenarios (1) the k8s object is missing or (2) the k8s object is marked as "deleted". In the latter case we should attempt to decorate the Flyte error message with the cause of deletion. In Pods there is are reason and message fields which should contain more precise information. This would help users debug these cases.

Goal: What should the final outcome look like, ideally?

When detecting externally deleted k8s resources Flyte should do it's best to decorate the error message with the cause for deletion.

Describe alternatives you've considered

Leaving it as is, with no additional context.

Propose: Link/Inline OR Additional context

The deletion timestamp is handled in FlytePropeller and FlytePlugins k8s array tasks separately. A fix would be to "best effort" parse the object and decorate the error with a more actionable message.

Are you sure this issue hasn't been raised already?

• Yes

Have you read the Code of Conduct?

• Yes

[div-devs]

can you assign this to me please

[github-actions]

Hello 👋, this issue has been inactive for over 9 months. To help maintain a clean and focused backlog, we'll be marking this issue as stale and will engage on it to decide if it is still applicable.
Thank you for your contribution and understanding! 🙏