When running in a k8s cluster with mutating webhooks, tasks don't finish since the side car is still working [Core Feature]

[NotMatthewGriffin]

Motivation: Why do you think this is important?
Currently when running a flyte task in a kubernetes cluster that uses mutating webhooks to add sidecars to all pods, the task and associated pod will run until terminated in the flyte web console. Inside of the pod, the container actually running the flyte task has completed and exited with a good status but the sidecar container is still running. To work around this a user must specify the task in their workflow as a pod task like @task(task_config=Pod(pod_spec=pod_spec, pimary_container_name="container_name")). This means that users need to know the configuration of the cluster ( like will any sidecars be added automatically) in order to successfully create/run a flyte workflow.

Goal: What should the final outcome look like, ideally?
Ideally a user of flyte wouldn't need to declare their task as a pod task in order to run it in a kubernetes cluster that uses mutating webhooks to add sidecars to pods. The user could declare their task like @task and it would run the flyte task pod until the flyte task container finishes then the rest of the workflow would be run.

Describe alternatives you've considered
The only other alternative I've considered is continuing to use the workaround of declaring all workflows/tasks as pod tasks.

[welcome]

Thank you for opening your first issue here! 🛠

[kumare3]

@NotMatthewGriffin thank you for the issue. I have heard from a few users about this issue. The only problem is there are a few folks that run @task on a non k8s cluster and the container abstraction keeps it simple. But, it is completely possible to ensure that on K8s we can assert that the main container exit is the only signal that is needed to mark completion.

How urgent do you think this work is?

[kumare3]

can we slot this for november?

[NotMatthewGriffin]

@kumare3 I don't think this issue is show stopping for anyone since we know of the workaround. Mostly its just a little inconvenient to require extra configuration for running in some k8s clusters and not others.

I would be happy if this was addressed in November :)

If this is something I could help address I would also be happy to help.

[github-actions]

Hello 👋, This issue has been inactive for over 9 months. To help maintain a clean and focused backlog, we'll be marking this issue as stale and will close the issue if we detect no activity in the next 7 days. Thank you for your contribution and understanding! 🙏

[github-actions]

Hello 👋, This issue has been inactive for over 9 months and hasn't received any updates since it was marked as stale. We'll be closing this issue for now, but if you believe this issue is still relevant, please feel free to reopen it. Thank you for your contribution and understanding! 🙏