Create a route to terminate a dapr sidecar

[gafeol]

In what area(s)?

/area operator

Describe the feature

Supporting a request that will allow the termination of sidecars, like the POST on /quitquitquit for Istio sidecars.

This would be useful for CronJobs, for instance:
Without a sidecar, a Pod for a CronJob would reach the state "Completed" after the task was done. A pod with a sidecar, on the other hand, never reaches the "Completed" state, because the sidecar is not terminated at the same time as the original task is completed. This causes the CronJob Pod to remain permanently on a "Running" state.
This is specially bad if one is using the "Forbid" concurrencyPolicy for Cronjobs, since a CronJob would not be able to run a second time if it has never finished in the first place.

If I'm not mistaken, there’s a way to kill Istio sidecars by sending a POST request to /quitquitquit, so I was wondering if there could be a similar feature on Dapr.

I found related issues to this problem on kubernetes and on istio.

[yaron2]

@gafeol do you think this poses a security risk where malicious code can shut down the Dapr container?

[gafeol]

I believe you're right, having such an endpoint would allow any code inside container to disable the sidecar.
Still, there should be a way to terminate sidecars when the main container terminates, so that CronJobs can successfully restart.
Would you have any suggestions on how to solve this?

[jjcollinge]

One idea, you could use a similar mechanism to https://github.com/karlkfi/kubexit, where the sidecar injector wraps the "app" process as a "supervisor" and writes a terminal status (tombstone) to a shared volume or dapr endpoint directly with the app's terminal status. Once dapr has detected this terminal status, it can gracefully shuts down if appropriate. You would limit this to pods in cron[jobs].

[jjcollinge]

I guess the above tombstoning approach could also be susceptible to malicious code too. At a basic level, you could even just send signals (SIGTERM) from the supervisor to the app but that's not particularly secure either.

I think it's probably best to explore trying to use container lifecycle hooks such as preStop and a dapr endpoint with some level of authentication (maybe just a shared secret generated during deployment). That way only the hook should be able to terminate the sidecar.

[jjcollinge]

#1030 would enable us to have an authenticated kill endpoint. It would have to assume the token isn't leaked by the the app though.

[artursouza]

Since we have Dapr API token now, this endpoint should be safe. A reminder, it should work on HTTP, GRPC and from every SDK.

[artursouza]

This was closed as part of the work done in #2689