Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prevent termination of pods with long-running functions using KEDA #179

Open
conreaux opened this issue Jul 20, 2023 · 2 comments
Open

Prevent termination of pods with long-running functions using KEDA #179

conreaux opened this issue Jul 20, 2023 · 2 comments
Labels
enhancement New feature or request external Not strictly related to this project

Comments

@conreaux
Copy link

conreaux commented Jul 20, 2023

What is the general guidance for preventing termination of pods with long-running functions while using KEDA?
In testing, I've seen that during a scale down, the HPA can terminate a pod that is still running an orchestration/activity function.
This behavior is documented here: https://keda.sh/docs/1.4/concepts/scaling-deployments/#long-running-executions
What's the recommended method to prevent this from occurring?

@conreaux
Copy link
Author

Bumping topic. Also, how does this compare with MSSQL provider scale-in on the Elastic Premium and Consumption plans? Does it behave as described here where "functions that are currently executing are given time to finish executing." Does this need to be explicitly handled by the user with KEDA?

@cgillum
Copy link
Member

cgillum commented Aug 31, 2023

In the case of Azure Functions hosting, the Azure Functions runtime is able to detect when a function is still running and will try to not tear down the container until execution has finished.

It looks like there is more work that we'd need to do to support something similar in Kubernetes. The suggestion to respond to lifecycle hooks seems promising. For Azure Functions, however, the runtime would probably need to be augmented to support this. I don't think there's anything this MSSQL extension could do to prevent the host from shutting down, other than potentially exposing some signal indicating whether there is still work pending (which could be leveraged by some 3rd party component that responds to container lifecycle events).

@cgillum cgillum added enhancement New feature or request external Not strictly related to this project and removed P1 Priority 1 labels Aug 31, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request external Not strictly related to this project
Projects
None yet
Development

No branches or pull requests

3 participants