Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update quick install script to add Cluster Autoscaler and istiod memory leakage fix fields #2609

Merged
merged 1 commit into from
Feb 5, 2023

Conversation

swapkh91
Copy link
Contributor

Signed-off-by: Swapnesh Khare swapkh91@gmail.com

What this PR does / why we need it:

  1. Kubernetes Cluster Autoscaler (tested on GKE) expects pods to have cluster-autoscaler.kubernetes.io/safe-to-evict: "true" annotation to mark them safe for eviction from a node and scale down the node (ref). Istio specific pods do not have these annotations and hence are not scaled down if created on a new node during autoscaling.

  2. As of Istio version 1.15, the istiod component suffers from memory leakage (ref), which is fixed in 1.16 (ref) by disabling PILOT_ENABLE_CONFIG_DISTRIBUTION_TRACKING environment variable.

  3. istiod now takes 0.5M CPU and 2GB memory by default. These can be changed by setting fields in k8s key.

Type of changes
Please delete options that are not relevant.

  • Installation fix (fix or feature that would cause existing functionality to not work as expected)

Feature/Issue validation/testing:

  • Ran quick_install.sh with the updated script on cluster with Kubernetes version 1.22 (1.22.16-gke.1300)

  • Checked istiod and istio-ingressgateway pods to contain safe-to-evict annotation

  • Cluster Autoscaler successfully downscaled a new node

  • Logs
    Annotation set
    image

CPU and memory leakage before setting PILOT_ENABLE_CONFIG_DISTRIBUTION_TRACKING flag false
image

Release note:


Signed-off-by: Swapnesh Khare <swapkh91@gmail.com>
@yuzisun
Copy link
Member

yuzisun commented Feb 5, 2023

thanks @swapkh91 !

/lgtm
/approve

@yuzisun
Copy link
Member

yuzisun commented Feb 5, 2023

thanks @swapkh91 !

/lgtm
/approve

@kserve-oss-bot
Copy link
Collaborator

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: swapkh91, yuzisun

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@yuzisun yuzisun merged commit b15cd9e into kserve:master Feb 5, 2023
sriumcp pushed a commit to sriumcp/kserve that referenced this pull request Mar 7, 2023
Signed-off-by: Swapnesh Khare <swapkh91@gmail.com>
Signed-off-by: Srinivasan Parthasarathy <spartha@us.ibm.com>
Iamlovingit pushed a commit to Iamlovingit/kserve that referenced this pull request Oct 1, 2023
Signed-off-by: Swapnesh Khare <swapkh91@gmail.com>
Signed-off-by: iamlovingit <freecode666@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants