[etcd] Increase probe thresholds for better recovery

[kvaps]

What this PR does

Increases etcd probe thresholds to allow more time for cluster synchronization after pod restarts. This addresses issues where etcd members were being killed by startup probes before they could fully sync with the cluster, especially when VPA assigns minimal resources.

Changes:

• startupProbe.failureThreshold: 3 → 300 (allows 25 minutes for initial sync)
• livenessProbe.failureThreshold: default → 10 (reduces unnecessary restarts)
• readinessProbe.failureThreshold: default → 3

Release note

[etcd] Increase probe thresholds to prevent premature pod termination during cluster synchronization

Summary by CodeRabbit

• Chores
  • Improved etcd service stability with automatic health monitoring and failover detection capabilities in Kubernetes environments.

_{✏️ Tip: You can customize this high-level summary in your review settings.}

[coderabbitai]

Note

Other AI code review bot(s) detected

CodeRabbit has detected other AI code review bot(s) in this pull request and will avoid duplicating their findings in the review comments. This may lead to a less comprehensive review.

📝 Walkthrough

Walkthrough

Kubernetes probes (startup, liveness, readiness) are added to the etcd container configuration in the etcd-cluster.yaml template, enabling automated health checks with specified failure thresholds and check intervals.

Changes

Cohort / File(s)	Summary
Kubernetes Probe Configuration packages/extra/etcd/templates/etcd-cluster.yaml	Added three container probes: startupProbe (300 failures, 5s interval), livenessProbe (10 failures, 10s interval), and readinessProbe (3 failures, 5s interval) to etcd container

Estimated code review effort

🎯 1 (Trivial) | ⏱️ ~3 minutes

Poem

🐰 Hop hop, the etcd now beats so true,
With startup, liveness, readiness probes in view!
Health checks galore, a three-fold embrace,
Kubernetes knows when to keep pace! 💚✨

🚥 Pre-merge checks | ✅ 3

✅ Passed checks (3 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title accurately describes the main change: increasing etcd probe thresholds for better recovery, which aligns with the PR objectives of preventing premature pod termination during cluster synchronization.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[gemini-code-assist]

Summary of Changes

Hello @kvaps, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the resilience and stability of etcd clusters by adjusting the probe thresholds within the etcd-cluster.yaml template. The primary objective is to provide etcd pods with sufficient time to synchronize and stabilize after restarts, particularly in environments with tight resource allocation, thereby preventing premature termination and improving overall cluster health.

Highlights

• etcd Startup Probe Threshold Increased: The startupProbe.failureThreshold for etcd pods has been significantly increased from 3 to 300. This change extends the initial synchronization window to 25 minutes (300 failures * 5 seconds/period), which is crucial for preventing premature termination of etcd members during startup, especially when resource-constrained by VPA.
• etcd Liveness Probe Threshold Configured: The livenessProbe.failureThreshold has been explicitly set to 10. This adjustment aims to reduce unnecessary restarts of etcd pods by allowing more transient failures before a restart is triggered, improving stability.
• etcd Readiness Probe Threshold Configured: The readinessProbe.failureThreshold has been explicitly set to 3. This ensures that etcd pods are not marked as ready until they are fully functional, while still providing a degree of tolerance for minor, temporary issues.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request aims to improve etcd cluster recovery by increasing the probe thresholds. The new values for startupProbe, livenessProbe, and readinessProbe seem reasonable and well-justified for preventing premature pod termination. However, I've identified a critical issue where the probe definitions are incomplete, lacking the required probe handlers (exec, httpGet, etc.). This will likely cause pod creation to fail. My review includes a suggestion to add the necessary handlers to ensure the configuration is valid.

[github-actions]

Successfully created backport PR for release-0.40:

• [Backport release-0.40] [etcd] Increase probe thresholds for better recovery #1878