Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feat: add readiness and liveness probes to crossplane #4748

Merged
merged 2 commits into from Oct 9, 2023
Merged

feat: add readiness and liveness probes to crossplane #4748

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
f2e382e
feat: add readiness and liveness probes to crossplane
phisco Oct 9, 2023
990228f
chore: all lower case errors
phisco Oct 9, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
12 changes: 10 additions & 2 deletions cluster/charts/crossplane/templates/deployment.yaml
View file
Open in desktop
Expand Up @@ -123,9 +123,17 @@ spec:
name: {{ .Chart.Name }}
resources:
{{- toYaml .Values.resourcesCrossplane | nindent 12 }}
{{- if or .Values.metrics.enabled (.Values.webhooks.enabled) }}
livenessProbe:
httpGet:
path: /healthz
port: healthz
readinessProbe:
httpGet:
path: /readyz
port: healthz
ports:
{{- end }}
- name: healthz
containerPort: 5000
{{- if .Values.metrics.enabled }}
- name: metrics
containerPort: 8080
Expand Down
46 changes: 39 additions & 7 deletions cmd/crossplane/core/core.go
View file
Open in desktop
Expand Up @@ -36,6 +36,7 @@ import (
ctrl "sigs.k8s.io/controller-runtime"
"sigs.k8s.io/controller-runtime/pkg/cache"
"sigs.k8s.io/controller-runtime/pkg/client"
"sigs.k8s.io/controller-runtime/pkg/healthz"
"sigs.k8s.io/controller-runtime/pkg/webhook"

"github.com/crossplane/crossplane-runtime/pkg/certificates"
Expand Down Expand Up @@ -148,7 +149,7 @@ func (c *startCommand) Run(s *runtime.Scheme, log logging.Logger) error { //noli

cfg, err := ctrl.GetConfig()
if err != nil {
return errors.Wrap(err, "Cannot get config")
return errors.Wrap(err, "cannot get config")
}

cfg.WarningHandler = rest.NewWarningWriter(os.Stderr, rest.WarningWriterOptions{
Expand Down Expand Up @@ -188,9 +189,11 @@ func (c *startCommand) Run(s *runtime.Scheme, log logging.Logger) error { //noli
LeaderElectionResourceLock: resourcelock.LeasesResourceLock,
LeaseDuration: func() *time.Duration { d := 60 * time.Second; return &d }(),
RenewDeadline: func() *time.Duration { d := 50 * time.Second; return &d }(),

HealthProbeBindAddress: ":5000",
})
if err != nil {
return errors.Wrap(err, "Cannot create manager")
return errors.Wrap(err, "cannot create manager")
}

o := controller.Options{
Expand Down Expand Up @@ -250,7 +253,7 @@ func (c *startCommand) Run(s *runtime.Scheme, log logging.Logger) error { //noli
filepath.Join(c.TLSClientCertsDir, corev1.TLSPrivateKeyKey),
false)
if err != nil {
return errors.Wrap(err, "Cannot load TLS certificates for external secret stores")
return errors.Wrap(err, "cannot load TLS certificates for external secret stores")
}

o.ESSOptions = &controller.ESSOptions{
Expand All @@ -267,7 +270,7 @@ func (c *startCommand) Run(s *runtime.Scheme, log logging.Logger) error { //noli
}

if err := apiextensions.Setup(mgr, ao); err != nil {
return errors.Wrap(err, "Cannot setup API extension controllers")
return errors.Wrap(err, "cannot setup API extension controllers")
}

po := pkgcontroller.Options{
Expand All @@ -282,13 +285,13 @@ func (c *startCommand) Run(s *runtime.Scheme, log logging.Logger) error { //noli
if c.CABundlePath != "" {
rootCAs, err := ParseCertificatesFromPath(c.CABundlePath)
if err != nil {
return errors.Wrap(err, "Cannot parse CA bundle")
return errors.Wrap(err, "cannot parse CA bundle")
}
po.FetcherOptions = append(po.FetcherOptions, xpkg.WithCustomCA(rootCAs))
}

if err := pkg.Setup(mgr, po); err != nil {
return errors.Wrap(err, "Cannot add packages controllers to manager")
return errors.Wrap(err, "cannot add packages controllers to manager")
}

// Registering webhooks with the manager is what actually starts the webhook
Expand All @@ -310,5 +313,34 @@ func (c *startCommand) Run(s *runtime.Scheme, log logging.Logger) error { //noli
}
}

return errors.Wrap(mgr.Start(ctrl.SetupSignalHandler()), "Cannot start controller manager")
if err := c.SetupProbes(mgr); err != nil {
return errors.Wrap(err, "cannot setup probes")
}

return errors.Wrap(mgr.Start(ctrl.SetupSignalHandler()), "cannot start controller manager")
}

// SetupProbes sets up the health and ready probes.
func (c *startCommand) SetupProbes(mgr ctrl.Manager) error {
// Add default readiness probe
if err := mgr.AddReadyzCheck("ping", healthz.Ping); err != nil {
return errors.Wrap(err, "cannot create ping ready check")
}

// Add default health probe
if err := mgr.AddHealthzCheck("ping", healthz.Ping); err != nil {
return errors.Wrap(err, "cannot create ping health check")
}

// Add probes waiting for the webhook server if webhooks are enabled
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a way to add something like a started checker for the controllers?
Could it be helpful to report as ready only after controllers are operational, e.g. after acquiring the lease successfully.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could also check/do that only after we're aware of a use case that it would help.
Feel free to ignore for now.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, we could definitely do something, but I couldn't find many examples of other projects doing much more than this. Acquiring the lease could not be the right condition though as we want to support multiple replicas serving webhooks, @sttts was talking about caches being populated, any additional hint/pointer to prior art?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think readiness is connected to leader election. It's a blackbox property whether the pod is ready to serve traffic (via the webhook). Controllers are always async. No need to wait for them. What does waiting for them even mean?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree, @sttts. What did you have in mind by "waiting for informers" here? I was referring to that when I asked if you had any prior reference implementation to share.

Copy link
Contributor

@sttts sttts Oct 9, 2023 •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If functionality of the process depends on informers to sync, then they should be part of readiness. E.g. admission in kube cannot work correctly without synced informers. Hence, the kube-apiserver might return inconsistent results prior to that.

Don't think it matters here much.

You could delay readiness by mgr.GetCache().WaitForSync() or so.

if c.WebhookEnabled {
hookServer := mgr.GetWebhookServer()
if err := mgr.AddReadyzCheck("webhook", hookServer.StartedChecker()); err != nil {
return errors.Wrap(err, "cannot create webhook ready check")
}
if err := mgr.AddHealthzCheck("webhook", hookServer.StartedChecker()); err != nil {
return errors.Wrap(err, "cannot create webhook health check")
}
}
return nil
}