Security ‐ TLS certificate expiry

Description

This rule checks if all TLS certificates stored in Kubernetes secrets are valid and not expiring soon. It retrieves all secrets of type kubernetes.io/tls across all namespaces and validates their tls.crt data.

The rule warns if any TLS certificate will expire within 14 days and fails if any certificate has already expired.

Prerequisites

• Cluster-level access to list secrets across all namespaces (oc get secret --field-selector=type=kubernetes.io/tls -A).
• openssl and oc should be available.

Impact

Expired TLS certificates in Kubernetes secrets can cause:

• Ingress/Route failures: Routes and Ingress resources using expired TLS secrets will cause browsers to display security warnings or refuse connections entirely.
• Service-to-service communication breakdowns: Internal services relying on TLS secrets for mTLS or encrypted communication will fail to establish connections.
• Webhook failures: Admission webhooks, conversion webhooks, and API aggregation layers using expired certificates will reject API requests, potentially blocking cluster operations.
• Operator degradation: Operators that rely on TLS secrets for secure communication may enter a degraded state.

Root Cause

• Manually created certificates: TLS secrets created with manually generated certificates (e.g., via openssl) do not have automatic rotation and will eventually expire.
• cert-manager misconfiguration: If cert-manager is used for certificate management, misconfigured Certificate resources, expired issuers, or ACME challenge failures can prevent renewal.
• Orphaned secrets: TLS secrets left behind after removing applications or changing configurations may not be maintained.
• External CA expiry: If the certificate authority (CA) used to sign the certificates has expired, all derived certificates become invalid.

Diagnostics

List all TLS secrets and inspect their certificates:

# List all TLS secrets across namespaces
oc get secret --field-selector=type=kubernetes.io/tls -A

# Inspect a specific TLS secret's certificate
# Use -text instead of -enddate for full certificate details
oc get secret <secret-name> -n <namespace> -o jsonpath='{.data.tls\.crt}' | base64 -d | openssl x509 -enddate -noout

If using cert-manager, check certificate status:

oc get certificates -A
oc describe certificate <cert-name> -n <namespace>

Solution

1. Renew manually created certificates by generating a new certificate and updating the secret:

oc create secret tls <secret-name> -n <namespace> \
  --cert=new-tls.crt --key=new-tls.key \
  --dry-run=client -o yaml | oc apply -f -

2. Fix cert-manager renewals if cert-manager is in use:

# Check certificate status
oc get certificates -A

# Trigger manual renewal
oc annotate certificate <cert-name> -n <namespace> cert-manager.io/issueTime="$(date -u +%Y-%m-%dT%H:%M:%SZ)" --overwrite

# Or, use the cmctl CLI:
cmctl renew <cert-name> -n <namespace>

3. Remove orphaned secrets that are no longer referenced by any application or route:

# Identify which resources reference the secret
oc get routes,ingress -A -o json | grep <secret-name>

4. Update routes if a route is using an expired TLS secret, update the route with a renewed certificate:

oc patch route <route-name> -n <namespace> --type=merge -p \
  "{\"spec\":{\"tls\":{\"certificate\":\"$(cat new-tls.crt)\",\"key\":\"$(cat new-tls.key)\"}}}"

Resources

• OpenShift - Secured routes
• cert-manager documentation
• Kubernetes - TLS Secrets