CNTRLPLANE-1576: add event-ttl configuration to kube-apiserver #2520

tjungblu · 2025-10-07T14:56:49Z

This adds a minute/hour configuration duration to configure the event ttl setting in kube-apiserver. Default will stay 3h, as currently defined in KAS-O.

openshift-ci-robot · 2025-10-07T14:56:52Z

@tjungblu: This pull request references CNTRLPLANE-1576 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.21.0" version, but no target version was set.

In response to this:

This adds a minute/hour configuration duration to configure the event ttl setting in kube-apiserver. Default will stay 3h, as currently defined in KAS-O.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-ci · 2025-10-07T14:57:50Z

Hello @tjungblu! Some important instructions when contributing to openshift/api:
API design plays an important part in the user experience of OpenShift and as such API PRs are subject to a high level of scrutiny to ensure they follow our best practices. If you haven't already done so, please review the OpenShift API Conventions and ensure that your proposed changes are compliant. Following these conventions will help expedite the api review process for your PR.

openshift-ci · 2025-10-08T14:08:12Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign joelspeed for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

JoelSpeed · 2025-10-08T14:15:28Z

operator/v1/types_kubeapiserver.go

+	// The value must be parseable as a time duration value;
+	// see <https://pkg.go.dev/time#ParseDuration>.


We prefer not to use duration values anymore. Instead, create a int32 type, with units in the name

For example, this should be eventTTLMinutes.

We do this because not all clients are built in Go, and building a Go compatible duration parsing in other languages is not going to be fun

thanks for reminding me. I've changed this to be configurable in minutes via int32.

JoelSpeed · 2025-10-08T14:18:27Z

operator/v1/types_kubeapiserver.go

+	// If configured, it must be a value of 1m (one minute) or greater, we only allow setting
+	// minute and hour durations (e.g. 5m or 5h).


We should set a sensible maximum value for this TTL. Can you suggest a sensible maximum value for this?

yep, made it 2x the default of 3h now

JoelSpeed · 2025-10-08T14:23:30Z

operator/v1/types_kubeapiserver.go

+	// +kubebuilder:validation:Format=duration
+	// +kubebuilder:validation:Pattern=^(0|([0-9]+(\.[0-9]+)?(m|h))+)$
+	// +kubebuilder:validation:Type:=string
+	// +default="3h"


Since this is a configuration API, we want to reserve the right to change this value over time. This will be easier if we don't set the value through openapi defaulting.

It's better here to set the value in the controller by detecting that it is omitted.

We generally add a comment to the godoc to explain this scenario

// When omitted this means no opinion, and the platform is left to choose a reasonable default, which is subject to change over time. // The current default value is 3h.

added this, great idea

JoelSpeed · 2025-10-08T15:11:46Z

operator/v1/types_kubeapiserver.go

 type KubeAPIServerSpec struct {
 	StaticPodOperatorSpec `json:",inline"`
+
+	// eventTTL specifies the amount of time that the events are stored before being deleted.


Why would a user choose either a larger or smaller value for this? Is there a performance benefit? What is too large? What are the impacts of choosing something very small?

How deep do we want to document this on the godoc vs. the actual openshift documentation?
Or is this just for your personal curiosity?

I think we should give at least some guidance as to what the effect is of choosing a smaller or larger value, doesn't have to be super deep, but if we can make the API self service that's preferable

My main concern is that someone sees they can change it to 1m in the oc explain, so does so, without us giving them any hint as to potential issues this short feedback loop could introduce

This adds a minute based configuration to configure the event ttl setting in kube-apiserver. Default will stay 3h, as currently defined in KAS-O. Signed-off-by: Thomas Jungblut <tjungblu@redhat.com>

benluddy · 2025-10-08T15:39:20Z

operator/v1/types_kubeapiserver.go

+	//
+	// +kubebuilder:default:=180
+	// +kubebuilder:validation:Minimum=30
+	// +kubebuilder:validation:Maximum=360


Do we have users that need to increase the event TTL above 3h? Choosing 3h as the maximum would preserve the upper bound when it comes to benchmarking and publishing supported cluster sizes.

For reference, upstream default TTL is 1h and it is already considering very high for today's cluster sizes.

The only reason we have 3h downstream as the default value is because origin jobs have a 3h timeout and this allows to retain the events for the entire duration of the tests.

JoelSpeed · 2025-10-08T15:45:49Z

operator/v1/types_kubeapiserver.go

 	StaticPodOperatorSpec `json:",inline"`
+
+	// eventTTLMinutes specifies the amount of time that the events are stored before being deleted.
+	// This setting is allowed between 30 minutes minimum up to 6h (360 minutes).


Because the field value is an integer, it would be better to reverse these

Suggested change

// This setting is allowed between 30 minutes minimum up to 6h (360 minutes).

// The TTL is allowed between 30 minutes minimum up to a maximum of 360 minutes (6 hours).

JoelSpeed · 2025-10-08T15:46:06Z

operator/v1/types_kubeapiserver.go

+	// When omitted this means no opinion, and the platform is left to choose a reasonable default, which is subject to change over time.
+	// The current default value is 3h (180 minutes).
+	//
+	// +kubebuilder:default:=180


I left a comment on the old default, we shouldn't have the openapi default like this

JoelSpeed · 2025-10-08T15:46:37Z

Is there an EP for this new field? It should be added behind a feature gate

dgrisonnet · 2025-10-08T15:48:13Z

operator/v1/types_kubeapiserver.go

+	// The current default value is 3h (180 minutes).
+	//
+	// +kubebuilder:default:=180
+	// +kubebuilder:validation:Minimum=30


I think we can go as far down as 5 minutes. This would allow customers that persists events externally to reduce the amount of events stored in their cluster as much as possible while still being able to know what is happening right now in their cluster.

Also, when there is an event spew, the only way to recover the cluster quickly is to reduce the event-ttl as much as possible. For that some people set the ttl to 0 in vanilla Kubernetes, but I think that for debug purposes, it would still be useful to have some events around, so 5 minutes is better IMO.

openshift-ci · 2025-10-08T18:02:58Z

@tjungblu: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/okd-scos-e2e-aws-ovn	`6054d58`	link	false	`/test okd-scos-e2e-aws-ovn`
ci/prow/integration	`6054d58`	link	true	`/test integration`

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Oct 7, 2025

openshift-ci bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Oct 7, 2025

openshift-ci bot requested review from everettraven and JoelSpeed October 7, 2025 15:01

tjungblu mentioned this pull request Oct 8, 2025

CNTRLPLANE-1575: Add support for event-ttl in Kube API Server Operator openshift/enhancements#1857

Open

tjungblu force-pushed the CNTRLPLANE-1576 branch from b24d8da to e816e63 Compare October 8, 2025 14:07

JoelSpeed reviewed Oct 8, 2025

View reviewed changes

CNTRLPLANE-1576: add event-ttl configuration to kube-apiserver

6054d58

This adds a minute based configuration to configure the event ttl setting in kube-apiserver. Default will stay 3h, as currently defined in KAS-O. Signed-off-by: Thomas Jungblut <tjungblu@redhat.com>

tjungblu force-pushed the CNTRLPLANE-1576 branch from e816e63 to 6054d58 Compare October 8, 2025 15:22

benluddy reviewed Oct 8, 2025

View reviewed changes

JoelSpeed reviewed Oct 8, 2025

View reviewed changes

dgrisonnet reviewed Oct 8, 2025

View reviewed changes

		// The value must be parseable as a time duration value;
		// see <https://pkg.go.dev/time#ParseDuration>.

		// If configured, it must be a value of 1m (one minute) or greater, we only allow setting
		// minute and hour durations (e.g. 5m or 5h).

	// This setting is allowed between 30 minutes minimum up to 6h (360 minutes).
	// The TTL is allowed between 30 minutes minimum up to a maximum of 360 minutes (6 hours).

CNTRLPLANE-1576: add event-ttl configuration to kube-apiserver #2520

Are you sure you want to change the base?

CNTRLPLANE-1576: add event-ttl configuration to kube-apiserver #2520

Uh oh!

Conversation

tjungblu commented Oct 7, 2025

Uh oh!

openshift-ci-robot commented Oct 7, 2025 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

openshift-ci bot commented Oct 7, 2025

Uh oh!

openshift-ci bot commented Oct 8, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

JoelSpeed commented Oct 8, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

openshift-ci bot commented Oct 8, 2025

Uh oh!

Uh oh!

openshift-ci-robot commented Oct 7, 2025 •

edited by openshift-ci bot

Loading