fix: send initial-events-end bookmark for CachedResource virtual storage by maxpain · Pull Request #3875 · kcp-dev/kcp

maxpain · 2026-03-02T14:29:41Z

Summary

The unwrappingWatch in the CachedResource virtual storage never sends a BOOKMARK event with the k8s.io/initial-events-end annotation after replaying the initial list of cached objects. This causes clients using the WatchList feature (sendInitialEvents=true), which is the default since client-go v0.35.1 (Kubernetes 1.35), to log event bookmark expired warnings every 10 seconds.

Fixes #3871

Changes

After registering the informer event handler, start a goroutine that waits for handler.HasSynced() (all initial items delivered), then sends a BOOKMARK with the initial-events-end annotation and the latest resource version seen during the initial list replay.

Track the maximum resourceVersion via atomic.Value during initial event replay
After WaitForCacheSync, send a BOOKMARK with k8s.io/initial-events-end: "true" annotation
Only activate when sendInitialEvents=true && allowWatchBookmarks=true

How to reproduce

Enable CachedAPIs feature gate
Create a CachedResource with identity secret
Configure an APIExport with storage: virtual referencing the CachedResourceEndpointSlice
Bind the API in a tenant workspace
Watch the resource using a client with WatchList enabled (default in client-go >= 0.35.1):
```
kubectl get <resource> -w
```
Observe event bookmark expired warnings in client logs every 10 seconds

Send initial-events-end bookmark for CachedResource virtual storage

The unwrappingWatch in the replication virtual workspace never sent a BOOKMARK event with the k8s.io/initial-events-end annotation after replaying the initial list of cached objects. This caused clients using the WatchList feature (sendInitialEvents=true), which is enabled by default since client-go v0.35.1 (Kubernetes 1.35), to log "event bookmark expired" warnings every 10 seconds. After registering the informer event handler, start a goroutine that waits for handler.HasSynced() (all initial items delivered), then sends a BOOKMARK with the initial-events-end annotation and the latest resource version seen during the initial list. Signed-off-by: Max Makarov <maxpain@linux.com>

kcp-ci-bot · 2026-03-02T14:29:53Z

Hi @maxpain. Thanks for your PR.

I'm waiting for a kcp-dev member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

ntnn · 2026-03-02T14:37:27Z

/ok-to-test
/assign @gman0

Thanks for the PR!

gman0 · 2026-03-02T19:49:47Z

/kind bug

gman0 · 2026-03-02T19:49:55Z

/retest

gman0 · 2026-03-02T20:17:16Z

/retest

+ ./hack/run-with-prometheus.sh make test-e2e
Downloading Prometheus v2.42.0
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
100 55124  100 55124    0     0   285k      0 --:--:-- --:--:-- --:--:--  286k

gzip: stdin: not in gzip format
tar: Child returned status 1
tar: Error is not recoverable: exiting now

gman0 · 2026-03-02T21:27:28Z

/lgtm

Thanks!

ntnn · 2026-03-02T21:40:51Z

pkg/virtual/replication/builder/unwrap.go

+	// existing items. Clients using WatchList (sendInitialEvents=true) expect a
+	// BOOKMARK with the k8s.io/initial-events-end annotation to signal that the
+	// initial state has been fully delivered.
+	if innerListOpts != nil && innerListOpts.SendInitialEvents != nil && *innerListOpts.SendInitialEvents && innerListOpts.AllowWatchBookmarks {


Suggested change

if innerListOpts != nil && innerListOpts.SendInitialEvents != nil && *innerListOpts.SendInitialEvents && innerListOpts.AllowWatchBookmarks {

if ptr.Deref(innerListOpts.SendInitialEvents, false) && ptr.Deref(&innerListOpts.AllowWatchBookmarks, fasle) {

Just to make it read nicer. Otherwise lgtm, thanks for the PR!

please add to follow-up por. will tag to get this ball rolling

@mjudeikis-bot Create a follow-up PR implementing this change.

@mjudeikis-bot do the needed ^ create a PR for this please

mjudeikis · 2026-03-03T06:34:49Z

/lgtm
/approve

kcp-ci-bot · 2026-03-03T06:34:55Z

LGTM label has been added.

Details

Git tree hash: 020c525b6bee5175ffce11561c76d61bf98a4174

kcp-ci-bot · 2026-03-03T06:34:56Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: mjudeikis

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [mjudeikis]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

cnvergence · 2026-03-03T08:53:40Z

@maxpain can you add release note please? :)

ntnn · 2026-03-03T09:01:57Z

/release-note Send initial-events-end bookmark for CachedResource virtual storage

…Watch Simplify the SendInitialEvents nil-check using ptr.Deref from k8s.io/utils/ptr, which handles the nil pointer dereference idiom more readably. Suggested-by: ntnn Follows-up: kcp-dev#3875 Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

mjudeikis-bot · 2026-03-03T09:06:43Z

Done — opened #3879 with the ptr.Deref refactor (as requested by @ntnn and @mjudeikis). Kept the innerListOpts != nil guard since ptr.Deref doesn't protect against a nil receiver struct.

ntnn · 2026-03-03T09:10:55Z

time="2026-03-03T09:03:35Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://golang:1.24.13: pinging container registry registry-mirror.registry.svc.cluster.local:5001: Get "http://registry-mirror.registry.svc.cluster.local:5001/v2/\": dial tcp 10.96.195.28:5001: connect: connection refused"

uh oh

mjudeikis · 2026-03-03T09:11:37Z

/retest

maxpain · 2026-03-03T09:19:23Z

Thanks everyone for the reviews and the release note! Is there anything else needed from my side, or just waiting for CI to pass?

ntnn · 2026-03-03T09:20:40Z

Just waiting for CI to pass, the docker registry pod seems to have problems but that should resolve itself soon^(tm), otherwise someone will have look^^

ntnn · 2026-03-03T10:52:43Z

/retest

…Watch Simplify the SendInitialEvents nil-check using ptr.Deref from k8s.io/utils/ptr, which handles the nil pointer dereference idiom more readably. Suggested-by: ntnn Follows-up: kcp-dev#3875 Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

…Watch Simplify the SendInitialEvents nil-check using ptr.Deref from k8s.io/utils/ptr, which handles the nil pointer dereference idiom more readably. Suggested-by: ntnn Follows-up: kcp-dev#3875 Co-authored-by: Mangirdas Judeikis <mangirdas@judeikis.lt> Signed-off-by: mjudeikis-bot <mjudeikis-bot@faros.sh> Signed-off-by: Mangirdas Judeikis <mangirdas@judeikis.lt>

kcp-ci-bot added do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. do-not-merge/needs-kind Indicates a PR lacks a `kind/foo` label and requires one. labels Mar 2, 2026

kcp-ci-bot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. dco-signoff: no Indicates the PR's author has not signed the DCO. needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Mar 2, 2026

maxpain force-pushed the fix/cached-resource-initial-events-bookmark branch from ae60692 to 738d519 Compare March 2, 2026 14:30

kcp-ci-bot added dco-signoff: yes Indicates the PR's author has signed the DCO. and removed dco-signoff: no Indicates the PR's author has not signed the DCO. labels Mar 2, 2026

kcp-ci-bot assigned gman0 Mar 2, 2026

kcp-ci-bot added ok-to-test Indicates a non-member PR verified by an org member that is safe to test. and removed needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. labels Mar 2, 2026

kcp-ci-bot added kind/bug Categorizes issue or PR as related to a bug. and removed do-not-merge/needs-kind Indicates a PR lacks a `kind/foo` label and requires one. labels Mar 2, 2026

ntnn reviewed Mar 2, 2026

View reviewed changes

kcp-ci-bot assigned mjudeikis Mar 3, 2026

kcp-ci-bot added the lgtm Indicates that a PR is ready to be merged. label Mar 3, 2026

kcp-ci-bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 3, 2026

kcp-ci-bot added the release-note Denotes a PR that will be considered when it comes time to generate release notes. label Mar 3, 2026

kcp-ci-bot removed the do-not-merge/release-note-label-needed Indicates that a PR should not merge because it's missing one of the release note labels. label Mar 3, 2026

mjudeikis-bot mentioned this pull request Mar 3, 2026

refactor: use ptr.Deref for SendInitialEvents nil check in unwrappingWatch #3879

Merged

kcp-ci-bot merged commit 32e8b9b into kcp-dev:main Mar 3, 2026
13 checks passed

maxpain deleted the fix/cached-resource-initial-events-bookmark branch March 3, 2026 11:06

	if innerListOpts != nil && innerListOpts.SendInitialEvents != nil && *innerListOpts.SendInitialEvents && innerListOpts.AllowWatchBookmarks {
	if ptr.Deref(innerListOpts.SendInitialEvents, false) && ptr.Deref(&innerListOpts.AllowWatchBookmarks, fasle) {

Conversation

maxpain commented Mar 2, 2026 • edited by ntnn Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

How to reproduce

Uh oh!

kcp-ci-bot commented Mar 2, 2026

Uh oh!

ntnn commented Mar 2, 2026

Uh oh!

gman0 commented Mar 2, 2026

Uh oh!

gman0 commented Mar 2, 2026

Uh oh!

gman0 commented Mar 2, 2026

Uh oh!

gman0 commented Mar 2, 2026

Uh oh!

ntnn Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

ntnn Mar 2, 2026

Choose a reason for hiding this comment

Uh oh!

mjudeikis Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

ntnn Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

mjudeikis Mar 3, 2026

Choose a reason for hiding this comment

Uh oh!

mjudeikis commented Mar 3, 2026

Uh oh!

kcp-ci-bot commented Mar 3, 2026

Uh oh!

kcp-ci-bot commented Mar 3, 2026

Uh oh!

cnvergence commented Mar 3, 2026

Uh oh!

ntnn commented Mar 3, 2026

Uh oh!

mjudeikis-bot commented Mar 3, 2026

Uh oh!

ntnn commented Mar 3, 2026

Uh oh!

mjudeikis commented Mar 3, 2026

Uh oh!

maxpain commented Mar 3, 2026

Uh oh!

ntnn commented Mar 3, 2026

Uh oh!

ntnn commented Mar 3, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

7 participants

maxpain commented Mar 2, 2026 •

edited by ntnn

Loading