Add ListPager.EachListItem util #75849

jpbetz · 2019-03-28T23:27:40Z

Introduce a ListPager.EachListItem convenience utility for incrementally processing chunked List results. This makes it easy for a client to only request list chunks from the apiserver that it can actually keep up with processing. EachListItem buffers up to PageBufferSize (default: 10) pages in the background to minimize foreground wait time.

Example usage:

podPager = pager.New(pager.SimplePageFunc(func(opts metav1.ListOptions) (runtime.Object, error) {                                                                                                                 
  return client.CoreV1().Pods().List(opts)                                                                                                                                                   
}))
podPager.EachListItem(ctx, metav1.ListOptions{}, func(obj runtime.Object) error {                                                                                                       
  if pod, ok := obj.(*v1.Pod); ok {
    doSomethingWithThePod(pod)
  } else { ...}
  ...
}

There are some warts with this approach that I'd like to find a way of smoothing over, namely: (1) having to creating the pager (2) having to cast each item to the correct type. But that should be possible to add via code generators in the future.

The name and function signature were picked to be consistent with the existing meta.EachListItem function.

Add ListPager.EachListItem utility function to client-go to enable incremental processing of chunked list responses

/kind feature
/priority important-longterm
/sig api-machinery

jpbetz · 2019-03-28T23:30:13Z

@smarterclayton if you get a minute would you glance at the ListPager.EachListItem function I'm proposing here?

jpbetz · 2019-04-01T23:04:01Z

/cc @jingyih

k8s-ci-robot · 2019-04-01T23:04:02Z

@jpbetz: GitHub didn't allow me to request PR reviews from the following users: jingyih.

Note that only kubernetes members and repo collaborators can review this PR, and authors cannot review their own PRs.

In response to this:

/cc @jingyih

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

jpbetz · 2019-04-02T17:41:10Z

/retest

jpbetz · 2019-04-02T18:25:30Z

/retest

jpbetz · 2019-04-02T20:46:50Z

/retest

jpbetz · 2019-04-02T21:55:16Z

/cc @smarterclayton @jingyih @wojtek-t Looking for reviewers for this, would any of you have time to give it a look?

jpbetz · 2019-04-04T20:04:05Z

Friendly nudge for review

jingyih

I think I generally understand the approach. Added a few comments. Thanks!

jingyih · 2019-04-05T00:51:01Z

staging/src/k8s.io/client-go/tools/pager/pager.go

@@ -115,3 +127,86 @@ func (p *ListPager) List(ctx context.Context, options metav1.ListOptions) (runti
 		options.Continue = m.GetContinue()
 	}
 }
+
+// EachListItem invokes fn on each runtime.Object in the list. Any error immediately terminates the


It might be helpful to add more details here.

Function description says it invokes fn on each object in the list. In function signature, there is no list. So it has to be generated (retrieved) internally in this function.

The listing inside this function is similar but not identical to (*ListPager) List(). In the sense that it does not fall back to full list on resource expire error. But I feel user may assume inside this function, listpager is using the same mechanism to list? You already mentioned a "Expired" error may be returned. I just feel a little more clarification on the internal listing mechanism might be helpful.

Thanks, I've rewritten this with more detail.

jingyih · 2019-04-05T01:07:19Z

staging/src/k8s.io/client-go/tools/pager/pager.go

+			return nil
+		})
+		if err == stoppedErr {
+			err = nil // stoppedErr is an internal signal


I understand what this line does, returning due to been signaled to stop is not actually an error.

Do we need to define an internal error? Can we use context cancel function to signal eachListChunk() to stop? Upon closing stopC, fn can return nil (line 202), but the loop inside eachListChunk() will be stopped by ctx.Done().

You're right. A nested context works here. Since the context error should be return iff the caller cancels the context, reasoning through when/how cancel error get's handled is a bit subtle and I've added a comment explaining it. I've also added test coverage for cancelation, both for then a fn error results in an early exit and when the caller calls cancel.

jingyih · 2019-04-05T01:52:06Z

staging/src/k8s.io/client-go/tools/pager/pager.go

+		bgResultC <- err
+	}()
+
+	for o := range chunkC {


If listing chunk in the background fails (such as due to resource expired), we will still finish processing the existing chunks in the buffer before return?

Yes, I think from an API contract perspective, when a error occurs listing chunks, i think it's valid to either (1) stop calling fn as soon as possible, or (2) call fn on as many items as have been successfully retrieved. I went with #1 since it was trivial to implement. I imagine in different situations a client might benefit more from one of these and in other situations the other. Thoughts?

liggitt · 2019-04-05T03:21:25Z

staging/src/k8s.io/client-go/tools/pager/pager.go

+
+	stopC := make(chan struct{})
+	bgResultC := make(chan error, 1)
+	go func() {


defer runtime.HandleCrash() at the top of the goroutine?

Good catch, I should do something with panics. I've gone with runtime.RecoverFromPanic here. Hopefully it's appropriate.

runtime.RecoverFromPanic didn't work as I had expected (and from a quick grep, doesn't look like we use it in the k8s codebase), so I did a more direct recover.

Why you can't use runtime.HandleCrash() - it's pretty widely used in the codebase and allows to use additional custom handler too.

I had initially misunderstood it to catch the panic and swallow it. But it does still crash. Updating the PR to use it now.

liggitt · 2019-04-05T03:29:34Z

staging/src/k8s.io/client-go/tools/pager/pager.go

+// PageBufferSize chunks of list items concurrently in the background. If the chunk attempt fails a "Expired"
+// error may be returned.
+func (p *ListPager) eachListChunkBuffered(ctx context.Context, options metav1.ListOptions, fn func(obj runtime.Object) error) error {
+	chunkC := make(chan runtime.Object, p.PageBufferSize)


error if p.PageBufferSize <= 0?

Yes, set it to < since 0 size buffer is supported (tests demo this).

liggitt · 2019-04-05T03:31:50Z

staging/src/k8s.io/client-go/tools/pager/pager.go

+
+// eachListChunkBuffered invokes fn on each runtimeObject list chunk. It buffers up to
+// PageBufferSize chunks of list items concurrently in the background. If the chunk attempt fails a "Expired"
+// error may be returned.


need to document the return value ... I think if fn returns an error, processing stops and that error is returned. if fn does not return an error, any error encountered while fetching the list (including timeout or cancelled errors from the context) is returned

Thanks, yes, I've rewritten this with full details on errors.

jpbetz · 2019-04-09T16:49:49Z

@jingyih @liggitt Feedback applied. PTAL!

liggitt · 2019-04-11T02:10:48Z

/lgtm
/approve

k8s-ci-robot · 2019-04-11T02:11:21Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jpbetz, liggitt

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~staging/src/k8s.io/client-go/OWNERS~~ [liggitt]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

tedyu · 2019-04-11T15:33:59Z

staging/src/k8s.io/client-go/tools/pager/pager.go

@@ -48,6 +50,9 @@ type ListPager struct {
 	PageFn   ListPageFunc

 	FullListIfExpired bool
+
+	// Number of pages to buffer
+	PageBufferSize int32


I wonder if the buffer should be defined by the memory consumption instead of number of pages.

We can evaluate the size of a few chunks to dynamically determine the number of chunks to buffer, based on desired buffer size (in terms of memory).

Interesting. Maybe wait and see how this approach works out and then optimize as needed from there? We previously had quite a bit of code doing full lists, and so as we transition to paginated lists and this sort of incremental processing my expectation is we'll reduce memory usage, particularly for object kinds that have large counts. If we still hit scalability/performance limits once this is in use, that would seem like a good time to look into optimizing this further .

Sounds good.

k8s-ci-robot requested review from gmarek and liggitt March 28, 2019 23:28

jpbetz changed the title ~~Paginate node List in PodGC controller, add pager.EachListItem util~~ Paginate node List in PodGC controller, add ListPager.EachListItem util Mar 28, 2019

jpbetz changed the title ~~Paginate node List in PodGC controller, add ListPager.EachListItem util~~ [WIP] Paginate node List in PodGC controller, add ListPager.EachListItem util Mar 29, 2019

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Mar 29, 2019

jpbetz changed the title ~~[WIP] Paginate node List in PodGC controller, add ListPager.EachListItem util~~ Add ListPager.EachListItem util Apr 1, 2019

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Apr 1, 2019

jpbetz force-pushed the pagination-podgc branch from 3cdd565 to 5de2b33 Compare April 1, 2019 22:58

jpbetz force-pushed the pagination-podgc branch 2 times, most recently from 295fa6a to 6708a75 Compare April 2, 2019 17:07

jpbetz force-pushed the pagination-podgc branch from 6708a75 to bb6a508 Compare April 2, 2019 19:31

jpbetz mentioned this pull request Apr 2, 2019

nodelifecycle controller: Paginate pod List call in MarkAllPodsNotReady using ListPager.EachListItem #76055

Closed

jingyih reviewed Apr 5, 2019

View reviewed changes

liggitt reviewed Apr 5, 2019

View reviewed changes

jpbetz force-pushed the pagination-podgc branch from bb6a508 to b351760 Compare April 5, 2019 21:53

jpbetz force-pushed the pagination-podgc branch 2 times, most recently from 5079ed7 to 0c80995 Compare April 5, 2019 23:37

Add pager.EachPageItem utility function to incrementally process lists

6a64ee6

jpbetz force-pushed the pagination-podgc branch from 0c80995 to 6a64ee6 Compare April 10, 2019 22:04

k8s-ci-robot assigned liggitt Apr 11, 2019

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Apr 11, 2019

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Apr 11, 2019

k8s-ci-robot merged commit d5dbc00 into kubernetes:master Apr 11, 2019

tedyu reviewed Apr 11, 2019

View reviewed changes

pacoxu mentioned this pull request Feb 24, 2021

APIListChunking - feature gate to GA? #96497

Closed

brandond mentioned this pull request Nov 30, 2021

Secrets-encryption rotation k3s-io/k3s#4372

Merged

Abirdcfly mentioned this pull request Jul 19, 2022

make TestListPager_EachListItem rework #111241

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ListPager.EachListItem util #75849

Add ListPager.EachListItem util #75849

jpbetz commented Mar 28, 2019 •

edited

jpbetz commented Mar 28, 2019

jpbetz commented Apr 1, 2019

k8s-ci-robot commented Apr 1, 2019

jpbetz commented Apr 2, 2019

jpbetz commented Apr 2, 2019

jpbetz commented Apr 2, 2019

jpbetz commented Apr 2, 2019

jpbetz commented Apr 4, 2019

jingyih left a comment

jingyih Apr 5, 2019

jpbetz Apr 5, 2019

jingyih Apr 5, 2019

jpbetz Apr 5, 2019

jingyih Apr 5, 2019

jpbetz Apr 5, 2019

liggitt Apr 5, 2019

jpbetz Apr 5, 2019

jpbetz Apr 5, 2019

wojtek-t Apr 10, 2019

jpbetz Apr 10, 2019

liggitt Apr 5, 2019

jpbetz Apr 5, 2019

liggitt Apr 5, 2019

jpbetz Apr 5, 2019 •

edited

jpbetz commented Apr 9, 2019

liggitt commented Apr 11, 2019

k8s-ci-robot commented Apr 11, 2019

tedyu Apr 11, 2019 •

edited

jpbetz Apr 11, 2019

tedyu Apr 11, 2019

Add ListPager.EachListItem util #75849

Add ListPager.EachListItem util #75849

Conversation

jpbetz commented Mar 28, 2019 • edited

jpbetz commented Mar 28, 2019

jpbetz commented Apr 1, 2019

k8s-ci-robot commented Apr 1, 2019

jpbetz commented Apr 2, 2019

jpbetz commented Apr 2, 2019

jpbetz commented Apr 2, 2019

jpbetz commented Apr 2, 2019

jpbetz commented Apr 4, 2019

jingyih left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jpbetz Apr 5, 2019 • edited

Choose a reason for hiding this comment

jpbetz commented Apr 9, 2019

liggitt commented Apr 11, 2019

k8s-ci-robot commented Apr 11, 2019

tedyu Apr 11, 2019 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jpbetz commented Mar 28, 2019 •

edited

jpbetz Apr 5, 2019 •

edited

tedyu Apr 11, 2019 •

edited