New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support paged LIST queries from the Kubernetes API #365

Open
smarterclayton opened this Issue Jul 27, 2017 · 24 comments

Comments

Projects
None yet
@smarterclayton
Contributor

smarterclayton commented Jul 27, 2017

Feature Description

  • One-line feature description (can be used as a release note): Allow API consumers - especially those that must retrieve large sets of data - to retrieve results in pages so as to reduce the memory and size impact of those very large queries.
  • Primary contact (assignee): @smarterclayton
  • Responsible SIGs: sig-api-machinery, sig-scalability
  • Design proposal link (community repo): kubernetes/community#896
  • Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred: deads2k lavalamp
  • Approver (likely from SIG/area to which feature belongs): lavalamp
  • Feature target (which target equals to which milestone):
    • Alpha release target (1.8)
    • Beta release target (1.9)
    • Stable release target ()
@dixudx

This comment has been minimized.

Member

dixudx commented Jul 28, 2017

/cc

@idvoretskyi idvoretskyi added this to the 1.8 milestone Aug 3, 2017

@smarterclayton

This comment has been minimized.

Contributor

smarterclayton commented Aug 3, 2017

This work is part of the SLO improvements we have for kubernetes - helping us meet the requirements of much denser clusters. The key goal for alpha is to prove that it will help us reach the next scale limit for Kubernetes, and will take work across several releases to get there.

k8s-merge-robot added a commit to kubernetes/community that referenced this issue Aug 29, 2017

Merge pull request #896 from smarterclayton/list
Automatic merge from submit-queue

Design for consistent API chunking in Kubernetes

In order to reduce the memory allocation impact of very large LIST operations against the Kubernetes apiserver, it should be possible to receive a single large resource list as many individual page requests to the server.

Part of kubernetes/enhancements#365. Taken from kubernetes/kubernetes#49338. Original discussion in kubernetes/kubernetes#2349
@idvoretskyi

This comment has been minimized.

Member

idvoretskyi commented Sep 14, 2017

@smarterclayton @kubernetes/sig-api-machinery-feature-requests @kubernetes/sig-scalability-feature-requests can you confirm that this feature targets 1.8?

If yes, please, update the features tracking spreadsheet with the feature data, otherwise, let's remove this item from 1.8 milestone.

Thanks

@smarterclayton

This comment has been minimized.

Contributor

smarterclayton commented Sep 16, 2017

Yes, it was delivered for 1.8.

@smarterclayton

This comment has been minimized.

Contributor

smarterclayton commented Oct 10, 2017

Beta for 1.9

@smarterclayton smarterclayton modified the milestones: 1.8, 1.9 Oct 10, 2017

@smarterclayton

This comment has been minimized.

Contributor

smarterclayton commented Oct 27, 2017

Goals for 1.9 - expose in CLI for get.go
Goals for 1.10 - expose for all CLI commands, GA

@zacharysarah

This comment has been minimized.

Contributor

zacharysarah commented Nov 22, 2017

@smarterclayton 👋 Please open a documentation PR and add a link to the 1.9 tracking spreadsheet. Thanks in advance!

@zacharysarah

This comment has been minimized.

Contributor

zacharysarah commented Nov 28, 2017

@smarterclayton Bump for docs ☝️

/cc @idvoretskyi

@idvoretskyi

This comment has been minimized.

Member

idvoretskyi commented Nov 30, 2017

@smarterclayton @kubernetes/sig-api-machinery-feature-requests @kubernetes/sig-scalability-feature-requests any updates on the docs status?

A friendly reminder on docs deadline tomorrow.

/cc @zacharysarah

@smarterclayton

This comment has been minimized.

Contributor

smarterclayton commented Dec 2, 2017

@fejta-bot

This comment has been minimized.

fejta-bot commented Mar 2, 2018

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@smarterclayton

This comment has been minimized.

Contributor

smarterclayton commented Mar 8, 2018

This was beta in 1.9, we have not yet moved it to GA. Docs are complete, CLI commands support it.

@idvoretskyi

This comment has been minimized.

Member

idvoretskyi commented Mar 9, 2018

@smarterclayton when you are estimating the GA stage?

@smarterclayton

This comment has been minimized.

Contributor

smarterclayton commented Mar 9, 2018

@justaugustus

This comment has been minimized.

Member

justaugustus commented Apr 17, 2018

@smarterclayton
Any plans for this in 1.11?

If so, can you please ensure the feature is up-to-date with the appropriate:

  • Description
  • Milestone
  • Assignee(s)
  • Labels:
    • stage/{alpha,beta,stable}
    • sig/*
    • kind/feature

cc @idvoretskyi

@smarterclayton

This comment has been minimized.

Contributor

smarterclayton commented Apr 18, 2018

@deads2k and I chatted briefly. I think we need to discuss in api-machinery whether we want to go one step further and expose this for end users (like user interfaces) that want to do human focused paging before we move to stable. So at minimum this is 1.12.

@smarterclayton smarterclayton modified the milestones: v1.10, v1.12 Apr 18, 2018

@justaugustus

This comment has been minimized.

Member

justaugustus commented Apr 20, 2018

Thanks for the update, Clayton!

@justaugustus

This comment has been minimized.

Member

justaugustus commented Jul 18, 2018

@smarterclayton --

It looks like this feature is currently in the Kubernetes 1.12 Milestone.

If that is still accurate, please ensure that this issue is up-to-date with ALL of the following information:

  • One-line feature description (can be used as a release note):
  • Primary contact (assignee):
  • Responsible SIGs:
  • Design proposal link (community repo):
  • Link to e2e and/or unit tests:
  • Reviewer(s) - (for LGTM) recommend having 2+ reviewers (at least one from code-area OWNERS file) agreed to review. Reviewers from multiple companies preferred:
  • Approver (likely from SIG/area to which feature belongs):
  • Feature target (which target equals to which milestone):
    • Alpha release target (x.y)
    • Beta release target (x.y)
    • Stable release target (x.y)

Set the following:

  • Description
  • Assignee(s)
  • Labels:
    • stage/{alpha,beta,stable}
    • sig/*
    • kind/feature

Please note that the Features Freeze is July 31st, after which any incomplete Feature issues will require an Exception request to be accepted into the milestone.

In addition, please be aware of the following relevant deadlines:

  • Docs deadline (open placeholder PRs): 8/21
  • Test case freeze: 8/28

Please make sure all PRs for features have relevant release notes included as well.

Happy shipping!

/cc @justaugustus @kacole2 @robertsandoval @rajendar38

@jwierzbo-intel

This comment has been minimized.

jwierzbo-intel commented Jul 31, 2018

@smarterclayton Any plans to integrate this feature for extension-api-service (https://github.com/kubernetes/apiserver)?

As I see it also doesn't work for CRD objects..

EDIT: I found how to enable paging in extension api-server: flag Paging in k8s.io/apiserver/pkg/storage/storagebackend/config.go has to be set to true

But still don't know how to enable it for CRD models..

@justaugustus

This comment has been minimized.

Member

justaugustus commented Jul 31, 2018

@smarterclayton --
Feature Freeze is today. Are we planning on graduating this feature in Kubernetes 1.12?
If so, can you make sure everything is up-to-date, so I can include it on the 1.12 Feature tracking spreadsheet?

@justaugustus justaugustus removed this from the v1.12 milestone Aug 10, 2018

@justaugustus

This comment has been minimized.

Member

justaugustus commented Aug 10, 2018

Slack update:

deads2k [10:28 AM]
@justaugustus We wanted to solicit more comments first.  We mentioned it in apimachinery here: https://docs.google.com/document/d/1x9RNaaysyO0gXHIr1y50QFbiL1x8OWnk2v3XnrdkT5Y/edit#heading=h.lgl8gmvpt98r and we should nail down conformance before doing so, though the api is backwards compatible

justaugustus [10:33 AM]
@deads2k so hold until 1.13 at least?

deads2k [10:47 AM]
I expect so

Removed from the 1.12 milestone.

@alecmerdler

This comment has been minimized.

alecmerdler commented Aug 24, 2018

Are there any plans to add pagination support to custom resources?

@liggitt

This comment has been minimized.

Member

liggitt commented Aug 25, 2018

Are there any plans to add pagination support to custom resources?

Yes. It was an oversight they weren't enabled already. Opened kubernetes/kubernetes#67861 to enable them

@kacole2

This comment has been minimized.

Contributor

kacole2 commented Oct 8, 2018

Hi
This enhancement has been tracked before, so we'd like to check in and see if there are any plans for this to graduate stages in Kubernetes 1.13. This release is targeted to be more ‘stable’ and will have an aggressive timeline. Please only include this enhancement if there is a high level of confidence it will meet the following deadlines:

  • Docs (open placeholder PRs): 11/8
  • Code Slush: 11/9
  • Code Freeze Begins: 11/15
  • Docs Complete and Reviewed: 11/27

Please take a moment to update the milestones on your original post for future tracking and ping @kacole2 if it needs to be included in the 1.13 Enhancements Tracking Sheet

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment