Implement paginated lists in CLI, APIs, and storage #170
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
cipherboy
force-pushed
the
paginated-lists
branch
from
a45c37d
to
bb02ce1
Compare
cipherboy
force-pushed
the
paginated-lists
branch
2 times, most recently
from
23e5a0a
to
737b14e
Compare
|
This is great! I can attest to this being a highly desired feature. Notably, KV and PKI backend deployments can get large fast, so the ability to page over all data within them is super handy. The fact that this approach is also backwards compatible (doesn't add any output parameters like cursor paging,) is a nice bonus; that means nobody has to update any existing scripts. Passing the current |
cipherboy
force-pushed
the
paginated-lists
branch
from
737b14e
to
4439466
Compare
JanMa
requested changes
Mar 19, 2024
cipherboy
force-pushed
the
paginated-lists
branch
from
4439466
to
7be6c7a
Compare
JanMa
approved these changes
Mar 19, 2024
cipherboy
force-pushed
the
paginated-lists
branch
from
7be6c7a
to
1ffe33b
Compare
This adds the ability to list a specified number of entries, seeking until after after a given reference entry, to the storage backends. With efficient implementations, this should allow reducing the memory overhead of API LIST operations, though at the lack of consistency between fetches. This should allow more expensive tidy operations (such as PKI's) to more correctly release resources between operations and gracefully handle larger numbers of certificates. Additionally, this functionality will be useful for implementing better pagination on list views (again, like PKI's), as finer control over fetched entries from the server can be had, avoiding the need to keep the entire list in the browser's memory. This has broader applicability beyond PKI. Signed-off-by: Alexander Scheel <alexander.m.scheel@gmail.com>
Unlike most other endpoints, List does not take a generic Data parameter. For pagination, we wish to standardize the parameters (generally) and so use the same calling convention (for after and limit) as used in the storage APIs. In the future, if more endpoints offer paginated lists and wish to do so via other filtering mechanisms (or needing multi-part filters for nested references such as with PKI's revocation queue list endpoints), a generic ListWithData helper could be added instead. Signed-off-by: Alexander Scheel <alexander.m.scheel@gmail.com>
This exposes the new pagination filters on the `bao list` command line and expands the existing description text for both pagination and detailed list responses. Signed-off-by: Alexander Scheel <alexander.m.scheel@gmail.com>
This uses ListPage for most PKI list endpoints, with the exception of the complex (two-level: cluster + serial) endpoints for cross-cluster revocation queues. These hopefully drain rather quickly and it would be harder to adapt truncation for two parameters (and the CLI would likely have to allow general K/V parameters to list endpoints which it currently doesn't as one of the two after values would have to be named differently). However, this enables it for leaf and revoked certs (both very large), roles (which, due to the ACL model, can grow large), and the potentially less-large but still useful to have paginated on the upper end issuers and keys listings. Signed-off-by: Alexander Scheel <alexander.m.scheel@gmail.com>
As mentioned in the plugin proposal, external plugins which build against OpenBao's SDK but run against HashiCorp Vault will need a shim module to provide compatibility of the ListPage(...) call if it is used. While the plugin could do this itself (catching the error and then supplementing its own shim), doing this at the GRPC layer allows plugins to mostly ignore the differences between the two server versions. Signed-off-by: Alexander Scheel <alexander.m.scheel@gmail.com>
This upgrades remaining plugin list endpoints to support pagination, maximizing utility of the new CLI & API helpers. Many of these places (such as K/V, Transit keys, or SSH roles) have the potential for large quantities of entries, making these useful in general to have. This also updates the sys/raw interface to support pagination as well, as many plugins can write a large number of storage entires into any given folder. Signed-off-by: Alexander Scheel <alexander.m.scheel@gmail.com>
Signed-off-by: Alexander Scheel <alexander.m.scheel@gmail.com>
Also fix the missing references to the metrics system. Signed-off-by: Alexander Scheel <alexander.m.scheel@gmail.com>
naphelps
force-pushed
the
paginated-lists
branch
from
1ffe33b
to
cf6ce48
Compare
6 tasks
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This implements list paginations using an
after/limitstyle system, discussed in #140.In particular,
LISTAPI operations areGETrequests with a?list=trueparameter to an endpoint. This adds the ability (in the CLI, API, and underlying storage mechanism) to specify two additional attributes, a key,after, to show results whose key is strictly greater than (alphabetically) the specified value, and a number,limit, of results to show.This allows better user interface design: pagination can be done on the server (in addition to the UI), which allows the client to consume less resources (both locally and on the remote server), when lots of entries are present. Endpoints like PKI's revoked or valid certificates, Transit's lists of keys, or entries in K/V v2 are all common endpoints upgraded with pagination that benefit from this improvement.
This is an example of a change made possible by dropping support for alternative storage backends.
Resolves: #140