Implement general-purpose pagination functionality in KV API

[josiahhaswell]

Key hierarchies can grow to be quite large,and this introduces unnecessary and undesirable latencies during retrieval. It would be very nice if the KV API supported pagination. This could be done in a backwards-compatible fashion via query params (possibly something along the lines of)

/v1/kv/sample-key?start=0&end=10

Ordering would also generally be useful; adding

/v1/kv/sample-key?start=0&end=10&order=desc|asc

would sort the results by the lexicographical ordering of the keys under sample-key.

This could also be integrated consistently with recurse by performing a depth-first traversal of the keys in range [start, end] under the provided sort criterion and returning that set.

Thanks!
Josiah

[MatthiasGrandl]

Any news on this? It seems incredible that such a feature does not exist while filtering does.

[vnktram]

Can I pick this one?

[jkirschner-hashicorp]

Hi @vnktram,

Thanks for volunteering to contribute to Consul! Give me a bit of time to seek the Consul team's perspective on this first - I'll update you next week.

[dnephin]

Thank you for your interest in implementing this feature! I think before we review any implementation it would be good to outline how the problem will be solved. Implementing pagination in Consul is more difficult than it might be in other systems.

One of the challenges is that Consul does not use sequential IDs. The KV entries are indexed by the key name. The keys could change between requests, which could lead to duplicate or missing keys in the response. One option for solving this problem might be to use the CreateIndex as the ID when querying with pagination. That should prevent updates to existing keys from changing the results.

Another challenge is how to query the state store. We currently do not index by CreateIndex, and we do not use range queries (as far as I know). So we would need to add an index to the state store. We would also need to use https://pkg.go.dev/github.com/hashicorp/go-memdb#Txn.LowerBound to query the appropriate range of values (instead of Txn.Get).

If you are interested in picking up this issue, we'd encourage you to first write up a design for the solution (either as a comment in this issue, or as a new issue). The design should cover how the above two challenges will be addressed, and also detail how the new HTTP API query parameters will work, and if there is any interaction with existing API query parameters. That write up will make it much easier for us review any pull requests that implement this feature.

[vnktram]

@dnephin Ack. Let me document the proposed design with your suggestions in mind and share it shortly.

[mstrYoda]

Hi all, what is the current status of this issue?

@vnktram @dnephin ?