Consul KV api enhancements

[CJLove]

Feature Description

Extend the Consul KV rest api with additional blocking query options:

$ curl \
    https://consul.rocks/v1/kv/my-key?meta=true&filter=true&index=X

The meta=true option would result in returning keys and metadata but no values
The filter=true would only include keys whose modify indices were > X

$ curl \
    https://consul.rocks/v1/kv/my-key?recurse=true&filter=true&index=X

The filter=true option would only return key/value/metadata for keys whose modify indices were > X

Use Case(s)

There are two use cases of interest:

1. Support clients doing blocking query of a key prefix in order to detect keys being added/removed. Currently the blocking query returns the complete set of keys, so the client needs to maintain state and do set difference operations in order to detect added/removed keys.

2. Support clients doing blocking query of a key prefix in order to detect values being modified. Right now this has to be done via a blocking query returning values for all keys.
   As suggested in this thread (https://groups.google.com/forum/#!msg/consul-tool/Axow65BtSuY/TCLD5AFmCgAJ;context-place=forum/consul-tool) it would be great if there was an option for Consul to filter and return just the keys with a X-Consul-Index > the index specified in the query

[pierresouchay]

@vaLski That's actually a good idea, but instead of adding ?meta=true, I would rather add HTTP HEAD method (that would return meta data, but no body)

For the filter=true, If I understand correctly filter=true is only when using recurse=true or keys=true query parameter right?

I am not a big fan of the name however, I'll try to find something more descriptive, for instance: discard-old-keys=true for instance

[CJLove]

@pierresouchay Correct. The filter=true and index=X would be used with recurse=true or keys=true query parameter. I'm entirely open to whatever you like from a naming perspective.

[lokeshp11]

Is this implemented yet?
I am looking for an option to retrieve metadata especially ModifyIndex of my key-prefix so that based on updates only I will fetch respective tree instead of pulling every time.

Please suggest if this can be done ?

[pierresouchay]

@CJLove don't think so, on my side, I am doing this with watch on a full tree.

It works well enough even with many writes /sec on large clusters.

The only real issue is whether you have having many many large clusters with large values written very often, but for most usages, watching the whole KV works well enough

Here is how we do this in Consul-Jim's template of consul-templaterb: https://github.com/criteo/consul-templaterb/blob/master/samples/consul-ui/consul_keys.json.erb

On our side it works very well with thousands of keys on very large clusterd (but reasonable sizes and not that many writes/s)