Watch directory for changes of children

[mren]

Querying a dictionary returns a list of the children of this directory.
If I watch the directory with wait=true I wont receive an answer if I add a new children.

As the normal query returns the children I would expect the wait to have the same behavior.

In #603 there was a similar discussion.

$ curl -L '33.33.33.11:4001/v2/keys/directory' -XPUT -d dir=true
{"action":"set","node":{"key":"/directory","dir":true,"modifiedIndex":48038,"createdIndex":48038}}
$ curl -L '33.33.33.11:4001/v2/keys/directory/key' -XPUT -d value=value
{"action":"set","node":{"key":"/directory/key","value":"value","modifiedIndex":48110,"createdIndex":48110}}
$ curl -L '33.33.33.11:4001/v2/keys/directory?wait=true&waitIndex=48039'
# wont return

Are there any plans on watching for changes of the children?

[metaflow]

yes, you can specify recursive=true like this
curl -L http://127.0.0.1:4001/v2/keys/dir?wait=true&recursive=true
to wait for a new child. Please refer to api doc.

[mren]

Hey @metaflow, thanks for pointing out to recursive=true.
recursive=true has a different behavior, it will wait for all changes within the children and sub-children.
This would return more information than the normal result of the directory

I'm basically looking for the future dataset of the directory.

If I do curl -L http://127.0.0.1:4001/v2/keys/dir (without recursive=true) I get the list of children. This makes sense, as a directory does not contain any other data than the information about its children. If I add a new child to the directory the result of the curl command will change. Thus I would expect to get notified if I waited for changes on the directory without using recursive=true

[xiang90]

@mren You want add a level option? Like for only children, you will set recursive=true and level = 1?

[joshuaconner]

Since directories don't have a value per se, would it make sense for wait=true to return when the directory's immediate children change? If not, when should it return?

[chairfield]

I second the request to be able to watch for changes to direct descendants. If a level option makes the most sense architecturally, that would work just fine.

[fatmcgav]

I third the request to be able to watch a dir for changes to direct descendants...

However it sounds like adding a 'level' option might make more sense, as it could be used for both dirs and keys...

[kelseyhightower]

We have a ton of issues related to how watch should work, mainly around watching directories. I'm going to press for a roadmap doc on how we want watches to work in general and address these issues to align with that vision.

[drnic]

Any new discussions internally/escalation of road map for watching for new directories (and not all the other changes in a tree)?

[xiang90]

@drnic No. Sorry. We are working on v3. But for v3, etcd will only support flat key-value space and support watch on prefix or a key.

[drnic]

A WIP script that seems to do the trick:

#!/bin/bash


export ETCD_CLUSTER=${ETCD_CLUSTER:-10.244.4.2:4001}
event=$(curl -sL "http://${ETCD_CLUSTER}/v2/keys/service/?wait=true&recursive=true")

while :
do
  eventIndex=$(echo $event | jq -r ".node.modifiedIndex")
  nextEventIndex=$((eventIndex+1))
  if [[ "$(echo $event | jq -r .action)" == "create" ]]; then
    echo $event
  fi
  event=$(curl -sL "http://${ETCD_CLUSTER}/v2/keys/service/?wait=true&recursive=true&waitIndex=${nextEventIndex}")
done

[drnic]

@xiang90 what do you mean by "flat key-value space" relative to what we have now? happy to read a design doc/blog if that answers it (sorry to not know)

[xiang90]

@drnic https://github.com/coreos/etcd/blob/master/Documentation/rfc/v3api.md. Feedback is welcomed.

[drnic]

I guess first thing that pops to mind is "HTTP API supports a subset of APIs." - what won't be in HTTP API?

Sorry don't mean to borrow this Issue.

[xiang90]

@drnic Not sure yet. We will try best to support all.

[xiang90]

We are not going to implement this for v2. As already mentioned in this issue, v3 is a flag key-value space. So the children is irrelevant.