[feat] Watches/events on the member API should be possible

[purpleidea]

I'd like to take a watch on the membership list, and on leader changes. My understanding was this used to be possible in v2 because /v2/members/ or /leader/ could be used as the watch directory key. With v3 this doesn't seem to be possible.

The bug prefix of [regression] isn't meant in a negative way, but please feel free to change it to [feat] if that's more appropriate going forwards.

On irc @heyitsanthony agreed I should open the issue, and that there was a WIP idea on how to do this in v3, and gave this link: #4979 (comment)

I'm using the golang API.

Thanks in advance!

[heyitsanthony]

@purpleidea I don't believe etcd2 ever supported watching on the member list. This would be an enhancement AFAIK /cc @xiang90

[purpleidea]

@heyitsanthony Perhaps this is true! In any case, I think it would be valuable to have this feature. Renaming...

[purpleidea]

FWIW, I can emulate the behaviour I want by manually keeping a membership list and ensuring to update it everytime I make a call to the members API, however I do not know of a workaround to get an event when the leader changes. I think this would be much more important.

[xiang90]

@heyitsanthony This is pretty easy to do actually. We have the idea that writing a key into the key space when there is a membership change or a leader election. @philips suggests to keep this out of etcd from the beginning to keep key space clean by default. We can propagate the change to key space to allow watching without adding any APIs.

[heyitsanthony]

@xiang90 writing into the key space seems hacky. Other subsystems would benefit from looking like a key space (cf. all the List() rpcs), but there are cases where writing into the itself key space is a deal breaker (e.g., leases). If membership keys are written to the regular key space, then the keys are subject to client put/delete (or would need a layer to stop it or an auth rule or whatever), which also seems gross to me. Finally, it's baking API details into the key space instead of the grpc protocol where it probably belongs.

I think it's better design to make resources look like keys (via ASIDs) instead of exporting them as keys managed via mvcc. It's a little more effort up front but it's got better legs.

[heyitsanthony]

Basically, my position is that "everything looks like a key space" is a safer bet for maintaining system sanity over "write everything into the key space".

[xiang90]

@heyitsanthony What I am worrying about it that we will eventually add wait support for every set of API in etcd. Mapping the membership to a virtual key space seems fine. Or I should be clear. I hope users can use the key API to do the wait on a pre-defined key space. The actual information does not have to write through the mvcc storage.

[heyitsanthony]

@xiang90 so long as it's something like client.Watch("/", v3.WithPrefix(), v3.WithASID(ClusterASID)) I think we're in agreement. I can see how to do it with a unified key space and separate backends attached to different key regions (ala a virtual memory system) but it would be much more complex and difficult.

What's the concern with watch/wait capability on all resources?

[purpleidea]

Sorry for my noise, but it seems actually that member or leader removal/demotion events are the hard part, since if they happen because a machine is down it's hard to emulate this signal unless we get it from the lack of heartbeat data. Thanks again for your progress on this. I'll be providing more background and use cases at the upcoming CoreOSFest.

[xiang90]

moved to 3.3. not something super urgent that blocks release.

[xiang90]

moving to 3.4 again...

[purpleidea]

Is there something I can do to get this poked or help out with an upstream patch?

[purpleidea]

Hey, I've been looking into this again, and in particular, it would be very interesting if membership changes could be made into a clientv3.Op so that you could add them as part of a Txn transaction.

This is all interesting stuff that would be useful for elastically growing/shrinking/changing clusters.

[purpleidea]

(As an aside, the associated clientv3.Cmp to test if member is in cluster, and associated cmp tests would be excellent.)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

[purpleidea]

hi bot! please stop pinging here

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed after 21 days if no further activity occurs. Thank you for your contributions.

[purpleidea]

Bot!

[purpleidea]

@jmhbnz Hey, thanks for re-opening! As I guess you can tell, I've had this idea on my radar for quite some time. I think it would be very valuable for the future of etcd. I keep coming back here every year or so to check in whenever I have a new use-case for this feature =D

In any case, if you can remove the stale-bot permanently, that would be great, it annoys everyone I think. Cheers!

[jmhbnz]

There is some valuable discussion and ideas in this issue, let's not risk losing sight of it as recently played out with stale bot. Adding stage/tracked to prevent stale bot looking at it in future. Thanks for keeping an eye on this @purpleidea 🙏🏻