Question about clientv3 WithProgressNotify

[styleex]

I do not understand how this option works, why and how to use it.
The documentation for the etcd says the following (https://coreos.com/etcd/docs/latest/dev-guide/api_reference_v3.html):

The etcd server can decide how often it will send notifications based on current load.

In the documentation for clientv3 (https://godoc.org/github.com/coreos/etcd/clientv3#WithProgressNotify) :

WithProgressNotify makes watch server send periodic progress updates every 10 minutes when there is no incoming events

Whom to believe. Exactly 10 minutes?

Further. From the link (#5985) it can be seen that it is designed for application-level heatbeat.
Is this the only application of this option?
Is there an example of code or pseudocode as it should be used?

[gyuho]

You get notified in exactly 10 minutes, if no event happened to the watcher. Without WithProgressNotify, you won't get notification. Example's here https://github.com/coreos/etcd/blob/master/clientv3/example_watch_test.go#L85.

[styleex]

Thank you! I correctly understand that the typical use case is:

1. Create a watcher with WithProgressNotify
2. If there were no messages within 10 minutes, then something went wrong. Do I need to crash the program / re-create the watcher?

[gyuho]

@styleex

If there were no messages within 10 minutes, then something went wrong. Do I need to crash the program / re-create the watcher?

Event means watch event (CREATE, DELETE, UPDATE, https://github.com/coreos/etcd/blob/master/mvcc/mvccpb/kv.proto#L32-L34). So it is ok no events happen in 10 minutes, if your application did not update the key that's being watched. You do not need to recreate watcher.

[styleex]

@gyuho
I understand it.

The question is different. If the watcher created with WithProgressNotify for some reason fell off and the application did not know about it.
In this case, if ProgressNotify message did not arrive within 10 minutes, then you can conclude that some problems and something to do.
Do I get it right?

Or does progress_notify solve another task?

[gyuho]

If the watcher created with WithProgressNotify for some reason fell off and the application did not know about it.

Your application should handle the watch disconnect (e.g. resume watch with last revision you saw). Errors can be checked with Err() method on watch response. See https://github.com/coreos/etcd/blob/master/clientv3/integration/watch_test.go#L510-L520.

[heyitsanthony]

@styleex it's a way to track progress without any events on the watched keys, it's not exactly a keep alive mechanism. For example, suppose there's a watch at rev=10 on key "a", then etcd is compacted at rev=100. If there are no updates to "a" since rev=10 and the watch becomes disconnected, then it will try to resume at rev=10. etcd is compacted at rev=100, however, so the watch will get a compaction error on resume.

WithProgressNotify addresses this by periodically returning the current etcd store revision if no events have been sent over the watch. The client can then resume at that later revision instead of the compacted away revision.

[styleex]

@heyitsanthony
Good. It is already becoming more clear.
But, in this case, why progress_notify the message? After all, then a message will come with compaction_revision> 0? And I will just restore my watcher.

Or in the case described by you there will not be a message with compaction_revision> 0?

[styleex]

@gyuho Unfortunately I can not use the official golang client, but I use a small self-written wrapper over official python grpc library. And the program very rarely hangs (does not accept events, for now, for unknown reasons). In this case the grpc channel does not give any errors.
Everything looks as if the connection is established (the application even that at start read from etcd), but events do not come.

Therefore, I want to understand the capabilities of the API etcd to determine connection errors.

[heyitsanthony]

@styleex an application has no way of knowing if there were any events between 10 and 100 if there's a compaction, so it usually needs to run some extra application-specific reconciliation logic to account for possibly dropping events. The progress notification helps to avoid that.

I would not suggest using it (or watches in general) to detect connection errors. gRPC has a http2 ping keepaliave mechanism which should be a better fit.

[styleex]

@heyitsanthony Thanks for the clarification! Now I understand the purpose of this option.