Add option to wait until config is fully configured

[ondrej-fabry]

Problem

The Update method of ConfiguratorService currently returns right after the given config is committed.. This can in some cases return before the input config might not be fully configured yet causing race conditions for the gRPC clients that expect config to be fully configured after the Update method returns.

The current behavior is often confusing to users, because there is not anything that would clearly indicate that some part of config is not configured completely. Also there is currently no way to change this behavior in the agent or for the request.

Details

This happens in cases where transaction succeeds, but some part config values will have PENDING state. For example this can occur for VPP AF_PACKET interfaces which depend on event from dataplane (southband notification) for the Linux VETH interface they attach to.

Here is log for such case: https://gist.githubusercontent.com/edwarnicke/b864d45d1bd7e0ce216ab0d2f4505e2a/raw/074851781a0ae31b03bef8e3547f428d7e93a95c/gistfile1.txt

Solution

The solution for this problem should add a way to set the gRPC request (or perhaps even set this as default behavior by in some agent configuration) so it waits until all config values created by committing the transaction are in non-PENDING state or until request times out. For some configs, this can actually make the Update requests to "get stuck" forever if user inputs such configuration that is incomplete in way that requires another Update call instead of some dataplane event, but it will give user power to control this, so enough documentation for this option should suffice to avoid confusion.

Implementation

• Add field bool wait_done to UpdateRequest to control this behavior from client side.
• Check the field value in the ConfiguratorService and if set to true, after committing the transaction, watch statuses of all values from the transaction until all of them are in non-PENDING state.

---

CC @milanlenco @edwarnicke