Add KSR recovery from transient Etcd errors

[jmedved]

The main purpose of this pull request is to implement the recovery from transient Etcd errors that are not detected by the agent status monitor that also monitors Etcd. The agent status monitor does not detect fast Etcd restarts (a couple of seconds) that may result in data loss.

To monitor data loss in Etcd we utilize Etcd's capability to record revisions for every item stored in Etcd. A KSR status record was introduced in Etcd that for now holds just KSR statistics (a collection of stats for reflector in the KSR). The KSR status record is periodically updated. With every update, the etcd rev of the status record is checked with its last recorded value; if the etcd rev is not monotonically larger than the last recorded value, a resync is triggered. This functionality is implemented in plugin_impl_ksr.go (new functions: monitorEtcdStatus(), processEtcdMonitorEvent(), checkEtcdTransientError(), ksrHasSynced() and getKsrStats()).

To support the above chanes, the code base also underwent some cleanup:

• The Writer and Lister interfaces and mocks were consolidated into a single interface/mock. This cleaned up reflector initalization and testing.

• The ability to inject errors into the mock was made more fine grained (one for the ListValues() function, one for all other data operations) so that all error paths in KSR data resync can be tested in unit tests.

• Reflector object type handling was cleaned up (constants introduced for each object type and then used consistently throughout the code).

[coveralls]

Coverage decreased (-0.8%) to 76.065% when pulling c65ee1b on jmedved:master into f104b8b on contiv:master.

[lukasmacko]

typo mockKeyProtoValBroker - missing l

[tiewei]

💯