MB-45967 : Ensure creds cache entries are no more than 10 seconds old

Problem:

With the addition of creds cache in eventing consumer, the following
scenario can cause LCB_ERR_AUTHENTICATION_FAILURE:
* 2 KV nodes: N1 N2 1 eventing node: N3
* creds cache on N3 has entries cached for N1 and N2
* N2 is rebalanced out followed by rebalance in all while no ops are
hitting the src collection. Because of lack of ops lcb's event loop
won't progress hence cache will still have entry for N2 with *old/stale*
credentials. We now pass these stale creds for N2 to lcb for bootstrap
which results in auth_error.

Solution:

As long as there is at least one eventing node in cluster, rebalance
will take at least 13 seconds to completion. Hence as a safe value we
fetch the creds from cbauth if cached creds entry is older than 10
seconds.

Change-Id: I5a14318bc7fee651030983f1feb33b6dbf2c50eb
Reviewed-on: http://review.couchbase.org/c/eventing/+/154046
Reviewed-by: <ankit.prabhu@couchbase.com>
Reviewed-by: CI Bot
Tested-by: <abhishek.jindal@couchbase.com>

features/include/comm.h

@@ -24,6 +24,7 @@ struct CredsInfo {
   std::string msg;
   std::string username{"EVENTINGINVALID"};
   std::string password{"EVENTINGINVALID"};
+  time_t time_fetched;
 };
 
 struct KVNodesInfo {

features/src/comm.cc

@@ -128,14 +128,23 @@ void Communicator::WriteDebuggerURL(const std::string &url) {
   }
 }
 
+// Note: The decision to fetch entries older than 10 seconds is
+// based on the fact that as of 7.0 eventing rebalance takes at-least
+// 13 seconds to run. Hence, its not possible to squeeze in a rebalance out
+// followed by in operation within 10 seconds timeframe which addresses MB-45967
 CredsInfo Communicator::GetCredsCached(const std::string &endpoint) {
+  auto time_now = GetUnixTime();
   auto find = creds_cache_.find(endpoint);
-  if (find != creds_cache_.end()) {
+  if (find != creds_cache_.end() && (find->second.time_fetched >= time_now - 10)) {
     return find->second;
   }
 
   auto credentials = GetCreds(endpoint);
-  creds_cache_[endpoint] = credentials;
+  credentials.time_fetched = time_now;
+  if (credentials.is_valid) {
+    creds_cache_[endpoint] = credentials;
+  }
+  // sends back "EVENTINGINVALID" in case of failure
   return credentials;
 }