Wait for all notifications to be processed

[keith-turner]

While trying to completely automate running the stress test on the cluster, I realized I really need a fluo command line command to wait for all notifications to be processed. Scanning the notification locality groups is not good enough because of a race condition. The race conditions w/ simply scanning is that notifications not yet seen by the scan can cause notifications to be written behind the scan.

To check for this race condition I am thinking a wait command could do something like the following :

  while(true){
    int ts1 = oracle.allocateTimeStamp();
    int ntfyCount = countNotifications();
    int ts2 = oracle.allocateTimeStamp();
    if(ntfyCount == 0 && ts1 == (ts2 -1)){
      break;
    }
    int sleepTime = f(ntfyCount);
    sleep(sleepTime);
  }

[mikewalch]

In the pseudo code, you recommend counting the number of outstanding notifications and sleeping a certain amount of time based off that number. Is there particular reason for doing this?

I am concerned it will be difficult to get the algorithm/calculation correct (and cause the wait command to sleep too long or too little). How about just checking periodically after a a certain number of seconds (set by the user) if there are notifications still available?

[keith-turner]

Is there particular reason for doing this?

Was just thinking of avoiding scanning too much

I am concerned it will be difficult to get the algorithm/calculation correct

Should probably be

  int sleepTime = f(ntfyCount, numWorkers)

How about just checking periodically after a a certain number of seconds (set by the user)

I doubt the user would know what to put... I think we should make an attempt to figure something out or leave that option open for the future. Providing a user facing config option for that kinda locks us in to that, when I think the system can do things to figure out something reasonable.