[FIX] wait before checking scheduler is running on standby node

[prakshalj0512]

• add timeout before checking scheduler is running on standby node

Issue:

• if active node completely goes down, the standby node accurate identifies the situation and attempts to start the scheduler on the standby node
• however, not enough time is allocated to allow for the restart to happen on standby node

Resolution:

• moved the time.sleep logic to the startup_scheduler method so it’s always applied

[rssanders3]

By my review of the logic, the sleep will still be executed in the original version of the code The only difference appears to be that the self.logger.info("Finished starting Scheduler on host '" + str(host) + "'") Line will be executed only after the pause. I don't see how this would address waiting for the scheduler to start up any more then what its already doing.

[prakshalj0512]

root cause: https://github.com/teamclairvoyant/airflow-scheduler-failover-controller/blob/master/scheduler_failover_controller/failover/failover_controller.py#L103-L108
in the standby logic, it just calls the startup_scheduler method and immediately checks if the scheduler is up & running. moving the sleep to the startup_scheduler func will ensure that the sleep happens both when the scheduler is started on the active node (line 98) as well as the standby node.

[rssanders3]

Ok gotcha, makes sense. LGTM