in sync mode select only syncStandby as switchover candidate

[FxKu]

fixes #2276

On a two node clusters when performing a rolling update, it turned out that the recreated replica is chosen as switchover target before Patroni assigns it the synchronous status. Therefore, the switchover will fail with the following error:

2023-03-30 16:05:24,606 INFO: Updating synchronous privilege temporarily from ['test-sync-mode-0'] to []
2023-03-30 16:05:24,653 INFO: Assigning synchronous standby status to []
server signaled
2023-03-30 16:05:24,828 INFO: no action. I am (test-sync-mode-1), the leader with the lock
2023-03-30 16:05:34,670 INFO: no action. I am (test-sync-mode-1), the leader with the lock
2023-03-30 16:05:42.106 UTC [36] LOG {ticks: 0, maint: 0, retry: 0}
2023-03-30 16:05:44,682 INFO: no action. I am (test-sync-mode-1), the leader with the lock
2023-03-30 16:05:50,665 INFO: received failover request with leader=test-sync-mode-1 candidate=test-sync-mode-0 scheduled_at=None
2023-03-30 16:05:50,679 INFO: Got response from test-sync-mode-0 http://10.2.24.81:8008/patroni: {"state": "running", "postmaster_start_time": "2023-03-30 16:05:49.540527+00:00", "role": "replica", "server_version": 140007, "xlog": {"received_location": 134368768, "replayed_location": 134368768, "replayed_timestamp": "2023-03-30 16:05:24.756947+00:00", "paused": false}, "timeline": 2, "dcs_last_seen": 1680192350, "database_system_identifier": "7216365118537531469", "patroni": {"version": "3.0.1", "scope": "test-sync-mode"}}
2023-03-30 16:05:50,730 INFO: Lock owner: test-sync-mode-1; I am test-sync-mode-1
2023-03-30 16:05:50,778 WARNING: Failover candidate=test-sync-mode-0 does not match with sync_standbys=None
2023-03-30 16:05:50,778 WARNING: manual failover: members list is empty
2023-03-30 16:05:50,778 WARNING: manual failover: no healthy members found, failover is not possible
2023-03-30 16:05:50,778 INFO: Cleaning up failover key
2023-03-30 16:05:50,829 INFO: Assigning synchronous standby status to ['test-sync-mode-0']
server signaled

It would only work on the next sync. Therefore, this PR suggest to include a check if a SyncStandby candidate was found in the retry loop that calls the members endpoint. If synchronous mode is enabled the members call should be repeated when there's no syncStandby candidate.

[idanovinda]

👍

[FxKu]

👍