refactor restarting instances

[FxKu]

refactored restart of instances by moving them into a loop over pods. Same for patching Postgres with checkAndSetGlobalPostgreSQLConfiguration

From:

checkAndSetGlobalPostgreSQLConfiguration()
-> loop over pods and patch Patroni config (Postgresql.Parameters)
-> return true if bootstrap only parameters are defined in manifest to trigger restart

restartInstances()
-> call Patroni's `config` endpoint to get `ttl` duration for post restart wait
-> loop over pods and call Patroni's `restart` endpoint

To:

for pod in pods:
  syncPostgreSQLConfiguration(pod)

func syncPostgreSQLConfiguration(pod)
  call Patroni's `config` endpoint

  restartRequired := checkAndSetPostgreSQLConfiguration( pass config for comparison )
    -> patch Patroni config (Postgresql.Parameters)
    -> return true if bootstrap only parameters are defined in manifest *and differ from effective config* to trigger restart

  if restartRequired:
    -> call Patroni's `restart` endpoint

This PR also adds the following functionality:

• extends checkAndSetGlobalPostgreSQLConfiguration to compare Patroni config with c.Spec.Patroni, too:
• gathers all options to set in one config object and send it to a new c.patroni.SetConfig endpoint which supports the full config hierarchy compared to c.patroni.SetPostgresParameters which is only for postrgesql.parameters
• extend (and rename) test_decrease_max_connections e2e test (or create a new one) for config changes

[sdudoladov]

duplicate of 246-252 ?

[FxKu]

yes, there's a lot of duplicate code. Don't understand why. I thought to better do it, too 😄
Someone should tidy this part up.

[sdudoladov]

why does waiting or the master uses the self.test_namespace and waiting for the replica does not?

[FxKu]

an oversight, indeed. Might explain why the test fails atm

[sdudoladov]

but that only tells you if the Patroni config was updated, not if the change was actually applied on the PG level.
Patroni has the pending_restart flag to indicate such situation

$ curl -s http://localhost:8008/patroni | jq .
{
  "pending_restart": true,

[FxKu]

true. Added a check to query the settings in the database

[sdudoladov]

that is arguably still an error because the Sync will fail to complete without the list o pods

[FxKu]

The error was not replaced but moved into the loop. I took inspiration from line 276 where it's also just an info log line. The function would not fail with an empty list of pods. I've change the log level to warning.

[Jan-M]

👍

[FxKu]

👍