Standby cluster rebase

[redbaron]

Support for standby cluster mode

xref: #428

[sdudoladov]

can you please add the example manifest for the standby cluster ?

[sdudoladov]

can you please add the example manifest for the standby cluster ?

[redbaron]

added section to a manifests/complete-postgres-manifest.yaml file

[FxKu]

I think, it also makes sense to mention the required fields to make it work and add a link to the Patroni docs

[sdudoladov]

@erthalion can you please have a look ?

[erthalion]

@redbaron do you want to continue development of this PR? Could you then address the commentaries, and afterwards we can work on it more actively.

[redbaron]

Hi,  Yes, I ll address comments this week 15 Apr 2019, 15:05 by notifications@github.com:

…

@redbaron <https://github.com/redbaron>> do you want to continue development of this PR? Could you then address the commentaries, and afterwards we can work on it more actively. — You are receiving this because you were mentioned. Reply to this email directly, > view it on GitHub <#507 (comment)>> , or > mute the thread <https://github.com/notifications/unsubscribe-auth/AABA8B_qxWuYN2vFFo1VUPaqMeKQTaAyks5vhIcigaJpZM4bXdqx>> .

[redbaron]

@erthalion , I've rebased PR and updated manifest with an example, could you have a look please?

[FxKu]

@redbaron thanks for the update. Actually, we have (sort or) build up on your work in the meantime with #562 which we plan to merge pretty soon. Please, have a look and tell if it would work for you so we can close this PR here.

[redbaron]

interesting, I'll have a look, thanks! Sent with [ProtonMail](https://protonmail.com) Secure Email. ‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

…

On Wednesday, 19 June 2019 14:10, Felix Kunde ***@***.***> wrote: ***@***.***(https://github.com/redbaron) thanks for the update. Actually, we have (sort or) build up on your work in the meantime with [#562](#562) which we plan to merge pretty soon. Please, have a look and tell if it would work for you so we can close this PR here. — You are receiving this because you were mentioned. Reply to this email directly, [view it on GitHub](#507?email_source=notifications&email_token=AAAEB4BJANXB3AGAFFJ7SWTP3IV4JA5CNFSM4G253KY2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYBZ6PQ#issuecomment-503553854), or [mute the thread](https://github.com/notifications/unsubscribe-auth/AAAEB4A3K7WPBOBIRNRIJ2LP3IV4JANCNFSM4G253KYQ).

[redbaron]

@FxKu , there is a very little overlap except for word "standby". This PR utilizes patroni's booststrap.dcs.standby_cluster config and creates a streaming replication,

that one seems to configure standby with WAL shipping from S3

[senthilcodr]

@redbaron we also require this feature of standby cluster with streaming replication. When is this PR likely to be merged to the master?

[seuf]

For information : I've succefully rebased this PR in my workspace and now I can create a standby cluster replicating an existing database (outside of cluster). @redbaron I'd love to see your PR merged. Can re-base it please ?

[FxKu]

Can you move this part behind maximum_lag_on_failover. Only then it's under the patroni key. You can remove the black lines and maybe also # standby cluster example.. I see that some fields from the PatroniStandbyCluster struct are also missing. Are only these three fields required?

[sreenandan]

We are looking for standby cluster with streaming replication feature, when will this PR be merged?

[sreenandan]

I wanted to share my workaround with anyone who is looking for this feature.
The options that are available in patroni config (patronictl show-config) are available with endpoint.

So all you have to do is:

1. Create endpoint with the right config
   $ cat ep-standby_cluster.yaml
   apiVersion: v1
   kind: Endpoints
   metadata:
   annotations:
   config: '{"loop_wait": 10, "maximum_lag_on_failover": 33554432, "pause": null,
   "postgresql": {"parameters": {"archive_mode": "on", "archive_timeout": "1800s",
   "autovacuum_analyze_scale_factor": 0.02, "autovacuum_max_workers": 5, "autovacuum_vacuum_scale_factor":
   0.05, "checkpoint_completion_target": 0.9, "hot_standby": "on", "log_autovacuum_min_duration":
   0, "log_checkpoints": "on", "log_connections": "on", "log_disconnections": "on",
   "log_line_prefix": "%t [%p]: [%l-1] %c %x %d %u %a %h ", "log_lock_waits": "on",
   "log_min_duration_statement": 500, "log_statement": "ddl", "log_temp_files":
   0, "max_connections": "1000", "max_replication_slots": 12, "max_wal_senders":
   10, "tcp_keepalives_idle": 900, "tcp_keepalives_interval": 100, "track_functions":
   "all", "wal_level": "hot_standby", "wal_log_hints": "on"}, "use_pg_rewind":
   true, "use_slots": true}, "retry_timeout": 10, "standby_cluster": {"create_replica_methods":
   ["basebackup"], "host": "10.9.82.121", "port": 30721, "primary_slot_name": "site_c",
   "restore_command": "true"}, "ttl": 30}'
   creationTimestamp: "2021-06-04T22:00:03Z"
   labels:
   application: spilo
   cluster-name: acid-sitej
   name: acid-sitej-config
   namespace: default
   $ kubectl create -f ep-standby_cluster.yaml

2. Configure the right password for standby user in the form of secret
   $ cat secret-standbycluster.yaml
   apiVersion: v1
   data:
   password: cGluYWNvbGFkYQ==
   username: c3RhbmRieQ==
   kind: Secret
   metadata:
   creationTimestamp: "2021-06-04T21:59:35Z"
   labels:
   application: spilo
   cluster-name: acid-sitej
   team: acid
   name: standby.acid-sitej.credentials.postgresql.acid.zalan.do
   namespace: default
   selfLink: /api/v1/namespaces/default/secrets/standby.acid-sitej.credentials.postgresql.acid.zalan.do
   type: Opaque
   $ kubectl create -f secret-standbycluster.yaml

3. Create the postgresql cluster
   $ kubectl create -f standby_cluster.yaml

• Cluster: acid-sitej (6946198676853534789) ----+---------+----+-----------+
  | Member | Host | Role | State | TL | Lag in MB |
  +--------------+---------------+----------------+---------+----+-----------+
  | acid-sitej-0 | 172.21.15.248 | Standby Leader | running | 6 | |
  +--------------+---------------+----------------+---------+----+-----------+

Logs:
2021-06-05 01:06:13,607 INFO: No PostgreSQL configuration items changed, nothing to reload.
2021-06-05 01:06:13,611 INFO: Lock owner: None; I am acid-sitej-0
2021-06-05 01:06:13,827 INFO: trying to bootstrap a new standby leader
2021-06-05 01:06:15,380 INFO: replica has been created using basebackup
2021-06-05 01:06:15,381 INFO: bootstrapped clone from remote master postgresql://10.9.82.121:30721
2021-06-05 01:06:16 UTC [63]: [1-1] 60bace08.3f 0 LOG: Auto detecting pg_stat_kcache.linux_hz parameter...
2021-06-05 01:06:16 UTC [63]: [2-1] 60bace08.3f 0 LOG: pg_stat_kcache.linux_hz is set to 500000
2021-06-05 01:06:16 UTC [63]: [3-1] 60bace08.3f 0 LOG: listening on IPv4 address "0.0.0.0", port 5432
2021-06-05 01:06:16 UTC [63]: [4-1] 60bace08.3f 0 LOG: listening on IPv6 address "::", port 5432
2021-06-05 01:06:16,472 INFO: postmaster pid=63
/var/run/postgresql:5432 - no response
2021-06-05 01:06:16 UTC [63]: [5-1] 60bace08.3f 0 LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2021-06-05 01:06:16 UTC [63]: [6-1] 60bace08.3f 0 LOG: redirecting log output to logging collector process
2021-06-05 01:06:16 UTC [63]: [7-1] 60bace08.3f 0 HINT: Future log output will appear in directory "../pg_log".
/var/run/postgresql:5432 - accepting connections
/var/run/postgresql:5432 - accepting connections
2021-06-05 01:06:17,564 WARNING: Could not activate Linux watchdog device: "Can't open watchdog device: [Errno 2] No such file or directory: '/dev/watchdog'"
2021-06-05 01:06:17,725 INFO: establishing a new patroni connection to the postgres cluster
2021-06-05 01:06:17,823 INFO: initialized a new cluster
2021-06-05 01:06:17,825 INFO: Lock owner: acid-sitej-0; I am acid-sitej-0
2021-06-05 01:06:17,895 INFO: no action. i am the standby leader with the lock

You can also scale the standby cluster
$ kubectl scale sts acid-sitej --replicas=3
statefulset.apps/acid-sitej scaled

$ patronictl -c /home/postgres/postgres.yml list

• Cluster: acid-sitej (6946198676853534789) ----+---------+----+-----------+
  | Member | Host | Role | State | TL | Lag in MB |
  +--------------+---------------+----------------+---------+----+-----------+
  | acid-sitej-0 | 172.21.15.248 | Standby Leader | running | 6 | |
  | acid-sitej-1 | 172.21.2.55 | Replica | running | 6 | 0 |
  | acid-sitej-2 | 172.21.2.71 | Replica | running | 6 | 0 |
  +--------------+---------------+----------------+---------+----+-----------+

[jawabuu]

3. kubectl create -f standby_cluster.yaml

@sreenandan
Thanks for the workaround.
What are the contents of
standby_cluster.yaml?
Could you also format your yamls.

[jawabuu]

@FxKu Ping

[redbaron]

Anyone interested please open a new PR. Feel free to reuse code if you find it useful.