fix: set wal_sender_timeout to 5min when joining through pg_basebackup

[fcanovai]

We explicitly set a high-enough wal_sender_timeout for join-related pg_basebackup executions. A short timeout could not be enough in case the instance is slow to send data, like when the I/O is overloaded.

Fixes #3337

[github-actions]

❗ By default, the pull request is configured to backport to all release branches.

• To stop backporting this pr, remove the label: backport-requested ◀️ or add the label 'do not backport'
• To stop backporting this pr to a certain release branch, remove the specific branch label: release-x.y

[leonardoce]

E2e tests: https://github.com/EnterpriseDB/cloudnative-pg/actions/runs/7260051781

[leonardoce]

Oops... this doesn't work with PostgreSQL 11:

{"level":"info","ts":"2023-12-19T13:28:38Z","msg":"DB not available, will retry","logging_pod":"cluster-example-2-join","err":"failed to connect to `host=cluster-example-rw user=streaming_replica database=postgres`: server error (FATAL: parameter \"wal_sender_timeout\" cannot be changed now (SQLSTATE 55P02))"}

❯ k exec -ti cluster-example-1 -- psql
Defaulted container "postgres" out of: postgres, bootstrap-controller (init)
psql (11.22 (Debian 11.22-1.pgdg110+1))
Type "help" for help.

postgres=# select version();
                                                            version                                                            
-------------------------------------------------------------------------------------------------------------------------------
 PostgreSQL 11.22 (Debian 11.22-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit
(1 row)

postgres=# \x
Expanded display is on.
postgres=# select * from pg_settings where name  = 'wal_sender_timeout';
-[ RECORD 1 ]---+---------------------------------------------------
name            | wal_sender_timeout
setting         | 5000
unit            | ms
category        | Replication / Sending Servers
short_desc      | Sets the maximum time to wait for WAL replication.
extra_desc      | 
context         | sighup
vartype         | integer
source          | configuration file
min_val         | 0
max_val         | 2147483647
enumvals        | 
boot_val        | 60000
reset_val       | 5000
sourcefile      | /var/lib/postgresql/data/pgdata/custom.conf
sourceline      | 31
pending_restart | f

postgres=# 

[leonardoce]

E2e: https://github.com/EnterpriseDB/cloudnative-pg/actions/runs/7262754322

[armru]

Would it make sense to make this parameter configurable?

[leonardoce]

Would it make sense to make this parameter configurable?

Probably yes, but if you can't have a fsync in 5 minutes, you're in big problems. Bigger than being able to clone a replica.
@fcanovai what do you think about it?

[leonardoce]

E2e: https://github.com/EnterpriseDB/cloudnative-pg/actions/runs/7264867924

[leonardoce]

We changed it to 0, meaning that we disable wal_sender_timeout for pg_basebackup. Let's see if the E2e tests are fine.
If they are, we can merge this.

[leonardoce]

/ok-to-merge e2e tests are fine