Automate in place major upgrade for PostgreSQL #301

vitabaks · 2023-04-12T17:03:51Z

issue #162

Example:

ansible-playbook pg_upgrade.yml -e "pg_old_version=13 pg_new_version=14"

The upgrade is supported starting from PostgreSQL 9.3 and up to the latest PostgreSQL version.

Upgrade Plan and more details can be found here

update TODO list

required for upgrade replicas with rsync

write to pgbouncer_pool_pause.log information about queries that were terminated (after 30 seconds of waiting).

- Add separate tasks for PostgreSQL 9.x and 10+ versions. - Implement additional pre-checks to ensure smooth operations. - Refactor tasks to improve readability and maintainability.

… blockers

vitabaks · 2023-08-17T11:28:21Z

Test 7: final test before release

PLAY [Upgrade PostgreSQL 14 to version 15 (PostgreSQL HA Cluster: postgres-cluster)] ***

TASK [Gathering Facts] *********************************************************
ok: [10.172.0.20]
ok: [10.172.0.21]
ok: [10.172.0.22]
[WARNING]: Module remote_tmp /var/lib/postgresql/.ansible/tmp did not exist and
was created with a mode of 0700, this may cause issues when running as another
user. To avoid this, create the remote_tmp dir with the correct permissions
manually

TASK [[Prepare] Get Patroni Cluster Leader Node] *******************************
ok: [10.172.0.20]
ok: [10.172.0.22]
ok: [10.172.0.21]

TASK [[Prepare] Add host to group "primary" (in-memory inventory)] *************
ok: [10.172.0.20] => (item=10.172.0.20)

TASK [[Prepare] Add hosts to group "secondary" (in-memory inventory)] **********
ok: [10.172.0.20] => (item=10.172.0.21)
ok: [10.172.0.20] => (item=10.172.0.22)

TASK [Print Patroni Cluster info] **********************************************
ok: [10.172.0.20] => {
    "msg": [
        "Cluster Name: postgres-cluster",
        "Cluster Leader: pgnode01"
    ]
}

PLAY [(1/7) PRE-UPGRADE: Perform Pre-Checks] ***********************************

TASK [Include OS-specific variables] *******************************************
ok: [10.172.0.20]
ok: [10.172.0.21]
ok: [10.172.0.22]

TASK [Include upgrade variables] ***********************************************
ok: [10.172.0.20]
ok: [10.172.0.21]
ok: [10.172.0.22]

TASK [Running Pre-Checks] ******************************************************

TASK [upgrade : [Pre-Check] Make sure the ansible required Python library is installed] ***
ok: [10.172.0.22] => (item=pexpect)
ok: [10.172.0.21] => (item=pexpect)
ok: [10.172.0.20] => (item=pexpect)

TASK [upgrade : [Pre-Check] Test PostgreSQL database access using a unix socket] ***
ok: [10.172.0.20]
ok: [10.172.0.21]
ok: [10.172.0.22]

TASK [upgrade : [Pre-Check] Check the current version of PostgreSQL] ***********
ok: [10.172.0.20]

TASK [upgrade : Set variable 'current_pg_version'] *****************************
ok: [10.172.0.20]

TASK [upgrade : [Pre-Check] Ensure new data directory is different from the current one] ***
ok: [10.172.0.20]

TASK [upgrade : [Pre-Check] Make sure that physical replication is active] *****
ok: [10.172.0.20]

TASK [upgrade : [Pre-Check] Make sure there is no high replication lag (more than 10.00 MB)] ***
ok: [10.172.0.20]

TASK [upgrade : [Pre-Check] Make sure there are no long-running transactions (more than 15 seconds)] ***
ok: [10.172.0.21]
ok: [10.172.0.22]
ok: [10.172.0.20]

TASK [upgrade : [Pre-Check] Make sure that SSH key-based authentication is configured between cluster nodes] ***
included: /workspace/postgresql_cluster/roles/upgrade/tasks/ssh-keys.yml for 10.172.0.20, 10.172.0.21, 10.172.0.22

TASK [upgrade : Make sure that the openssh-client package is installed] ********
ok: [10.172.0.20]
ok: [10.172.0.21]
ok: [10.172.0.22]

TASK [upgrade : Make sure the SSH key for user "postgres" exists] **************
changed: [10.172.0.22]
changed: [10.172.0.20]
changed: [10.172.0.21]

TASK [upgrade : Fetch public SSH keys from database servers] *******************
ok: [10.172.0.21]
ok: [10.172.0.20]
ok: [10.172.0.22]

TASK [upgrade : Add public SSH keys to authorized_keys] ************************
changed: [10.172.0.22]
changed: [10.172.0.21]
changed: [10.172.0.20]

TASK [upgrade : Remove public SSH keys from localhost] *************************
ok: [10.172.0.20 -> localhost] => (item=10.172.0.20)
ok: [10.172.0.20 -> localhost] => (item=10.172.0.21)
ok: [10.172.0.20 -> localhost] => (item=10.172.0.22)

TASK [upgrade : known_hosts | for each host, scan for its ssh public key] ******
ok: [10.172.0.21] => (item=10.172.0.20)
ok: [10.172.0.20] => (item=10.172.0.20)
ok: [10.172.0.22] => (item=10.172.0.20)
ok: [10.172.0.20] => (item=10.172.0.21)
ok: [10.172.0.21] => (item=10.172.0.21)
ok: [10.172.0.22] => (item=10.172.0.21)
ok: [10.172.0.21] => (item=10.172.0.22)
ok: [10.172.0.20] => (item=10.172.0.22)
ok: [10.172.0.22] => (item=10.172.0.22)

TASK [upgrade : known_hosts | for each host, add/update the public key in the "~postgres/.ssh/known_hosts"] ***
changed: [10.172.0.21] => (item=None)
changed: [10.172.0.20] => (item=None)
changed: [10.172.0.22] => (item=None)
changed: [10.172.0.21] => (item=None)
changed: [10.172.0.22] => (item=None)
changed: [10.172.0.20] => (item=None)
changed: [10.172.0.20] => (item=None)
changed: [10.172.0.20]
changed: [10.172.0.21] => (item=None)
changed: [10.172.0.21]
changed: [10.172.0.22] => (item=None)
changed: [10.172.0.22]

TASK [upgrade : [Pre-Check] Make sure that the rsync package are installed] ****
ok: [10.172.0.21]
ok: [10.172.0.20]
ok: [10.172.0.22]

TASK [upgrade : [Pre-Check] Rsync Checks: create testrsync file on Primary] ****
changed: [10.172.0.20]

TASK [upgrade : [Pre-Check] Rsync Checks: test rsync and ssh key access] *******
changed: [10.172.0.20] => (item=10.172.0.21)
changed: [10.172.0.20] => (item=10.172.0.22)

TASK [upgrade : [Pre-Check] Cleanup testrsync file] ****************************
changed: [10.172.0.20]
changed: [10.172.0.22]
changed: [10.172.0.21]

TASK [upgrade : [Pre-Check] Check if PostgreSQL tablespaces exist] *************
ok: [10.172.0.20]

TASK [upgrade : [Pre-Check] Test PgBouncer access via localhost] ***************
ok: [10.172.0.20]
ok: [10.172.0.21]
ok: [10.172.0.22]

PLAY [(2/7) PRE-UPGRADE: Install new PostgreSQL packages] **********************

TASK [Include OS-specific variables] *******************************************
ok: [10.172.0.20]
ok: [10.172.0.21]
ok: [10.172.0.22]

TASK [Include upgrade variables] ***********************************************
ok: [10.172.0.20]
ok: [10.172.0.21]
ok: [10.172.0.22]

TASK [Install packages] ********************************************************

TASK [upgrade : Update apt cache] **********************************************
ok: [10.172.0.20]
ok: [10.172.0.21]
ok: [10.172.0.22]

TASK [upgrade : Install PostgreSQL 15 packages] ********************************
changed: [10.172.0.22] => (item=postgresql-15)
changed: [10.172.0.20] => (item=postgresql-15)
changed: [10.172.0.21] => (item=postgresql-15)
ok: [10.172.0.22] => (item=postgresql-client-15)
ok: [10.172.0.20] => (item=postgresql-client-15)
ok: [10.172.0.21] => (item=postgresql-client-15)
changed: [10.172.0.20] => (item=postgresql-server-dev-15)
changed: [10.172.0.22] => (item=postgresql-server-dev-15)
changed: [10.172.0.21] => (item=postgresql-server-dev-15)
ok: [10.172.0.20] => (item=postgresql-contrib-15)
ok: [10.172.0.22] => (item=postgresql-contrib-15)
ok: [10.172.0.21] => (item=postgresql-contrib-15)

TASK [upgrade : Install TimescaleDB package for PostgreSQL 15] *****************
changed: [10.172.0.20] => (item= timescaledb-2-postgresql-15 
)
changed: [10.172.0.21] => (item= timescaledb-2-postgresql-15 
)
changed: [10.172.0.22] => (item= timescaledb-2-postgresql-15 
)

PLAY [(3/7) PRE-UPGRADE: Initialize new db, schema compatibility check, and pg_upgrade --check] ***

TASK [Include OS-specific variables] *******************************************
ok: [10.172.0.20]
ok: [10.172.0.21]
ok: [10.172.0.22]

TASK [Include upgrade variables] ***********************************************
ok: [10.172.0.20]
ok: [10.172.0.21]
ok: [10.172.0.22]

TASK [Create Data directory and initdb] ****************************************

TASK [upgrade : Make sure new PostgreSQL data directory "/pgdata/15/main" exists] ***
changed: [10.172.0.22]
changed: [10.172.0.21]
changed: [10.172.0.20]

TASK [upgrade : Make sure new PostgreSQL data directory "/pgdata/15/main" is not initialized] ***
ok: [10.172.0.21]
ok: [10.172.0.20]
ok: [10.172.0.22]

TASK [upgrade : Get the current install user] **********************************
ok: [10.172.0.20]

TASK [upgrade : Get the current encodig and data_checksums settings] ***********
ok: [10.172.0.20] => (item=server_encoding)
ok: [10.172.0.20] => (item=lc_collate)
ok: [10.172.0.20] => (item=lc_ctype)
ok: [10.172.0.20] => (item=lc_messages)
ok: [10.172.0.20] => (item=lc_monetary)
ok: [10.172.0.20] => (item=lc_numeric)
ok: [10.172.0.20] => (item=lc_time)
ok: [10.172.0.20] => (item=data_checksums)

TASK [upgrade : Initialize new PostgreSQL data directory with default config files] ***
changed: [10.172.0.20]
changed: [10.172.0.22]
changed: [10.172.0.21]

TASK [Copy files] **************************************************************

TASK [upgrade : Get the current shared_preload_libraries settings] *************
ok: [10.172.0.20]

TASK [upgrade : Check if PostgreSQL is running] ********************************
ok: [10.172.0.20]

TASK [upgrade : Start new PostgreSQL on port 5433 to check the schema compatibility] ***
changed: [10.172.0.20]

TASK [upgrade : Wait for PostgreSQL to start] **********************************
changed: [10.172.0.20]

TASK [upgrade : Check the compatibility of the database schema with the PostgreSQL 15] ***
changed: [10.172.0.20]

TASK [upgrade : Wait for the schema compatibility check to complete.] **********
changed: [10.172.0.20]

TASK [upgrade : Checking the result of the schema compatibility] ***************
ok: [10.172.0.20]

TASK [upgrade : Result of checking the compatibility of the scheme - success] ***
ok: [10.172.0.20] => {
    "msg": "The database schema are compatible with PostgreSQL 15"
}

TASK [upgrade : Drop new PostgreSQL to re-initdb (perform pg_dropcluster)] *****
changed: [10.172.0.20]

TASK [upgrade : Reinitialize the database after checking schema compatibility] ***
included: /workspace/postgresql_cluster/roles/upgrade/tasks/initdb.yml for 10.172.0.20

TASK [upgrade : Make sure new PostgreSQL data directory "/pgdata/15/main" exists] ***
changed: [10.172.0.20]

TASK [upgrade : Make sure new PostgreSQL data directory "/pgdata/15/main" is not initialized] ***
ok: [10.172.0.20]

TASK [upgrade : Get the current install user] **********************************
ok: [10.172.0.20]

TASK [upgrade : Get the current encodig and data_checksums settings] ***********
ok: [10.172.0.20] => (item=server_encoding)
ok: [10.172.0.20] => (item=lc_collate)
ok: [10.172.0.20] => (item=lc_ctype)
ok: [10.172.0.20] => (item=lc_messages)
ok: [10.172.0.20] => (item=lc_monetary)
ok: [10.172.0.20] => (item=lc_numeric)
ok: [10.172.0.20] => (item=lc_time)
ok: [10.172.0.20] => (item=data_checksums)

TASK [upgrade : Initialize new PostgreSQL data directory with default config files] ***
changed: [10.172.0.20]

TASK [upgrade : Get the current shared_preload_libraries settings] *************
ok: [10.172.0.20]

TASK [upgrade : Set the variable: pg_shared_preload_libraries_value] ***********
ok: [10.172.0.20]

TASK [upgrade : Verify the two clusters are compatible (pg_upgrade --check)] ***
changed: [10.172.0.20]

TASK [upgrade : Print the result of the pg_upgrade check] **********************
ok: [10.172.0.20] => {
    "failed_when_result": false,
    "pg_upgrade_check_result.stdout_lines": [
        "Performing Consistency Checks on Old Live Server",
        "------------------------------------------------",
        "Checking cluster versions                                   ok",
        "Checking database user is the install user                  ok",
        "Checking database connection settings                       ok",
        "Checking for prepared transactions                          ok",
        "Checking for system-defined composite types in user tables  ok",
        "Checking for reg* data types in user tables                 ok",
        "Checking for contrib/isn with bigint-passing mismatch       ok",
        "Checking for presence of required libraries                 ok",
        "Checking database user is the install user                  ok",
        "Checking for prepared transactions                          ok",
        "Checking for new cluster tablespace directories             ok",
        "",
        "*Clusters are compatible*"
    ]
}

PLAY [(4/7) PRE-UPGRADE: Prepare the Patroni configuration] ********************

TASK [Include OS-specific variables] *******************************************
ok: [10.172.0.20]
ok: [10.172.0.21]
ok: [10.172.0.22]

TASK [Include upgrade variables] ***********************************************
ok: [10.172.0.20]
ok: [10.172.0.21]
ok: [10.172.0.22]

TASK [Patroni config] **********************************************************

TASK [upgrade : Edit patroni.yml | update parameters: data_dir, bin_dir, config_dir] ***
changed: [10.172.0.22] => (item=data_dir: /pgdata/15/main)
changed: [10.172.0.21] => (item=data_dir: /pgdata/15/main)
changed: [10.172.0.20] => (item=data_dir: /pgdata/15/main)
changed: [10.172.0.21] => (item=bin_dir: /usr/lib/postgresql/15/bin)
changed: [10.172.0.22] => (item=bin_dir: /usr/lib/postgresql/15/bin)
changed: [10.172.0.20] => (item=bin_dir: /usr/lib/postgresql/15/bin)
changed: [10.172.0.21] => (item=config_dir: /etc/postgresql/15/main)
changed: [10.172.0.22] => (item=config_dir: /etc/postgresql/15/main)
changed: [10.172.0.20] => (item=config_dir: /etc/postgresql/15/main)

TASK [upgrade : Edit patroni.yml | check if the 'stats_temp_directory' parameter is specified] ***
ok: [10.172.0.21]
ok: [10.172.0.20]
ok: [10.172.0.22]

TASK [upgrade : Edit patroni.yml | remove parameter: 'stats_temp_directory'] ***
changed: [10.172.0.20]
changed: [10.172.0.21]
changed: [10.172.0.22]

TASK [upgrade : Copy pg_hba.conf to /etc/postgresql/15/main] *******************
changed: [10.172.0.21]
changed: [10.172.0.20]
changed: [10.172.0.22]

PLAY [(5/7) UPGRADE: Upgrade PostgreSQL] ***************************************

TASK [Include OS-specific variables] *******************************************
ok: [10.172.0.20]
ok: [10.172.0.21]
ok: [10.172.0.22]

TASK [Include upgrade variables] ***********************************************
ok: [10.172.0.20]
ok: [10.172.0.21]
ok: [10.172.0.22]

TASK [Enable maintenance mode] *************************************************

TASK [upgrade : Pause Patroni cluster (enable maintenance mode)] ***************
changed: [10.172.0.20]

TASK [upgrade : Stop confd service] ********************************************
changed: [10.172.0.20]
changed: [10.172.0.21]
changed: [10.172.0.22]

TASK [upgrade : Update haproxy conf file (disable http-checks)] ****************
changed: [10.172.0.20]
changed: [10.172.0.22]
changed: [10.172.0.21]

TASK [upgrade : Reload haproxy service] ****************************************
changed: [10.172.0.22]
changed: [10.172.0.20]
changed: [10.172.0.21]

TASK [upgrade : Stop Patroni service] ******************************************
changed: [10.172.0.21]
changed: [10.172.0.20]
changed: [10.172.0.22]
FAILED - RETRYING: [10.172.0.20]: Wait until the Patroni cluster is stopped (30 retries left).
FAILED - RETRYING: [10.172.0.20]: Wait until the Patroni cluster is stopped (29 retries left).

TASK [upgrade : Wait until the Patroni cluster is stopped] *********************
ok: [10.172.0.20]

TASK [Stop Services] ***********************************************************

TASK [upgrade : Execute CHECKPOINT before stopping PostgreSQL] *****************
changed: [10.172.0.20]
changed: [10.172.0.21]
changed: [10.172.0.22]
FAILED - RETRYING: [10.172.0.21]: Wait for the CHECKPOINT to complete (180 retries left).
FAILED - RETRYING: [10.172.0.22]: Wait for the CHECKPOINT to complete (180 retries left).
FAILED - RETRYING: [10.172.0.20]: Wait for the CHECKPOINT to complete (180 retries left).
FAILED - RETRYING: [10.172.0.21]: Wait for the CHECKPOINT to complete (179 retries left).
FAILED - RETRYING: [10.172.0.22]: Wait for the CHECKPOINT to complete (179 retries left).
FAILED - RETRYING: [10.172.0.20]: Wait for the CHECKPOINT to complete (179 retries left).

TASK [upgrade : Wait for the CHECKPOINT to complete] ***************************
changed: [10.172.0.21]
changed: [10.172.0.22]
changed: [10.172.0.20]

TASK [upgrade : Wait until replication lag is less than 10.00 MB] **************
ok: [10.172.0.20]

TASK [upgrade : Perform PAUSE on all pgbouncers servers] ***********************
included: /workspace/postgresql_cluster/roles/upgrade/tasks/pgbouncer_pause.yml for 10.172.0.20, 10.172.0.21, 10.172.0.22

TASK [upgrade : PAUSE PgBouncer pools] *****************************************
changed: [10.172.0.20]

TASK [upgrade : Stop PostgreSQL on the Leader] *********************************
changed: [10.172.0.20]

TASK [upgrade : Stop PostgreSQL on the Replica] ********************************
changed: [10.172.0.21]
changed: [10.172.0.22]

TASK [upgrade : Check if PostgreSQL 14 is stopped] *****************************
ok: [10.172.0.21]
ok: [10.172.0.20]
ok: [10.172.0.22]

TASK [upgrade : Check if PostgreSQL 15 is stopped] *****************************
ok: [10.172.0.21]
ok: [10.172.0.22]
ok: [10.172.0.20]

TASK [Check 'Latest checkpoint location'] **************************************

TASK [upgrade : Get 'Latest checkpoint location' on the Leader] ****************
ok: [10.172.0.20]

TASK [upgrade : Get 'Latest checkpoint location' on the Replicas] **************
ok: [10.172.0.22]
ok: [10.172.0.21]

TASK [upgrade : Print 'Latest checkpoint location' for the Leader] *************
ok: [10.172.0.20] => {
    "msg": "Leader's latest checkpoint location: 0/6E000028"
}

TASK [upgrade : Print 'Latest checkpoint location' for the Replica] ************
ok: [10.172.0.21] => {
    "msg": "Replica: 10.172.0.21, latest checkpoint location: 0/6E000028"
}
ok: [10.172.0.22] => {
    "msg": "Replica: 10.172.0.22, latest checkpoint location: 0/6E000028"
}

TASK [upgrade : Check if all 'Latest checkpoint location' values match] ********
ok: [10.172.0.21]
ok: [10.172.0.22]

TASK [upgrade : SUCCESS: 'Latest checkpoint location' values match on all cluster nodes] ***
ok: [10.172.0.20] => {
    "msg": "'Latest checkpoint location' is the same on the leader and its standbys"
}

TASK [Upgrade Primary] *********************************************************

TASK [upgrade : Upgrade the PostgreSQL to version 15 on the Primary (using pg_upgrade --link)] ***
changed: [10.172.0.20]

TASK [upgrade : Print the result of the pg_upgrade] ****************************
ok: [10.172.0.20] => {
    "msg": [
        [
            "Performing Consistency Checks",
            "-----------------------------",
            "Checking cluster versions                                   ok",
            "Checking database user is the install user                  ok",
            "Checking database connection settings                       ok",
            "Checking for prepared transactions                          ok",
            "Checking for system-defined composite types in user tables  ok",
            "Checking for reg* data types in user tables                 ok",
            "Checking for contrib/isn with bigint-passing mismatch       ok",
            "Creating dump of global objects                             ok",
            "Creating dump of database schemas                           ok",
            "Checking for presence of required libraries                 ok",
            "Checking database user is the install user                  ok",
            "Checking for prepared transactions                          ok",
            "Checking for new cluster tablespace directories             ok",
            "",
            "If pg_upgrade fails after this point, you must re-initdb the",
            "new cluster before continuing.",
            "",
            "Performing Upgrade",
            "------------------",
            "Analyzing all rows in the new cluster                       ok",
            "Freezing all rows in the new cluster                        ok",
            "Deleting files from new pg_xact                             ok",
            "Copying old pg_xact to new server                           ok",
            "Setting oldest XID for new cluster                          ok",
            "Setting next transaction ID and epoch for new cluster       ok",
            "Deleting files from new pg_multixact/offsets                ok",
            "Copying old pg_multixact/offsets to new server              ok",
            "Deleting files from new pg_multixact/members                ok",
            "Copying old pg_multixact/members to new server              ok",
            "Setting next multixact ID and offset for new cluster        ok",
            "Resetting WAL archives                                      ok",
            "Setting frozenxid and minmxid counters in new cluster       ok",
            "Restoring global objects in the new cluster                 ok",
            "Restoring database schemas in the new cluster               ok",
            "Adding \".old\" suffix to old global/pg_control               ok",
            "",
            "If you want to start the old cluster, you will need to remove",
            "the \".old\" suffix from /pgdata/14/main/global/pg_control.old.",
            "Because \"link\" mode was used, the old cluster cannot be safely",
            "started once the new cluster has been started.",
            "",
            "Linking user relation files                                 ok",
            "Setting next OID for new cluster                            ok",
            "Sync data directory to disk                                 ok",
            "Creating script to delete old cluster                       ok",
            "Checking for extension updates                              ok",
            "",
            "Upgrade Complete",
            "----------------",
            "Optimizer statistics are not transferred by pg_upgrade.",
            "Once you start the new server, consider running:",
            "    /usr/lib/postgresql/15/bin/vacuumdb -U postgres --all --analyze-in-stages",
            "",
            "Running this script will delete the old cluster's data files:",
            "    ./delete_old_cluster.sh"
        ]
    ]
}

TASK [Upgrade Secondary] *******************************************************

TASK [upgrade : Make sure that the new data directory "/pgdata/15/main" are empty on the Replica] ***
changed: [10.172.0.21] => (item=absent)
changed: [10.172.0.22] => (item=absent)
changed: [10.172.0.21] => (item=directory)
changed: [10.172.0.22] => (item=directory)

TASK [upgrade : Upgrade the PostgreSQL on the Replica (using rsync --hard-links)] ***
changed: [10.172.0.20]
FAILED - RETRYING: [10.172.0.20]: Wait for the rsync to complete. (1800 retries left).

TASK [upgrade : Wait for the rsync to complete.] *******************************
changed: [10.172.0.20]

TASK [Create WAL dir symlink] **************************************************

TASK [upgrade : Make sure /pgdata/15/main/pg_wal is not symlink] ***************
ok: [10.172.0.20]
ok: [10.172.0.22]
ok: [10.172.0.21]

TASK [upgrade : Make sure the custom WAL directory "/pgwal/15/pg_wal" exists and is empty] ***
ok: [10.172.0.22] => (item=absent)
ok: [10.172.0.21] => (item=absent)
ok: [10.172.0.20] => (item=absent)
changed: [10.172.0.20] => (item=directory)
changed: [10.172.0.22] => (item=directory)
changed: [10.172.0.21] => (item=directory)

TASK [upgrade : Synchronize /pgdata/15/main/pg_wal to /pgwal/15/pg_wal] ********
changed: [10.172.0.21]
changed: [10.172.0.20]
changed: [10.172.0.22]

TASK [upgrade : Rename pg_wal to pg_wal_old] ***********************************
changed: [10.172.0.20]
changed: [10.172.0.21]
changed: [10.172.0.22]

TASK [upgrade : Create symlink /pgdata/15/main/pg_wal -> /pgwal/15/pg_wal] *****
changed: [10.172.0.22]
changed: [10.172.0.20]
changed: [10.172.0.21]

TASK [upgrade : Remove pg_wal_old directory] ***********************************
changed: [10.172.0.20]
changed: [10.172.0.22]
changed: [10.172.0.21]

TASK [Remove old cluster from DCS] *********************************************

TASK [upgrade : Remove existing cluster "postgres-cluster" from DCS] ***********
changed: [10.172.0.20]

TASK [Start Services] **********************************************************

TASK [upgrade : Start Patroni service on the Cluster Leader] *******************
changed: [10.172.0.20]

TASK [upgrade : Wait for Patroni port "8008" to become open on the host] *******
ok: [10.172.0.20]

TASK [upgrade : Check Patroni is healthy on the Leader] ************************
ok: [10.172.0.20]

TASK [upgrade : Perform RESUME PgBouncer pools on the Leader] ******************
included: /workspace/postgresql_cluster/roles/upgrade/tasks/pgbouncer_resume.yml for 10.172.0.20

TASK [upgrade : RESUME PgBouncer pools] ****************************************
changed: [10.172.0.20]

TASK [upgrade : Start Patroni service on the Cluster Replica] ******************
changed: [10.172.0.22]
changed: [10.172.0.21]

TASK [upgrade : Wait for Patroni port "8008" to become open on the host] *******
ok: [10.172.0.21]
ok: [10.172.0.22]

TASK [upgrade : Check Patroni is healthy on the Replica] ***********************
ok: [10.172.0.21]
ok: [10.172.0.22]

TASK [upgrade : Perform RESUME PgBouncer pools on the Replica] *****************
included: /workspace/postgresql_cluster/roles/upgrade/tasks/pgbouncer_resume.yml for 10.172.0.21, 10.172.0.22

TASK [upgrade : RESUME PgBouncer pools] ****************************************
changed: [10.172.0.22]
changed: [10.172.0.21]

TASK [upgrade : Check PostgreSQL is started and accepting connections] *********
ok: [10.172.0.20]
ok: [10.172.0.21]
ok: [10.172.0.22]

TASK [Disable maintenance mode] ************************************************

TASK [upgrade : Update haproxy conf file (enable http-checks)] *****************
changed: [10.172.0.20]
changed: [10.172.0.21]
changed: [10.172.0.22]

TASK [upgrade : Reload haproxy service] ****************************************
changed: [10.172.0.20]
changed: [10.172.0.21]
changed: [10.172.0.22]

TASK [upgrade : Start confd service] *******************************************
changed: [10.172.0.21]
changed: [10.172.0.22]
changed: [10.172.0.20]

PLAY [(6/7) POST-UPGRADE: Perform Post-Checks and Update extensions] ***********

TASK [Include OS-specific variables] *******************************************
ok: [10.172.0.20]
ok: [10.172.0.21]
ok: [10.172.0.22]

TASK [Include upgrade variables] ***********************************************
ok: [10.172.0.20]
ok: [10.172.0.21]
ok: [10.172.0.22]

TASK [Running Post-Checks] *****************************************************

TASK [upgrade : Make sure that physical replication is active] *****************
ok: [10.172.0.20]

TASK [upgrade : Create a table "test_replication" with 10000 rows on the Primary] ***
changed: [10.172.0.20]

TASK [upgrade : Wait until the PostgreSQL replica is synchronized] *************
ok: [10.172.0.22]
ok: [10.172.0.21]

TASK [upgrade : Drop a table "test_replication"] *******************************
changed: [10.172.0.20]

TASK [upgrade : Print the result of checking the number of records] ************
ok: [10.172.0.21] => {
    "msg": [
        "The PostgreSQL Replication is OK for replica pgnode02",
        "The number of records in the test_replication table the same as the Primary (10000 rows)"
    ]
}
ok: [10.172.0.22] => {
    "msg": [
        "The PostgreSQL Replication is OK for replica pgnode03",
        "The number of records in the test_replication table the same as the Primary (10000 rows)"
    ]
}

TASK [Update extensions] *******************************************************

TASK [upgrade : Get a list of databases] ***************************************
ok: [10.172.0.20]

TASK [upgrade : Update extensions in each database] ****************************
included: /workspace/postgresql_cluster/roles/upgrade/tasks/update_extensions.yml for 10.172.0.20 => (item=template1)
included: /workspace/postgresql_cluster/roles/upgrade/tasks/update_extensions.yml for 10.172.0.20 => (item=postgres)

TASK [upgrade : Get list of installed PostgreSQL extensions (database: template1)] ***
ok: [10.172.0.20]

TASK [upgrade : Get list of old PostgreSQL extensions (database: template1)] ***
ok: [10.172.0.20]

TASK [upgrade : The extensions are up-to-date (database: template1)] ***********
ok: [10.172.0.20] => {
    "msg": [
        "The extension versions are up-to-date for the database template1",
        "No update is required."
    ]
}

TASK [upgrade : Get list of installed PostgreSQL extensions (database: postgres)] ***
ok: [10.172.0.20]

TASK [upgrade : Get list of old PostgreSQL extensions (database: postgres)] ****
ok: [10.172.0.20]

TASK [upgrade : The extensions are up-to-date (database: postgres)] ************
ok: [10.172.0.20] => {
    "msg": [
        "The extension versions are up-to-date for the database postgres",
        "No update is required."
    ]
}

PLAY [(7/7) POST-UPGRADE: Analyze a PostgreSQL database (update optimizer statistics) and Post-Upgrade tasks] ***

TASK [Include OS-specific variables] *******************************************
ok: [10.172.0.20]
ok: [10.172.0.21]
ok: [10.172.0.22]

TASK [Include upgrade variables] ***********************************************
ok: [10.172.0.20]
ok: [10.172.0.21]
ok: [10.172.0.22]

TASK [Analyze database] ********************************************************

TASK [upgrade : Start pg_terminator script: Monitor locks and terminate the 'ANALYZE' blockers] ***
changed: [10.172.0.20]

TASK [upgrade : Run vacuumdb to analyze the PostgreSQL databases] **************
changed: [10.172.0.20]
FAILED - RETRYING: [10.172.0.20]: Collecting statistics in progress. Wait for the analyze to complete. (360 retries left).

TASK [upgrade : Collecting statistics in progress. Wait for the analyze to complete.] ***
changed: [10.172.0.20]

TASK [upgrade : Stop pg_terminator script] *************************************
changed: [10.172.0.20]

TASK [Running Post-Upgrade tasks] **********************************************

TASK [upgrade : Ensure the current data directory is the new data directory] ***
ok: [10.172.0.20]
ok: [10.172.0.21]
ok: [10.172.0.22]

TASK [upgrade : Delete the old PostgreSQL data directory (perform pg_dropcluster)] ***
changed: [10.172.0.20]
changed: [10.172.0.21]
changed: [10.172.0.22]

TASK [upgrade : Delete the old PostgreSQL WAL directory] ***********************
ok: [10.172.0.20]
ok: [10.172.0.21]
ok: [10.172.0.22]

TASK [upgrade : Remove old PostgreSQL packages] ********************************
changed: [10.172.0.21] => (item=postgresql-14)
changed: [10.172.0.22] => (item=postgresql-14)
changed: [10.172.0.20] => (item=postgresql-14)
changed: [10.172.0.20] => (item=postgresql-client-14)
changed: [10.172.0.22] => (item=postgresql-client-14)
changed: [10.172.0.21] => (item=postgresql-client-14)
ok: [10.172.0.20] => (item=postgresql-server-dev-14)
ok: [10.172.0.21] => (item=postgresql-server-dev-14)
ok: [10.172.0.22] => (item=postgresql-server-dev-14)
ok: [10.172.0.20] => (item=postgresql-contrib-14)
ok: [10.172.0.21] => (item=postgresql-contrib-14)
ok: [10.172.0.22] => (item=postgresql-contrib-14)

TASK [upgrade : Check the Patroni cluster state] *******************************
ok: [10.172.0.20]

TASK [upgrade : Check the current PostgreSQL version] **************************
ok: [10.172.0.20]

TASK [upgrade : List the Patroni cluster members] ******************************
ok: [10.172.0.20] => {
    "msg": [
        "+ Cluster: postgres-cluster -------+-----------+----+-----------+",
        "| Member   | Host        | Role    | State     | TL | Lag in MB |",
        "+----------+-------------+---------+-----------+----+-----------+",
        "| pgnode01 | 10.172.0.20 | Leader  | running   |  2 |           |",
        "| pgnode02 | 10.172.0.21 | Replica | streaming |  2 |         0 |",
        "| pgnode03 | 10.172.0.22 | Replica | streaming |  2 |         0 |",
        "+----------+-------------+---------+-----------+----+-----------+"
    ]
}

TASK [upgrade : Upgrade completed] *********************************************
ok: [10.172.0.20] => {
    "msg": [
        "PostgreSQL upgrade completed.",
        "Current version: 15.4 (Ubuntu 15.4-1.pgdg22.04+1)"
    ]
}

PLAY RECAP *********************************************************************
10.172.0.20                : ok=142  changed=52   unreachable=0    failed=0    skipped=107  rescued=0    ignored=0
10.172.0.21                : ok=83   changed=31   unreachable=0    failed=0    skipped=124  rescued=0    ignored=0
10.172.0.22                : ok=83   changed=31   unreachable=0    failed=0    skipped=124  rescued=0    ignored=0

passed

add upgrade role

c0d6efc

vitabaks marked this pull request as draft April 12, 2023 17:04

vitabaks self-assigned this Apr 12, 2023

vitabaks added the new feature label Apr 12, 2023

vitabaks mentioned this pull request Apr 12, 2023

Automate major version upgrades for PostgreSQL #162

Closed

vitabaks added 25 commits April 13, 2023 11:51

upgrade_secondary: update task name

1501115

fix no-jinja-when

97ed3a3

Update upgrade.yml

d22fce4

Update upgrade_postgres.yml

9861fbe

update TODO list

Update vars

71e3a92

Update update_config.yml

a6dd6e9

Update update_config.yml

8aecbdf

Update extensions in each database

5d4ca13

Update statistics.yml

3783bcf

Update upgrade_postgres.yml

7c31dee

Configure SSH Key-Based Authentication between cluster nodes

0f62894

required for upgrade replicas with rsync

Update ssh-keys.yml

8ecd3ad

Update ssh-keys.yml

3d6df52

Update upgrade_postgres.yml

e8e9e45

pre_checks: fix typo

250135d

Make sure that the openssh-client package is installed

e15bed5

Update upgrade_postgres.yml

ad28077

ssh-keys: add become_user

fa1d45a

Update ssh-keys.yml

df7ee0d

molecule: setup ssh service

21a56a5

Update converge.yml

690e497

Update converge.yml

20e3b9c

Update schema_compatibility.yml

368ca47

Update pre_checks.yml

2aae0b6

Update upgrade_check.yml

2ec51c3

vitabaks added 22 commits August 11, 2023 14:11

pgbouncer_pause: more info about terminated queries

804fee6

write to pgbouncer_pool_pause.log information about queries that were terminated (after 30 seconds of waiting).

Enhance playbook compatibility with multiple PostgreSQL versions

2b91495

- Add separate tasks for PostgreSQL 9.x and 10+ versions. - Implement additional pre-checks to ensure smooth operations. - Refactor tasks to improve readability and maintainability.

maintenance: Update haproxy task name

566dde9

Update rollback tasks name

d42b11c

Add run_once for task "Long-running transactions detected"

47b6f16

Install latest PostgreSQL version

32c8776

Add schema_compatibility_check_port variable

121ee9d

Update upgrade.yml

6e30953

Create README.md

b50826b

Update README.md

d744a24

README: Add step 5

57121bc

README: Add step 6 and 7

b0a4861

Update README.md

679c353

Add 'vacuumdb_parallel_jobs' variable

ae9b280

README: describe vacuumdb_parallel_jobs variable

eb4b030

Update pre_checks.yml

013bb7f

Update pg_upgrade.yml

9107750

Update README and tasks names

6611035

pre-checks: Make sure that the cluster ip address (VIP) is running

3d80ce2

README: Add recommendations

cba97d3

Add pg_terminator script to monitor locks and terminate the 'ANALYZE'…

ae10d01

… blockers

Update pg_terminator script query

351a735

vitabaks and others added 4 commits August 17, 2023 16:17

update the project's README

ae8f9b1

Describe Rollback

3991206

Move pg_upgrade test to a separate molecule scenario

f4865fb

fix [empty-lines]

4f41a7d

vitabaks merged commit 7d3d02b into master Aug 17, 2023
19 checks passed

vitabaks deleted the pg-upgrade branch December 26, 2023 09:03

vitabaks mentioned this pull request Dec 26, 2023

TODO: Implement Zero-Downtime Major Upgrades #537

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Automate in place major upgrade for PostgreSQL #301

Automate in place major upgrade for PostgreSQL #301

vitabaks commented Apr 12, 2023 •

edited

vitabaks commented Aug 17, 2023

Automate in place major upgrade for PostgreSQL #301

Automate in place major upgrade for PostgreSQL #301

Conversation

vitabaks commented Apr 12, 2023 • edited

Upgrade Plan and more details can be found here

vitabaks commented Aug 17, 2023

Test 7: final test before release

vitabaks commented Apr 12, 2023 •

edited