From d3c1d4f4568c2b85c7a05e186f032acdbdde3374 Mon Sep 17 00:00:00 2001 From: TJ Moore Date: Fri, 11 Oct 2019 17:43:01 -0400 Subject: [PATCH 1/3] Postgres Operator Upgrade updates An updated procedure for upgrading the Postgres Operator from version 4.0.1 to 4.1 is required to account for recent configuration updates and environmental changes. These changes left certain Operator features nonfunctional. The new upgrade procedure is designed to ensure all applicable changes are accounted for in the most convenient way possible. --- hugo/content/Upgrade/upgrade35to4.md | 8 +- hugo/content/Upgrade/upgrade40to41.md | 182 ++++++++------------------ 2 files changed, 61 insertions(+), 129 deletions(-) diff --git a/hugo/content/Upgrade/upgrade35to4.md b/hugo/content/Upgrade/upgrade35to4.md index da160d8bf9..1a52debbe2 100644 --- a/hugo/content/Upgrade/upgrade35to4.md +++ b/hugo/content/Upgrade/upgrade35to4.md @@ -1,13 +1,13 @@ --- -title: "Upgrade PGO 3.5 to 4" -Latest Release: 4.1.0 {docdate} +title: "Upgrade PGO 3.5 to 4.1" +Latest Release: 4.1 {docdate} draft: false weight: 8 --- -## Upgrading a Cluster from Version 3.5.x to PGO 4 +## Upgrading a Cluster from Version 3.5.x to PGO 4.1 -This section will outline the procedure to upgrade a given cluster created using Postgres Operator 3.5.x to PGO version 4.1. +This section will outline the procedure to upgrade a given cluster created using Postgres Operator 3.5.x to PGO version 4.1 {{% notice info %}} diff --git a/hugo/content/Upgrade/upgrade40to41.md b/hugo/content/Upgrade/upgrade40to41.md index 55ab1d8633..7cb563bb6c 100644 --- a/hugo/content/Upgrade/upgrade40to41.md +++ b/hugo/content/Upgrade/upgrade40to41.md @@ -20,36 +20,38 @@ You will need the following items to complete the upgrade: * The latest 4.1.0 code for the Postgres Operator available * The latest 4.1.0 PGO client binary -* Ensure you have a copy of the current pgouser and pgorole files for your environment and ensure you do not overwrite the current files. -Because user and role management has changed in 4.1.0, you will need to get these files if you do not want to manually add all existing users/roles into Kubernetes secrets. - -If you do not have access to the current files, you can pull the information via the configmap with the following command: - - kubectl describe configmap pgo-config -n - -With output similar to the default, shown below: -``` -pgorole: ----- -pgoadmin: Cat, Ls, ShowNamespace, CreateDump, RestoreDump, ScaleCluster, CreateSchedule, DeleteSchedule, ShowSchedule, DeletePgbouncer, CreatePgbouncer, DeletePgpool, CreatePgpool, Restore, RestorePgbasebackup, ShowSecrets, Reload, ShowConfig, Status, DfCluster, DeleteCluster, ShowCluster, CreateCluster, TestCluster, ShowBackup, DeleteBackup, CreateBackup, Label, Load, CreatePolicy, DeletePolicy, ShowPolicy, ApplyPolicy, ShowWorkflow, ShowPVC, CreateUpgrade, CreateUser, DeleteUser, User, Version, CreateFailover, UpdateCluster, CreateBenchmark, ShowBenchmark, DeleteBenchmark -pgoreader: Cat, Ls, ShowNamespace, Status, ShowConfig, DfCluster, ShowCluster, TestCluster, ShowBackup, ShowPolicy, ShowWorkflow, ShowPVC, Version, ShowSchedule, ShowBenchmark - -pgouser: ----- -username:password:pgoadmin: -readonlyuser:password:pgoreader: -``` Finally, these instructions assume you are executing from $PGOROOT in a terminal window and that your user has admin privileges in your Kubernetes or Openshift environment. ##### Step 0 You will most likely want to run: - + pgo show config -n Save this output to compare once the procedure has been completed to ensure none of the current configuration changes are missing. + ##### Step 1 +For the cluster(s) you wish to upgrade, scale down any replicas, if necessary, then delete the cluster + + pgo delete cluster + +{{% notice warning %}} + +Please note the name of each cluster, the namespace used, and be sure not to delete the associated PVCs or CRDs! + +{{% /notice %}} + + +##### Step 2 +Delete the 4.0.1 version of the Operator by executing: + + $PGOROOT/deploy/cleanup.sh + $PGOROOT/deploy/remove-crd.sh + $PGOROOT/deploy/cleanup-rbac.sh + + +##### Step 3 Update environment variables in the bashrc: export PGO_VERSION=4.1.0 @@ -64,7 +66,7 @@ export PGO_INSTALLATION_NAME=devtest # for setting the pgo apiserver port, disabling TLS or not verifying TLS # if TLS is disabled, ensure setip() function port is updated and http is used in place of https -export PGO_APISERVER_PORT=8443 # Defaults: 8443 for TLS enabled, 8080 for TLS disabled +export PGO_APISERVER_PORT=8443 # Defaults: 8443 for TLS enabled, 8080 for TLS disabled export DISABLE_TLS=false export TLS_NO_VERIFY=false @@ -81,31 +83,34 @@ Finally source the updated bash file: source ~/.bashrc -##### Step 2 -Update the pgo.yaml file in `$PGOROOT/conf/postgres-operator/pgo.yaml` +##### Step 4 + +Ensure you have checked out the latest 4.1.0 version of the source code and update the pgo.yaml file in `$PGOROOT/conf/postgres-operator/pgo.yaml` You will want to use the 4.1.0 pgo.yaml file and update custom settings such as image locations, storage, and resource configs. -##### Step 3 +##### Step 5 Create an initial Operator Admin user account. You will need to edit the `$PGOROOT/deploy/install-bootstrap-creds.sh` file to configure the username and password that you want for the Admin account. The default values are: ``` export PGOADMIN_USERNAME=pgoadmin export PGOADMIN_PASSWORD=examplepassword ``` -You will need to update the `$HOME/.pgouser`file to match the values you set in order to use the Operator. Once you create more accounts in a later step you can change this file to login in via the PGO CLIi as that user. +You will need to update the `$HOME/.pgouser`file to match the values you set in order to use the Operator. Additional accounts can be created later following the steps described in the 'Operator Security' section of the main [Bash Installation Guide] ( {{< relref "installation/operator-install.md" >}}). Once these accounts are created, you can change this file to login in via the PGO CLI as that user. + +##### Step 6 -You can verify the secrets are created by running: +Install the 4.1.0 Operator: - Kubectl get secrets -n +Setup the configured namespaces: -``` -pgorole-pgoadmin Opaque 2 19s -pgouser-pgoadmin Opaque 3 18s -``` + make setupnamespaces -##### Step 4 -Install the 4.1.0 Operator: +Install the RBAC configurations: + + make installrbac + +Deploy the Postgres Operator: make deployoperator @@ -113,9 +118,8 @@ Verify the Operator is running: kubectl get pod -n - -##### Step 5 -Update the PGO client binary to 4.1.0 by replacing the existing 4.0 binary with the latest 4.1.0 binary available. +##### Step 7 +Next, update the PGO client binary to 4.1.0 by replacing the existing 4.0 binary with the latest 4.1.0 binary available. You can run: @@ -124,107 +128,35 @@ You can run: to ensure you are replacing the current binary. -##### Step 6 +##### Step 8 You will want to make sure that any and all configuration changes have been updated. You can run: pgo show config -n -This will print out the current configuration that the Operator will be using. +This will print out the current configuration that the Operator will be using. To ensure that you made any required configuration changes, you can compare with Step 0 to make sure you did not miss anything. If you happened to miss a setting, update the pgo.yaml file and rerun: make deployoperator -##### Step 7 -Add the previous users and roles into Kubernetes as Secrets so that Operator 4.1 can use them using the following steps. - -We will be running `$PGOROOT/deploy/upgrade-creds.sh`, there are 2 ways to pass in the user and roles as described next. - -###### Option 1 -The original location in 4.0 for the pgouser and pgorole files was in the `$PGOROOT/conf/postgres-operator` folder. If you put the backed up files into that folder, the upgrade-creds script will find them automatically. - -###### Option 2 -However, if you prefer not to mix the 4.0 files into the 4.1 files you can pass in the files on the command line using the command below: - - $PGOROOT/deploy/upgrade-creds.sh /path/to/pgorole /path/to/pgouser - -After running the upgrade-creds script you can verify the users and roles have been created as Secrets by running: - - kubectl get secrets -n $PGO_OPERATOR_NAMESPACE - -##### Step 8 +##### Step 9 The Operator is now upgraded to 4.1.0 and all users and roles have been recreated. Verify this by running: pgo version +##### Step 10 +To verify cluster status, run + pgo test -n +Output should be similar to: +``` +psql -p 5432 -h 10.104.74.189 -U postgres postgres is Working +psql -p 5432 -h 10.104.74.189 -U postgres userdb is Working +psql -p 5432 -h 10.104.74.189 -U primaryuser postgres is Working +psql -p 5432 -h 10.104.74.189 -U primaryuser userdb is Working +psql -p 5432 -h 10.104.74.189 -U testuser postgres is Working +psql -p 5432 -h 10.104.74.189 -U testuser userdb is Working +``` +##### Step 11 +Scale up to the required number of replicas, as needed. -## Postgres Operator Container Upgrade Procedure - -At this point, the Operator will be running version 4.1.0, and new clusters will be built using the appropriate specifications defined in your pgo.yaml file. For the existing clusters, upgrades can be performed with the following steps. - -{{% notice info %}} - -Before beginning your upgrade procedure, be sure to consult the [Compatibility Requirements Page] -( {{< relref "configuration/compatibility.md" >}}) for container dependency information. - -{{% / notice %}} - -To start, execute the minor upgrade command: - - pgo upgrade cluster - -If you would like to know more about the Postgres Cluster minor upgrade, please see the [Upgrade Page] -( {{< relref "upgrade/_index.md" >}}) for additional information. - -By default this command updates the cluster with the values in the pgo.yaml. If however you are running more than one version of Postgres clusters you can run the following command to upgrade any clusters that do not match what is in your current configuration. - - pgo upgrade --ccp-image-tag= - -There is a bug in the operator where the image version for the backrest repo deployment is not updated with a pgo upgrade. As a workaround for this you need to redeploy the backrest shared repo deployment with the correct image version. - - kubectl get deployment -backrest-shared-repo -o yaml > -backrest-repo.yaml - -Edit the file - - -backrest-repo.yaml - -And set to the image (example below, please update to match your image repository details): - - crunchydata/container-suite/pgo-backrest-repo:centos7-4.1.0 - -Next you will need to delete the current backrest repo deployment and recreate it with the updated yaml: -``` -kubectl delete deployment -backrest-shared-repo -kubectl create -f -backrest-repo.yaml -``` - -Verify that the correct images are being used for the cluster. Run - - pgo show cluster -on your cluster and check the version. Describe each of the pods in your cluster and verify that the image that is being used is correct. - -``` -pgo show cluster -kubectl get pods -kubectl describe pod - -kubectl describe pod -backrest-shared-repo- -``` - -Finally, make sure that the correct version of pgbackrest is being used and verify backups are working. The versions of pgbackrest that are returned in the primary and backrest pods should match: -``` -kubectl get pods -kubectl exec -it - -- pgbackrest version -kubectl exec -it -backrest-shared-repo- -- pgbackrest version -pgo backup --backup-type=pgbackrest -``` - -You've now completed the upgrade and are running Crunchy PostgreSQL Operator v4.1.0, you can confirm this by running - - pgo version - -from the command line and running - - pgo show cluster - -on each cluster. For this minor upgrade, most existing settings and related services (such as pgbouncer, backup schedules and existing policies) are expected to work, but should be tested for functionality and adjusted or recreated as necessary. From 4c1fc1f4efd8b9047380a479f74e75e78c2e9300 Mon Sep 17 00:00:00 2001 From: TJ Moore Date: Tue, 15 Oct 2019 11:45:09 -0400 Subject: [PATCH 2/3] Updated the PGO 4.0.1 to 4.1.0 procedure to include a missing step outlining the process to recreate the 4.0.1 clusters in the new 4.1.0 environment. Also added a reference to the pgo scaledown command's help for use when needing to scale down the existing PGO 4.0.1 pgclusters. --- hugo/content/Upgrade/upgrade40to41.md | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/hugo/content/Upgrade/upgrade40to41.md b/hugo/content/Upgrade/upgrade40to41.md index 7cb563bb6c..28fe6aef0f 100644 --- a/hugo/content/Upgrade/upgrade40to41.md +++ b/hugo/content/Upgrade/upgrade40to41.md @@ -32,9 +32,9 @@ Save this output to compare once the procedure has been completed to ensure none ##### Step 1 -For the cluster(s) you wish to upgrade, scale down any replicas, if necessary, then delete the cluster +For the cluster(s) you wish to upgrade, scale down any replicas, if necessary (see `pgo scaledown --help` for more information on command usage) page for more information), then delete the cluster - pgo delete cluster + pgo delete cluster {{% notice warning %}} @@ -146,6 +146,11 @@ Verify this by running: pgo version ##### Step 10 +Once the Operator is installed and functional, create new 4.1 clusters with the same name as was used previously. This will allow the new clusters to utilize the existing PVCs. + + pgo create cluster -n + +##### Step 11 To verify cluster status, run pgo test -n Output should be similar to: @@ -157,6 +162,6 @@ psql -p 5432 -h 10.104.74.189 -U primaryuser userdb is Working psql -p 5432 -h 10.104.74.189 -U testuser postgres is Working psql -p 5432 -h 10.104.74.189 -U testuser userdb is Working ``` -##### Step 11 +##### Step 12 Scale up to the required number of replicas, as needed. From 2693d80e248a3281880da8fca34539ae67a7fcb9 Mon Sep 17 00:00:00 2001 From: TJ Moore Date: Tue, 15 Oct 2019 15:11:41 -0400 Subject: [PATCH 3/3] Added a separate upgrade procedure for users using the Ansible installation method for their Operator installation. Procedure is designed to be inclusive. Other updates reflect renaming of the existing procedure to specify it is for Bash installations, and an update to relevant links. --- hugo/content/Upgrade/_index.md | 6 +- hugo/content/Upgrade/upgrade40to41_ansible.md | 80 +++++++++++++++++++ ...upgrade40to41.md => upgrade40to41_bash.md} | 6 +- 3 files changed, 87 insertions(+), 5 deletions(-) create mode 100644 hugo/content/Upgrade/upgrade40to41_ansible.md rename hugo/content/Upgrade/{upgrade40to41.md => upgrade40to41_bash.md} (97%) diff --git a/hugo/content/Upgrade/_index.md b/hugo/content/Upgrade/_index.md index 4ee19523c4..df08e5af19 100644 --- a/hugo/content/Upgrade/_index.md +++ b/hugo/content/Upgrade/_index.md @@ -16,7 +16,9 @@ This section of the documentation shows specific steps required to upgrade diffe [Upgrade Postgres Operator from 3.5 to 4.1] ( {{< relref "upgrade/upgrade35to4.md" >}}) -[Upgrade Postgres Operator from 4.0.1 to 4.1.0] ( {{< relref "upgrade/upgrade40to41.md" >}}) +[Upgrade Postgres Operator from 4.0.1 to 4.1.0 (Bash)] ( {{< relref "upgrade/upgrade40to41_bash.md" >}}) + +[Upgrade Postgres Operator from 4.0.1 to 4.1.0 (Ansible)] ( {{< relref "upgrade/upgrade40to41_ansible.md" >}}) ## Upgrading A Postgres Cluster @@ -36,4 +38,4 @@ In this example, we are upgrading a cluster from PostgreSQL 11.4 to 11.5 using t `pgo upgrade mycluster --ccp-image-tag=centos7-11.5-2.4.2` -For more information, please see the `pgo upgrade` documentation [here.] ( {{< relref "operatorcli/cli/pgo_upgrade.md" >}}) \ No newline at end of file +For more information, please see the `pgo upgrade` documentation [here.] ( {{< relref "operatorcli/cli/pgo_upgrade.md" >}}) diff --git a/hugo/content/Upgrade/upgrade40to41_ansible.md b/hugo/content/Upgrade/upgrade40to41_ansible.md new file mode 100644 index 0000000000..a22bcd5e15 --- /dev/null +++ b/hugo/content/Upgrade/upgrade40to41_ansible.md @@ -0,0 +1,80 @@ +--- +title: "Upgrade PGO 4.0.1 to 4.1.0 (Ansible)" +Latest Release: 4.1.0 {docdate} +draft: false +weight: 8 +--- + +## Postgres Operator Ansible Upgrade Procedure from 4.0.1 to 4.1.0 + +This procedure will give instructions on how to upgrade to Postgres Operator 4.1.0 when using the Ansible installation method. + +{{% notice info %}} + +As with any upgrade, please ensure you have taken recent backups of all relevant data! + +{{% / notice %}} + +##### Prerequisites. +You will need the following items to complete the upgrade: + +* The latest 4.1.0 code for the Postgres Operator available + +These instructions assume you are executing in a terminal window and that your user has admin privileges in your Kubernetes or Openshift environment. + + +##### Step 0 +For the cluster(s) you wish to upgrade, scale down any replicas, if necessary (see `pgo scaledown --help` for more information on command usage) page for more information), then delete the cluster + + pgo delete cluster + +{{% notice warning %}} + +Please note the name of each cluster, the namespace used, and be sure not to delete the associated PVCs or CRDs! + +{{% /notice %}} + + +##### Step 1 + +Save a copy of your current inventory file with a new name (such as `inventory.backup)` and checkout the latest 4.1 tag of the Postgres Operator. + + +##### Step 2 +Update the new inventory file with the appropriate values for your new Operator installation, as described in the [Ansible Install Prerequisites] ( {{< relref "installation/install-with-ansible/prerequisites.md" >}}) and the [Compatibility Requirements Guide] ( {{< relref "configuration/compatibility.md" >}}). + + +##### Step 3 + +Now you can upgrade your Operator installation and configure your connection settings as described in the [Ansible Update Page] ( {{< relref "installation/install-with-ansible/updating-operator.md" >}}). + + +##### Step 4 +Verify the Operator is running: + + kubectl get pod -n + +And that it is upgraded to the appropriate version + + pgo version + +##### Step 5 +Once the Operator is installed and functional, create new 4.1 clusters with the same name as was used previously. This will allow the new clusters to utilize the existing PVCs. + + pgo create cluster -n + +##### Step 6 +To verify cluster status, run + pgo test -n +Output should be similar to: +``` +psql -p 5432 -h 10.104.74.189 -U postgres postgres is Working +psql -p 5432 -h 10.104.74.189 -U postgres userdb is Working +psql -p 5432 -h 10.104.74.189 -U primaryuser postgres is Working +psql -p 5432 -h 10.104.74.189 -U primaryuser userdb is Working +psql -p 5432 -h 10.104.74.189 -U testuser postgres is Working +psql -p 5432 -h 10.104.74.189 -U testuser userdb is Working +``` +##### Step 7 +Scale up to the required number of replicas, as needed. + diff --git a/hugo/content/Upgrade/upgrade40to41.md b/hugo/content/Upgrade/upgrade40to41_bash.md similarity index 97% rename from hugo/content/Upgrade/upgrade40to41.md rename to hugo/content/Upgrade/upgrade40to41_bash.md index 28fe6aef0f..3d264abff8 100644 --- a/hugo/content/Upgrade/upgrade40to41.md +++ b/hugo/content/Upgrade/upgrade40to41_bash.md @@ -1,13 +1,13 @@ --- -title: "Upgrade PGO 4.0.1 to 4.1.0" +title: "Upgrade PGO 4.0.1 to 4.1.0 (Bash)" Latest Release: 4.1.0 {docdate} draft: false weight: 8 --- -## Postgres Operator Upgrade Procedure from 4.0.1 to 4.1.0 +## Postgres Operator Bash Upgrade Procedure from 4.0.1 to 4.1.0 -This procedure will give instructions on how to upgrade to Postgres Operator 4.1.0. +This procedure will give instructions on how to upgrade to Postgres Operator 4.1.0 when using the Bash installation method. {{% notice info %}}