fix postgresql 13 recovery #4521

openoms · 2024-04-04T23:52:35Z

openoms · 2024-04-05T11:54:41Z

Testing the image build from this branch: https://github.com/raspiblitz/raspiblitz/actions/runs/8567463654

still not looking good when recovering on an RPi5 loaded with a pg13 on disk

pg_lsclusters 
Ver Cluster Port Status Owner     Data directory              Log file
13  main    5433 online postgres  /var/lib/postgresql/13/main /var/log/postgresql/postgresql-13-main.log
15  main    5432 down   <unknown> /var/lib/postgresql/15/main /var/log/postgresql/postgresql-15-main.log

₿ sudo -u postgres psql -l
psql: error: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory
        Is the server running locally and accepting connections on that socket?

# ACTIVATE BTCPAYSERVER                                                                                                                                                             
# install postreSQL                                                                                                                                                                 
# Using the default PostgreSQL version: 15                                                                                                                                          
# Install PostgreSQL                                                                                                                                                                
deb http://apt.postgresql.org/pub/repos/apt/ bookworm-pgdg main                                                                                                                     
                                                                                                                                                                                    
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.                                                                                                     
                                                                                                                                                                                    
Hit:1 http://deb.debian.org/debian bookworm InRelease                                                                                                                               
Hit:2 http://deb.debian.org/debian-security bookworm-security InRelease                                                                                                             
Hit:3 http://deb.debian.org/debian bookworm-updates InRelease                                                                                                                       
Hit:4 https://repo.i2pd.xyz/debian bookworm InRelease                                                                                                                               
Hit:5 http://archive.raspberrypi.com/debian bookworm InRelease                                                                                                                      
Get:6 http://apt.postgresql.org/pub/repos/apt bookworm-pgdg InRelease [123 kB]                                                                                                      
Get:7 http://apt.postgresql.org/pub/repos/apt bookworm-pgdg/main arm64 Packages [300 kB]                                                                                            
Hit:8 tor+http://apow7mjfryruh65chtdydfmqfpj5btws7nbocgtaovhvezgccyjazpqd.onion/torproject.org bookworm InRelease                                                                   
Fetched 423 kB in 11s (39.8 kB/s)                                                                                                                                                   
Reading package lists...                                                                                                                                                            
Building dependency tree...                                                                                                                                                         
Reading state information...                                                                                                                                                        
4 packages can be upgraded. Run 'apt list --upgradable' to see them.                                                                                                                
                                                                                                                                                                                    
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.                                                                                                     
                                                                                                                                                                                    
Reading package lists...                                                                                                                                                            
Building dependency tree...                                                                                                                                                         
Reading state information...                                                                                                                                                        
The following package was automatically installed and is no longer required:                                                                                                        
  pi-language-support                                                                                                                                                               
Use 'sudo apt autoremove' to remove it.                                                                                                                                             
The following additional packages will be installed:                                                                                                                                
  libcommon-sense-perl libjson-perl libjson-xs-perl libllvm16                                                                                                                       
  libtypes-serialiser-perl postgresql-client-15 postgresql-client-common                                                                                                            
  postgresql-common ssl-cert sysstat                                                                                                                                                
Suggested packages:                                                                                                                                                                 
  postgresql-doc-15 isag                                                                                                                                                            
The following NEW packages will be installed:                                                                                                                                       
  libcommon-sense-perl libjson-perl libjson-xs-perl libllvm16                                                                                                                       
  libtypes-serialiser-perl postgresql-15 postgresql-client-15                                                                                                                       
  postgresql-client-common postgresql-common ssl-cert sysstat                                                                                                                       
0 upgraded, 11 newly installed, 0 to remove and 4 not upgraded.                                                                                                                     
Need to get 39.6 MB of archives.                                                                                                                                                    
After this operation, 186 MB of additional disk space will be used.                                                                                                                 
Get:1 http://deb.debian.org/debian bookworm/main arm64 libjson-perl all 4.10000-1 [87.5 kB]                                                                                         
Get:2 http://deb.debian.org/debian bookworm/main arm64 ssl-cert all 1.1.2 [21.1 kB]                                                                                                 
Get:3 http://deb.debian.org/debian bookworm/main arm64 libcommon-sense-perl arm64 3.75-3 [23.0 kB]                                                                                  
Get:4 http://deb.debian.org/debian bookworm/main arm64 libtypes-serialiser-perl all 1.01-1 [12.2 kB]                                                                                
Get:5 http://deb.debian.org/debian bookworm/main arm64 libjson-xs-perl arm64 4.030-2+b1 [90.7 kB] 

Get:6 http://deb.debian.org/debian bookworm/main arm64 libllvm16 arm64 1:16.0.6-15~deb12u1 [20.6 MB]                                                                                
Get:7 http://apt.postgresql.org/pub/repos/apt bookworm-pgdg/main arm64 postgresql-client-common all 257.pgdg120+1 [36.7 kB]                                                         
Get:8 http://apt.postgresql.org/pub/repos/apt bookworm-pgdg/main arm64 postgresql-common all 257.pgdg120+1 [182 kB]                                                                 
Get:9 http://apt.postgresql.org/pub/repos/apt bookworm-pgdg/main arm64 postgresql-client-15 arm64 15.6-1.pgdg120+2 [1,649 kB]                                                       
Get:10 http://apt.postgresql.org/pub/repos/apt bookworm-pgdg/main arm64 postgresql-15 arm64 15.6-1.pgdg120+2 [16.2 MB]                                                              
Get:11 http://deb.debian.org/debian bookworm/main arm64 sysstat arm64 12.6.1-1 [570 kB]                                                                                             
Preconfiguring packages ...                                                                                                                                                         
Fetched 39.6 MB in 1s (27.5 MB/s)                                                                                                                                                   
Selecting previously unselected package libjson-perl.                                                                                                                               
(Reading database ... 128478 files and directories currently installed.)                                                                                                            
Preparing to unpack .../00-libjson-perl_4.10000-1_all.deb ...                                                                                                                       
Unpacking libjson-perl (4.10000-1) ...                                                                                                                                              
Selecting previously unselected package postgresql-client-common.                                                                                                                   
Preparing to unpack .../01-postgresql-client-common_257.pgdg120+1_all.deb ...                                                                                                       
Unpacking postgresql-client-common (257.pgdg120+1) ...                                                                                                                              
Selecting previously unselected package ssl-cert.                                                                                                                                   
Preparing to unpack .../02-ssl-cert_1.1.2_all.deb ...                                                                                                                               
Unpacking ssl-cert (1.1.2) ...                                                                                                                                                      
Selecting previously unselected package postgresql-common.                                                                                                                          
Preparing to unpack .../03-postgresql-common_257.pgdg120+1_all.deb ...                                                                                                              
Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common'                                                                                       
Unpacking postgresql-common (257.pgdg120+1) ...                                                                                                                                     
Selecting previously unselected package libcommon-sense-perl:arm64.                                                                                                                 
Preparing to unpack .../04-libcommon-sense-perl_3.75-3_arm64.deb ...                                                                                                                
Unpacking libcommon-sense-perl:arm64 (3.75-3) ...                                                                                                                                   
Selecting previously unselected package libtypes-serialiser-perl.                                                                                                                   
Preparing to unpack .../05-libtypes-serialiser-perl_1.01-1_all.deb ...                                                                                                              
Unpacking libtypes-serialiser-perl (1.01-1) ...                                                                                                                                     
Selecting previously unselected package libjson-xs-perl.                                                                                                                            
Preparing to unpack .../06-libjson-xs-perl_4.030-2+b1_arm64.deb ...                                                                                                                 
Unpacking libjson-xs-perl (4.030-2+b1) ...                                                                                                                                          
Selecting previously unselected package libllvm16:arm64.                                                                                                                            
Preparing to unpack .../07-libllvm16_1%3a16.0.6-15~deb12u1_arm64.deb ...                                                                                                            
Unpacking libllvm16:arm64 (1:16.0.6-15~deb12u1) ...                                                                                                                                 
Selecting previously unselected package postgresql-client-15.                                                                                                                       
Preparing to unpack .../08-postgresql-client-15_15.6-1.pgdg120+2_arm64.deb ...                                                                                                      
Unpacking postgresql-client-15 (15.6-1.pgdg120+2) ...                                                                                                                               
Selecting previously unselected package postgresql-15.                                                                                                                              
Preparing to unpack .../09-postgresql-15_15.6-1.pgdg120+2_arm64.deb ...                                                                                                             
Unpacking postgresql-15 (15.6-1.pgdg120+2) ...                                                                                                                                      
Selecting previously unselected package sysstat.                                                                                                                                    
Preparing to unpack .../10-sysstat_12.6.1-1_arm64.deb ...                                                                                                                           
Unpacking sysstat (12.6.1-1) ...                                                                                                                                                    
Setting up postgresql-client-common (257.pgdg120+1) ...                                                                                                                             
Setting up libcommon-sense-perl:arm64 (3.75-3) ...                                                                                                                                  
Setting up postgresql-client-15 (15.6-1.pgdg120+2) ...                                                                                                                              
update-alternatives: using /usr/share/postgresql/15/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode                                            
Setting up ssl-cert (1.1.2) ...                                                                                                                                                     
Setting up libtypes-serialiser-perl (1.01-1) ...                                                                                                                                    
Setting up libjson-perl (4.10000-1) ...                                                                                                                                             
Setting up libllvm16:arm64 (1:16.0.6-15~deb12u1) ...                                                                                                                                
Setting up sysstat (12.6.1-1) ...                                                                                                                                                   
                                                                                                                                                                                    
Creating config file /etc/default/sysstat with new version                                                                                                                          
update-alternatives: using /usr/bin/sar.sysstat to provide /usr/bin/sar (sar) in auto mode                                                                                          
Created symlink /etc/systemd/system/sysstat.service.wants/sysstat-collect.timer → /lib/systemd/system/sysstat-collect.timer.                                                        
Created symlink /etc/systemd/system/sysstat.service.wants/sysstat-summary.timer → /lib/systemd/system/sysstat-summary.timer.                                                        
Created symlink /etc/systemd/system/multi-user.target.wants/sysstat.service → /lib/systemd/system/sysstat.service.                                                                  
Setting up libjson-xs-perl (4.030-2+b1) ...                                                                                                                                         
Setting up postgresql-common (257.pgdg120+1) ...                                                                                                                                    
                                                                                                                                                                                    
Creating config file /etc/postgresql-common/createcluster.conf with new version                                                                                                     
Building PostgreSQL dictionaries from installed myspell/hunspell packages...                                                                                                        
  en_gb                                                                                                                                                                             
  en_us                                                                                                                                                                             
Removing obsolete dictionary files:                                                                                                                                                 
'/etc/apt/trusted.gpg.d/apt.postgresql.org.gpg' -> '/usr/share/postgresql-common/pgdg/apt.postgresql.org.gpg'   


Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service → /lib/systemd/system/postgresql.service.                                                            
Setting up postgresql-15 (15.6-1.pgdg120+2) ...                                                                                                                                     
Creating new PostgreSQL cluster 15/main ...                                                                                                                                         
/usr/lib/postgresql/15/bin/initdb -D /var/lib/postgresql/15/main --auth-local peer --auth-host scram-sha-256 --no-instructions                                                      
The files belonging to this database system will be owned by user "postgres".                                                                                                       
This user must also own the server process.                                                                                                                                         
                                                                                                                                                                                    
The database cluster will be initialized with locale "en_GB.UTF-8".                                                                                                                 
The default database encoding has accordingly been set to "UTF8".                                                                                                                   
The default text search configuration will be set to "english".                                                                                                                     
                                                                                                                                                                                    
Data page checksums are disabled.                                                                                                                                                   
                                                                                                                                                                                    
fixing permissions on existing directory /var/lib/postgresql/15/main ... ok                                                                                                         
creating subdirectories ... ok                                                                                                                                                      
selecting dynamic shared memory implementation ... posix                                                                                                                            
selecting default max_connections ... 100                                                                                                                                           
selecting default shared_buffers ... 128MB                                                                                                                                          
selecting default time zone ... Europe/London                                                                                                                                       
creating configuration files ... ok                                                                                                                                                 
running bootstrap script ... ok                                                                                                                                                     
performing post-bootstrap initialization ... ok                                                                                                                                     
syncing data to disk ... ok                                                                                                                                                         
Processing triggers for man-db (2.11.2-2) ...                                                                                                                                       
Processing triggers for libc-bin (2.36-9+rpt2+deb12u4) ...                                                                                                                          
# There is old data for pg 13, start and upgrade cluster ...                                                                                                                        
                                                                                                                                                                                    
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.                                                                                                     
                                                                                                                                                                                    
Reading package lists...                                                                                                                                                            
Building dependency tree...                                                                                                                                                         
Reading state information...                                                                                                                                                        
The following package was automatically installed and is no longer required:                                                                                                        
  pi-language-support                                                                                                                                                               
Use 'sudo apt autoremove' to remove it.                                                                                                                                             
The following additional packages will be installed:                                                                                                                                
  postgresql-client-13                                                                                                                                                              
Suggested packages:                                                                                                                                                                 
  postgresql-doc-13                                                                                                                                                                 
The following NEW packages will be installed:                                                                                                                                       
  postgresql-13 postgresql-client-13                                                                                                                                                
0 upgraded, 2 newly installed, 0 to remove and 4 not upgraded.                                                                                                                      
Need to get 16.5 MB of archives.                                                                                                                                                    
After this operation, 59.5 MB of additional disk space will be used.                                                                                                                
Get:1 http://apt.postgresql.org/pub/repos/apt bookworm-pgdg/main arm64 postgresql-client-13 arm64 13.14-1.pgdg120+2 [1,474 kB]                                                      
Get:2 http://apt.postgresql.org/pub/repos/apt bookworm-pgdg/main arm64 postgresql-13 arm64 13.14-1.pgdg120+2 [15.0 MB]                                                              
Preconfiguring packages ...                                                                                                                                                         
Fetched 16.5 MB in 1s (19.6 MB/s)                                                                                                                                                   
Selecting previously unselected package postgresql-client-13.                                                                                                                       
(Reading database ... 130784 files and directories currently installed.)                                                                                                            
Preparing to unpack .../postgresql-client-13_13.14-1.pgdg120+2_arm64.deb ...                                                                                                        
Unpacking postgresql-client-13 (13.14-1.pgdg120+2) ...                                                                                                                              
Selecting previously unselected package postgresql-13.                                                                                                                              
Preparing to unpack .../postgresql-13_13.14-1.pgdg120+2_arm64.deb ...                                                                                                               
Unpacking postgresql-13 (13.14-1.pgdg120+2) ...                                                                                                                                     
Setting up postgresql-client-13 (13.14-1.pgdg120+2) ...                                                                                                                             
Setting up postgresql-13 (13.14-1.pgdg120+2) ...                                                                                                                                    
Creating new PostgreSQL cluster 13/main ...                                                                                                                                         
/usr/lib/postgresql/13/bin/initdb -D /var/lib/postgresql/13/main --auth-local peer --auth-host md5                                                                                  
The files belonging to this database system will be owned by user "postgres".                                                                                                       
This user must also own the server process.                                                                                                                                         
                                                                                                                                                                                    
The database cluster will be initialized with locale "en_GB.UTF-8".                                                                                                                 
The default database encoding has accordingly been set to "UTF8".                                                                                                                   
The default text search configuration will be set to "english".                                                                                                                     
                                                                                                                                                                                    
Data page checksums are disabled.   

fixing permissions on existing directory /var/lib/postgresql/13/main ... ok                                                                                                         
creating subdirectories ... ok                                                                                                                                                      
selecting dynamic shared memory implementation ... posix                                                                                                                            
selecting default max_connections ... 100                                                                                                                                           
selecting default shared_buffers ... 128MB                                                                                                                                          
selecting default time zone ... Europe/London                                                                                                                                       
creating configuration files ... ok                                                                                                                                                 
running bootstrap script ... ok                                                                                                                                                     
performing post-bootstrap initialization ... ok                                                                                                                                     
syncing data to disk ... ok                                                                                                                                                         
                                                                                                                                                                                    
Success. You can now start the database server using:                                                                                                                               
                                                                                                                                                                                    
    pg_ctlcluster 13 main start                                                                                                                                                     
                                                                                                                                                                                    
Processing triggers for postgresql-common (257.pgdg120+1) ...                                                                                                                       
Building PostgreSQL dictionaries from installed myspell/hunspell packages...                                                                                                        
  en_gb                                                                                                                                                                             
  en_us                                                                                                                                                                             
Removing obsolete dictionary files:                                                                                                                                                 
# Create pg 13 config                                                                                                                                                               
Error: cluster configuration already exists                                                                                                                                         
Setting default password for postgres user                                                                                                                                          
could not change directory to "/root": Permission denied                                                                                                                            
ALTER ROLE                                                                                                                                                                          
Error: cluster configuration already exists                                                                                                                                         
Error: target cluster 15/main already exists                                                                                                                                        
Provisioning LND Port

sudo systemctl status postgresql@15-main
× postgresql@15-main.service - PostgreSQL Cluster 15-main
     Loaded: loaded (/lib/systemd/system/postgresql@.service; enabled-runtime; preset: enabled)
     Active: failed (Result: protocol) since Fri 2024-04-05 13:10:39 BST; 4h 27min ago
   Duration: 257ms
        CPU: 42ms

Apr 05 13:10:38 raspberrypi systemd[1]: Starting postgresql@15-main.service - PostgreSQL Cluster 15-main...
Apr 05 13:10:39 raspberrypi postgresql@15-main[227131]: Error: /var/lib/postgresql/15/main is not accessible or does not exist
Apr 05 13:10:39 raspberrypi systemd[1]: postgresql@15-main.service: Can't open PID file /run/postgresql/15-main.pid (yet?) after start: No such file or directory
Apr 05 13:10:39 raspberrypi systemd[1]: postgresql@15-main.service: Failed with result 'protocol'.
Apr 05 13:10:39 raspberrypi systemd[1]: Failed to start postgresql@15-main.service - PostgreSQL Cluster 15-main.

rootzoll · 2024-04-05T13:11:13Z

still not looking good when recovering on an RPi5 loaded with a pg13 on disk

so this is still a showstopper for v1.11.0 - correct?

openoms · 2024-04-05T14:33:29Z

still not looking good when recovering on an RPi5 loaded with a pg13 on disk

so this is still a showstopper for v1.11.0 - correct?

yes it seems that's still the case. Will need to clear all clusters before recovering I think, but will need to test more.

openoms · 2024-04-05T17:27:34Z

Set to test the new image from this branch:
https://github.com/raspiblitz/raspiblitz/actions/runs/8573507665

openoms · 2024-04-05T19:08:18Z

recovered from pg13 was ok on the last one so looking good.

next image with a small tweak now: https://github.com/raspiblitz/raspiblitz/actions/runs/8574541229

manual testing is still needed as cannot reproduce the process fully in bats

openoms · 2024-04-06T08:18:51Z

Looking good now, merging and cherry-pick to v1.11

₿ pg_lsclusters
Ver Cluster Port Status Owner    Data directory              Log file
15  main    5432 online postgres /var/lib/postgresql/15/main /var/log/postgresql/postgresql-15-main.log
₿ sudo -u postgres psql -l
                                                      List of databases
       Name       |  Owner   | Encoding |   Collate   |    Ctype    | ICU Locale | Locale Provider |    Access privileges    
------------------+----------+----------+-------------+-------------+------------+-----------------+-------------------------
 btcpaymainnet    | postgres | UTF8     | C           | C           |            | libc            | =Tc/postgres           +
                  |          |          |             |             |            |                 | postgres=CTc/postgres  +
                  |          |          |             |             |            |                 | btcpay=CTc/postgres
 nbxplorermainnet | postgres | UTF8     | C           | C           |            | libc            | =Tc/postgres           +
                  |          |          |             |             |            |                 | postgres=CTc/postgres  +
                  |          |          |             |             |            |                 | nbxplorer=CTc/postgres
 postgres         | postgres | UTF8     | en_GB.UTF-8 | en_GB.UTF-8 |            | libc            | 
 template0        | postgres | UTF8     | en_GB.UTF-8 | en_GB.UTF-8 |            | libc            | =c/postgres            +
                  |          |          |             |             |            |                 | postgres=CTc/postgres
 template1        | postgres | UTF8     | en_GB.UTF-8 | en_GB.UTF-8 |            | libc            | =c/postgres            +
                  |          |          |             |             |            |                 | postgres=CTc/postgres
 testdb13         | postgres | UTF8     | C           | C           |            | libc            | =Tc/postgres           +
                  |          |          |             |             |            |                 | postgres=CTc/postgres  +
                  |          |          |             |             |            |                 | testuser13=CTc/postgres
(6 rows)

# ACTIVATE BTCPAYSERVER                                                                                                                                                             
# install postreSQL                                                                                                                                                                 
# Using the default PostgreSQL version: 15                                                                                                                                          
# Install PostgreSQL                                                                                                                                                                
deb http://apt.postgresql.org/pub/repos/apt/ bookworm-pgdg main                                                                                                                     
                                                                                                                                                                                    
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.                                                                                                     
                                                                                                                                                                                    
Hit:1 http://archive.raspberrypi.com/debian bookworm InRelease                                                                                                                      
Hit:2 http://deb.debian.org/debian bookworm InRelease                                                                                                                               
Get:3 http://apt.postgresql.org/pub/repos/apt bookworm-pgdg InRelease [123 kB]                                                                                                      
Hit:4 http://deb.debian.org/debian-security bookworm-security InRelease                                                                                                             
Hit:5 http://deb.debian.org/debian bookworm-updates InRelease                                                                                                                       
Hit:6 https://repo.i2pd.xyz/debian bookworm InRelease                                     
Get:7 http://apt.postgresql.org/pub/repos/apt bookworm-pgdg/main arm64 Packages [300 kB]                                                                                            
Hit:8 tor+http://apow7mjfryruh65chtdydfmqfpj5btws7nbocgtaovhvezgccyjazpqd.onion/torproject.org bookworm InRelease                                                                   
Fetched 423 kB in 4s (110 kB/s)                                                           
Reading package lists...                                                                  
Building dependency tree...                                                               
Reading state information...                                                              
4 packages can be upgraded. Run 'apt list --upgradable' to see them.                                                                                                                
                                                                                          
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.                                                                                                     
                                                                                                                                                                                    
Reading package lists...                                                                                                                                                            
Building dependency tree...                                                                                                                                                         
Reading state information...                                                                                                                                                        
The following package was automatically installed and is no longer required:                                                                                                        
  pi-language-support                                                                     
Use 'sudo apt autoremove' to remove it.                                                   
The following additional packages will be installed:                                      
  libcommon-sense-perl libjson-perl libjson-xs-perl libllvm16                                                                                                                       
  libtypes-serialiser-perl postgresql-client-15 postgresql-client-common                  
  postgresql-common ssl-cert sysstat                                                      
Suggested packages:                                                                       
  postgresql-doc-15 isag                                                                                                                                                            
The following NEW packages will be installed:                                                                                                                                       
  libcommon-sense-perl libjson-perl libjson-xs-perl libllvm16                                                                                                                       
  libtypes-serialiser-perl postgresql-15 postgresql-client-15                                                                                                                       
  postgresql-client-common postgresql-common ssl-cert sysstat                                                                                                                       
0 upgraded, 11 newly installed, 0 to remove and 4 not upgraded.                                                                                                                     
Need to get 39.6 MB of archives.                                                                                                                                                    
After this operation, 186 MB of additional disk space will be used.                       
Get:1 http://apt.postgresql.org/pub/repos/apt bookworm-pgdg/main arm64 postgresql-client-common all 257.pgdg120+1 [36.7 kB]                                                         
Get:2 http://deb.debian.org/debian bookworm/main arm64 libjson-perl all 4.10000-1 [87.5 kB]                                                                                         
Get:3 http://apt.postgresql.org/pub/repos/apt bookworm-pgdg/main arm64 postgresql-common all 257.pgdg120+1 [182 kB]                                                                 
Get:4 http://deb.debian.org/debian bookworm/main arm64 ssl-cert all 1.1.2 [21.1 kB]                                                                                                 
Get:5 http://deb.debian.org/debian bookworm/main arm64 libcommon-sense-perl arm64 3.75-3 [23.0 kB]                                                                                  
Get:6 http://deb.debian.org/debian bookworm/main arm64 libtypes-serialiser-perl all 1.01-1 [12.2 kB]                                                                                
Get:7 http://deb.debian.org/debian bookworm/main arm64 libjson-xs-perl arm64 4.030-2+b1 [90.7 kB]
Get:8 http://deb.debian.org/debian bookworm/main arm64 libllvm16 arm64 1:16.0.6-15~deb12u1 [20.6 MB]                                                                                
Get:9 http://apt.postgresql.org/pub/repos/apt bookworm-pgdg/main arm64 postgresql-client-15 arm64 15.6-1.pgdg120+2 [1,649 kB] 
Get:10 http://apt.postgresql.org/pub/repos/apt bookworm-pgdg/main arm64 postgresql-15 arm64 15.6-1.pgdg120+2 [16.2 MB]                                                              
Get:11 http://deb.debian.org/debian bookworm/main arm64 sysstat arm64 12.6.1-1 [570 kB]                                                                                             
Preconfiguring packages ...                                                               
Fetched 39.6 MB in 2s (22.8 MB/s)                                                                                                                                                   
Selecting previously unselected package libjson-perl.                                                                                                                               
(Reading database ... 128478 files and directories currently installed.)                                                                                                            
Preparing to unpack .../00-libjson-perl_4.10000-1_all.deb ...                                                                                                                       
Unpacking libjson-perl (4.10000-1) ...                                                                                                                                              
Selecting previously unselected package postgresql-client-common.                                                                                                                   
Preparing to unpack .../01-postgresql-client-common_257.pgdg120+1_all.deb ...             
Unpacking postgresql-client-common (257.pgdg120+1) ...                                    
Selecting previously unselected package ssl-cert. 

Preparing to unpack .../02-ssl-cert_1.1.2_all.deb ...                                                                                                                               
Unpacking ssl-cert (1.1.2) ...                                                                                                                                                      
Selecting previously unselected package postgresql-common.                                                                                                                          
Preparing to unpack .../03-postgresql-common_257.pgdg120+1_all.deb ...                                                                                                              
Adding 'diversion of /usr/bin/pg_config to /usr/bin/pg_config.libpq-dev by postgresql-common'                                                                                       
Unpacking postgresql-common (257.pgdg120+1) ...                                                                                                                                     
Selecting previously unselected package libcommon-sense-perl:arm64.                                                                                                                 
Preparing to unpack .../04-libcommon-sense-perl_3.75-3_arm64.deb ...                                                                                                                
Unpacking libcommon-sense-perl:arm64 (3.75-3) ...                                                                                                                                   
Selecting previously unselected package libtypes-serialiser-perl.                                                                                                                   
Preparing to unpack .../05-libtypes-serialiser-perl_1.01-1_all.deb ...                                                                                                              
Unpacking libtypes-serialiser-perl (1.01-1) ...                                                                                                                                     
Selecting previously unselected package libjson-xs-perl.                                                                                                                            
Preparing to unpack .../06-libjson-xs-perl_4.030-2+b1_arm64.deb ...                                                                                                                 
Unpacking libjson-xs-perl (4.030-2+b1) ...                                                                                                                                          
Selecting previously unselected package libllvm16:arm64.                                                                                                                            
Preparing to unpack .../07-libllvm16_1%3a16.0.6-15~deb12u1_arm64.deb ...                  
Unpacking libllvm16:arm64 (1:16.0.6-15~deb12u1) ...                                                                                                                                 
Selecting previously unselected package postgresql-client-15.                                                                                                                       
Preparing to unpack .../08-postgresql-client-15_15.6-1.pgdg120+2_arm64.deb ...            
Unpacking postgresql-client-15 (15.6-1.pgdg120+2) ...                                     
Selecting previously unselected package postgresql-15.                                    
Preparing to unpack .../09-postgresql-15_15.6-1.pgdg120+2_arm64.deb ...                   
Unpacking postgresql-15 (15.6-1.pgdg120+2) ...                                                                                                                                      
Selecting previously unselected package sysstat.                                          
Preparing to unpack .../10-sysstat_12.6.1-1_arm64.deb ...                                                                                                                           
Unpacking sysstat (12.6.1-1) ...                                                                                                                                                    
Setting up postgresql-client-common (257.pgdg120+1) ...                                                                                                                             
Setting up libcommon-sense-perl:arm64 (3.75-3) ...                                                                                                                                  
Setting up postgresql-client-15 (15.6-1.pgdg120+2) ...                                                                                                                              
update-alternatives: using /usr/share/postgresql/15/man/man1/psql.1.gz to provide /usr/share/man/man1/psql.1.gz (psql.1.gz) in auto mode                                            
Setting up ssl-cert (1.1.2) ...                                                           
Setting up libtypes-serialiser-perl (1.01-1) ...                                          
Setting up libjson-perl (4.10000-1) ...                                                   
Setting up libllvm16:arm64 (1:16.0.6-15~deb12u1) ...                                                                                                                                
Setting up sysstat (12.6.1-1) ...                                                         
                                                                                          
Creating config file /etc/default/sysstat with new version                                
update-alternatives: using /usr/bin/sar.sysstat to provide /usr/bin/sar (sar) in auto mode                                                                                          
Created symlink /etc/systemd/system/sysstat.service.wants/sysstat-collect.timer → /lib/systemd/system/sysstat-collect.timer.                                                        
Created symlink /etc/systemd/system/sysstat.service.wants/sysstat-summary.timer → /lib/systemd/system/sysstat-summary.timer.                                                        
Created symlink /etc/systemd/system/multi-user.target.wants/sysstat.service → /lib/systemd/system/sysstat.service.                                                                  
Setting up libjson-xs-perl (4.030-2+b1) ...                                                                                                                                         
Setting up postgresql-common (257.pgdg120+1) ...                                                                                                                                    
                                                                                                                                                                                    
Creating config file /etc/postgresql-common/createcluster.conf with new version           
Building PostgreSQL dictionaries from installed myspell/hunspell packages...                                                                                                        
  en_gb                                                                                                                                                                             
  en_us                                                                                                                                                                             
Removing obsolete dictionary files:                                                                                                                                                 
'/etc/apt/trusted.gpg.d/apt.postgresql.org.gpg' -> '/usr/share/postgresql-common/pgdg/apt.postgresql.org.gpg'                                                                       
Created symlink /etc/systemd/system/multi-user.target.wants/postgresql.service → /lib/systemd/system/postgresql.service.                                                            
Setting up postgresql-15 (15.6-1.pgdg120+2) ...                                                                                                                                     
Creating new PostgreSQL cluster 15/main ...                                                                                                                                         
/usr/lib/postgresql/15/bin/initdb -D /var/lib/postgresql/15/main --auth-local peer --auth-host scram-sha-256 --no-instructions
The files belonging to this database system will be owned by user "postgres".                                                                                                       
This user must also own the server process.                                                                                                                                         
                                                                                          
The database cluster will be initialized with locale "en_GB.UTF-8".                                                                                                                 
The default database encoding has accordingly been set to "UTF8".                                                                                                                   
The default text search configuration will be set to "english".                                                                                                                     
                                                                                                                                                                                    
Data page checksums are disabled.                                                                                                                                                   
                                                                                                                                                                                    
fixing permissions on existing directory /var/lib/postgresql/15/main ... ok               
creating subdirectories ... ok                                                            
selecting dynamic shared memory implementation ... posix                 
electing default max_connections ... 100                                                                                                                                           
selecting default shared_buffers ... 128MB                                                                                                                                          
selecting default time zone ... Europe/London                                                                                                                                       
creating configuration files ... ok                                                                                                                                                 
running bootstrap script ... ok                                                                                                                                                     
performing post-bootstrap initialization ... ok                                                                                                                                     
syncing data to disk ... ok                                                                                                                                                         
Processing triggers for man-db (2.11.2-2) ...                                                                                                                                       
Processing triggers for libc-bin (2.36-9+rpt2+deb12u4) ...                                                                                                                          
# There is old data for pg 13, start and upgrade cluster ...                                                                                                                        
                                                                                                                                                                                    
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.                                                                                                     
                                                                                                                                                                                    
Reading package lists...                                                                                                                                                            
Building dependency tree...                                                                                                                                                         
Reading state information...                                                                                                                                                        
The following package was automatically installed and is no longer required:              
  pi-language-support                                                                                                                                                               
Use 'sudo apt autoremove' to remove it.                                                                                                                                             
The following additional packages will be installed:                                      
  postgresql-client-13                                                                    
Suggested packages:                                                                       
  postgresql-doc-13                                                                       
The following NEW packages will be installed:                                                                                                                                       
  postgresql-13 postgresql-client-13                                                      
0 upgraded, 2 newly installed, 0 to remove and 4 not upgraded.                                                                                                                      
Need to get 16.5 MB of archives.                                                                                                                                                    
After this operation, 59.5 MB of additional disk space will be used.                                                                                                                
Get:1 http://apt.postgresql.org/pub/repos/apt bookworm-pgdg/main arm64 postgresql-client-13 arm64 13.14-1.pgdg120+2 [1,474 kB]                                                      
Get:2 http://apt.postgresql.org/pub/repos/apt bookworm-pgdg/main arm64 postgresql-13 arm64 13.14-1.pgdg120+2 [15.0 MB]                                                              
Preconfiguring packages ...                                                                                                                                                         
Fetched 16.5 MB in 1s (24.8 MB/s)                                                         
Selecting previously unselected package postgresql-client-13.                             
(Reading database ... 130784 files and directories currently installed.)                  
Preparing to unpack .../postgresql-client-13_13.14-1.pgdg120+2_arm64.deb ...                                                                                                        
Unpacking postgresql-client-13 (13.14-1.pgdg120+2) ...                                    
Selecting previously unselected package postgresql-13.                                    
Preparing to unpack .../postgresql-13_13.14-1.pgdg120+2_arm64.deb ...                     
Unpacking postgresql-13 (13.14-1.pgdg120+2) ...                                                                                                                                     
Setting up postgresql-client-13 (13.14-1.pgdg120+2) ...                                                                                                                             
Setting up postgresql-13 (13.14-1.pgdg120+2) ...                                                                                                                                    
Creating new PostgreSQL cluster 13/main ...                                                                                                                                         
/usr/lib/postgresql/13/bin/initdb -D /var/lib/postgresql/13/main --auth-local peer --auth-host md5                                                                                  
The files belonging to this database system will be owned by user "postgres".                                                                                                       
This user must also own the server process.                                                                                                                                         
                                                                                          
The database cluster will be initialized with locale "en_GB.UTF-8".                                                                                                                 
The default database encoding has accordingly been set to "UTF8".                                                                                                                   
The default text search configuration will be set to "english".                                                                                                                     
                                                                                                                                                                                    
Data page checksums are disabled.                                                                                                                                                   
                                                                                                                                                                                    
fixing permissions on existing directory /var/lib/postgresql/13/main ... ok                                                                                                         
creating subdirectories ... ok                                                                                                                                                      
selecting dynamic shared memory implementation ... posix                                                                                                                            
selecting default max_connections ... 100                                                                                                                                           
selecting default shared_buffers ... 128MB                                                                                                                                          
selecting default time zone ... Europe/London                                             
creating configuration files ... ok                                                                                                                                                 
running bootstrap script ... ok                                                                                                                                                     
performing post-bootstrap initialization ... ok                                                                                                                                     
syncing data to disk ... ok                                                                                                                                                         
                                                                                                                                                                                    
Success. You can now start the database server using:                                                                                                                               
                                                                                          
    pg_ctlcluster 13 main start           
    Processing triggers for postgresql-common (257.pgdg120+1) ...                                                                                                                       
Building PostgreSQL dictionaries from installed myspell/hunspell packages...                                                                                                        
  en_gb                                                                                                                                                                             
  en_us                                                                                                                                                                             
Removing obsolete dictionary files:                                                                                                                                                 
# Create pg 13 config                                                                                                                                                               
Error: cluster configuration already exists                                                                                                                                         
Setting default password for postgres user                                                                                                                                          
could not change directory to "/root": Permission denied                                                                                                                            
ALTER ROLE                                                                                                                                                                          
Error: cluster configuration already exists                                                                                                                                         
# backup /mnt/hdd/app-data/postgresql/15                                                                                                                                            
mv: cannot stat '/mnt/hdd/app-data/postgresql/15': No such file or directory                                                                                                        
# Drop empty pg 15 cluster                                                                                                                                                          
Warning: corrupted cluster: data directory does not exist                                                                                                                           
Stopping old cluster...                                                                                                                                                             
Restarting old cluster with restricted connections...                                     
Creating new PostgreSQL cluster 15/main ...                                                                                                                                         
/usr/lib/postgresql/15/bin/initdb -D /var/lib/postgresql/15/main --auth-local peer --auth-host scram-sha-256 --no-instructions --encoding UTF8 --lc-collate en_GB.UTF-8 --lc-ctype e
n_GB.UTF-8                                                                                
The files belonging to this database system will be owned by user "postgres".             
This user must also own the server process.                                               
                                                                                          
The database cluster will be initialized with locale "en_GB.UTF-8".                                                                                                                 
The default text search configuration will be set to "english".                           
                                                                                                                                                                                    
Data page checksums are disabled.                                                                                                                                                   
                                                                                                                                                                                    
fixing permissions on existing directory /var/lib/postgresql/15/main ... ok                                                                                                         
creating subdirectories ... ok                                                                                                                                                      
selecting dynamic shared memory implementation ... posix                                                                                                                            
selecting default max_connections ... 100                                                 
selecting default shared_buffers ... 128MB                                                
selecting default time zone ... Europe/London                                             
creating configuration files ... ok                                                                                                                                                 
running bootstrap script ... ok                                                           
performing post-bootstrap initialization ... ok                                           
syncing data to disk ... ok                                                               
                                                                                                                                                                                    
Copying old configuration files...                                                                                                                                                  
Copying old start.conf...                                                                                                                                                           
Copying old pg_ctl.conf...                                                                                                                                                          
Starting new cluster...                                                                                                                                                             
Running init phase upgrade hook scripts ...                                                                                                                                         
                                                                                                                                                                                    
Roles, databases, schemas, ACLs...                                                        
 set_config                                                                                                                                                                         
------------                                                                                                                                                                        
                                                                                                                                                                                    
(1 row)                                                                                                                                                                             
                                                                                                                                                                                    
 set_config                                                                                                                                                                         
------------                                                                                                                                                                        
                                                                                                                                                                                    
(1 row)                                                                                                                                                                             
                                                                                                                                                                                    
 set_config                                                                                                                                                                         
------------                                                                              
                                                                                                                                                                                    
(1 row)                                                                                                                                                                             
                                                                                                                                                                                    
 set_config                                                                                                                                                                         
------------                                                                                                                                                                        
                                                                                                                                                                                    
(1 row)                                                                                   
                                                                                          
Fixing hardcoded library paths for stored procedures...        
                                                                                                                                                                          [271/4744]
Fixing hardcoded library paths for stored procedures...                                                                                                                             
Upgrading database template1...                                                                                                                                                     
Fixing hardcoded library paths for stored procedures...                                                                                                                             
Upgrading database testdb13...                                                                                                                                                      
Fixing hardcoded library paths for stored procedures...                                                                                                                             
Upgrading database postgres...                                                                                                                                                      
Stopping target cluster...                                                                                                                                                          
Stopping old cluster...                                                                                                                                                             
Disabling automatic startup of old cluster...                                                                                                                                       
Starting upgraded cluster on port 5433...                                                                                                                                           
Running finish phase upgrade hook scripts ...                                                                                                                                       
vacuumdb: processing database "postgres": Generating minimal optimizer statistics (1 target)                                                                                        
vacuumdb: processing database "template1": Generating minimal optimizer statistics (1 target)                                                                                       
vacuumdb: processing database "testdb13": Generating minimal optimizer statistics (1 target)                                                                                        
vacuumdb: processing database "postgres": Generating medium optimizer statistics (10 targets)                                                                                       
vacuumdb: processing database "template1": Generating medium optimizer statistics (10 targets)                                                                                      
vacuumdb: processing database "testdb13": Generating medium optimizer statistics (10 targets)                                                                                       
vacuumdb: processing database "postgres": Generating default (full) optimizer statistics  
vacuumdb: processing database "template1": Generating default (full) optimizer statistics                                                                                           
vacuumdb: processing database "testdb13": Generating default (full) optimizer statistics                                                                                            
                                                                                          
Success. Please check that the upgraded cluster works. If it does,                        
you can remove the old cluster with                                                       
    pg_dropcluster 13 main                                                                
                                                                                                                                                                                    
Ver Cluster Port Status Owner    Data directory              Log file                     
13  main    5432 down   postgres /var/lib/postgresql/13/main /var/log/postgresql/postgresql-13-main.log                                                                             
Ver Cluster Port Status Owner    Data directory              Log file                                                                                                               
15  main    5433 online postgres /var/lib/postgresql/15/main /var/log/postgresql/postgresql-15-main.log                                                                             
# backup /mnt/hdd/app-data/postgresql/13                                                                                                                                            
Warning: corrupted cluster: data directory does not exist                                                                                                                           
                                                                                                                                                                                    
WARNING: apt does not have a stable CLI interface. Use with caution in scripts.           
                                                                                          
Reading package lists...                                                                  
Building dependency tree...                                                                                                                                                         
Reading state information...                                                              
The following packages were automatically installed and are no longer required:           
  pi-language-support postgresql-client-13                                                
Use 'sudo apt autoremove' to remove them.                                                                                                                                           
The following packages will be REMOVED:                                                                                                                                             
  postgresql-13                                                                                                                                                                     
0 upgraded, 0 newly installed, 1 to remove and 4 not upgraded.                                                                                                                      
After this operation, 52.3 MB disk space will be freed.                                                                                                                             
(Reading database ... 132610 files and directories currently installed.)                                                                                                            
Removing postgresql-13 (13.14-1.pgdg120+2) ...                                                                                                                                      
Processing triggers for postgresql-common (257.pgdg120+1) ...                             
Building PostgreSQL dictionaries from installed myspell/hunspell packages...                                                                                                        
  en_gb                                                                                                                                                                             
  en_us                                                                                                                                                                             
Removing obsolete dictionary files:                                                                                                                                                 
port = 5433                             # (change requires restart)                                                                                                                 
# Switch port back to 5432                                                                                                                                                          
# Restart posgresql.service                                                                                                                                                         
Synchronizing state of postgresql.service with SysV service script with /lib/systemd/systemd-sysv-install.                                                                          
Executing: /lib/systemd/systemd-sysv-install enable postgresql                                                                                                                      
Created symlink /etc/systemd/system/multi-user.target.wants/postgresql@15-main.service → /lib/systemd/system/postgresql@.service.                                                   
psql (PostgreSQL) 15.6 (Debian 15.6-1.pgdg120+2)                                                                                                                                    
# wait for the postgresql server to start                                                 
Setting default password for postgres user                                                                                                                                          
could not change directory to "/root": Permission denied                                                                                                                            
ALTER ROLE                                                                                                                                                                          
OK PostgreSQL installed

* postgres: add comments, backup and remove unused databases * bats: remove output, use --verbose-run locally * always drop empty pg 15 cluster * check cluster correctly, fix port after upgrade * always restart postgresql.service to fix port

openoms · 2024-04-06T08:25:42Z

cherry-picked to v1.11

* postgres: add comments, backup and remove unused databases * bats: remove output, use --verbose-run locally * always drop empty pg 15 cluster * check cluster correctly, fix port after upgrade * always restart postgresql.service to fix port

openoms requested a review from rootzoll as a code owner April 4, 2024 23:52

openoms changed the title ~~fix postgreql 13 recovery~~ fix postgresql 13 recovery Apr 4, 2024

openoms force-pushed the postgres-13-recover branch from 2ddcaad to 00f7a0a Compare April 4, 2024 23:57

openoms force-pushed the postgres-13-recover branch from d4c7123 to f352b1d Compare April 5, 2024 19:04

openoms added 5 commits April 6, 2024 09:54

postgres: add comments, backup and remove unused databases

3f395bc

bats: remove output, use --verbose-run locally

69296fc

always drop empty pg 15 cluster

61f73d2

check cluster correctly, fix port after upgrade

4e2789d

always restart postgresql.service to fix port

4e82d8c

openoms force-pushed the postgres-13-recover branch from f352b1d to 4e82d8c Compare April 6, 2024 07:57

openoms merged commit 3aa0bb4 into dev Apr 6, 2024
2 of 3 checks passed

openoms deleted the postgres-13-recover branch April 6, 2024 08:24

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix postgresql 13 recovery #4521

fix postgresql 13 recovery #4521

openoms commented Apr 4, 2024

openoms commented Apr 5, 2024 •

edited

rootzoll commented Apr 5, 2024

openoms commented Apr 5, 2024

openoms commented Apr 5, 2024

openoms commented Apr 5, 2024 •

edited

openoms commented Apr 6, 2024 •

edited

openoms commented Apr 6, 2024

fix postgresql 13 recovery #4521

fix postgresql 13 recovery #4521

Conversation

openoms commented Apr 4, 2024

openoms commented Apr 5, 2024 • edited

rootzoll commented Apr 5, 2024

openoms commented Apr 5, 2024

openoms commented Apr 5, 2024

openoms commented Apr 5, 2024 • edited

openoms commented Apr 6, 2024 • edited

openoms commented Apr 6, 2024

openoms commented Apr 5, 2024 •

edited

openoms commented Apr 5, 2024 •

edited

openoms commented Apr 6, 2024 •

edited