Segfault in delete_wal_older_than

[johnaohara]

Describe the bug

A segfault occurs when pgmoneta attempts to delete historic WALs

Dec 13 01:25:00 server02.example.com kernel: pgmoneta[3527739]: segfault at 5565a6579 ip 00007f125d869e41 sp 00007ffdaab1de90 error 4 in libc-2.28.so[7f125d7ce000+1bb000]
Dec 13 01:25:00 server02.example.com kernel: Code: 2e 0f 1f 84 00 00 00 00 00 f3 0f 1e fa 53 48 83 ec 10 48 8b 05 c8 30 32 00 48 8b 00 48 85 c0 0f 85 84 00 00 00 48 85 ff 74 6f <48> 8b 47 f8 48 8d 77 f0 a8 02 75 >
Dec 13 01:25:00 server02.example.com systemd[1]: Starting Check pmlogger instances are running...
Dec 13 01:25:00 server02.example.com systemd[1]: Started Process Core Dump (PID 3527741/UID 0).
Dec 13 01:25:00 server02.example.com systemd[1]: Started Check pmlogger instances are running.
Dec 13 01:25:00 server02.example.com systemd-coredump[3527743]: Process 3527739 (pgmoneta) of user 1001 dumped core.
                                                                                        
                                                                                        Stack trace of thread 3527739:
                                                                                        #0  0x00007f125d869e41 __libc_free (libc.so.6)
                                                                                        #1  0x00007f125dba817a delete_wal_older_than (libpgmoneta.so.0)
                                                                                        #2  0x00007f125dba83cb pgmoneta_delete_wal (libpgmoneta.so.0)
                                                                                        #3  0x00007f125dbcd0c4 retain_execute (libpgmoneta.so.0)
                                                                                        #4  0x00007f125dbbae7d pgmoneta_retention (libpgmoneta.so.0)
                                                                                        #5  0x00007f125f477f63 ev_invoke_pending (libev.so.4)
                                                                                        #6  0x00007f125f47bff8 ev_run (libev.so.4)
                                                                                        #7  0x00005565a601f55e ev_loop (pgmoneta)
                                                                                        #8  0x00007f125d808d85 __libc_start_main (libc.so.6)
                                                                                        #9  0x00005565a601ff6e _start (pgmoneta)

To Reproduce

Steps to reproduce the behavior.

Version

$ pgmoneta -V
pgmoneta 0.8.0

libc

glibc-2.28-225.el8_8.6.x86_64

OpenSSL

What is the version of OpenSSL ?

OS

Red Hat Enterprise Linux release 8.8 (Ootpa)

$ uname -a Linux server02.example.com 4.18.0-477.27.1.el8_8.x86_64 #1 SMP Thu Aug 31 10:29:22 EDT 2023 x86_64 x86_64 x86_64 GNU/Linux

Configuration

Can you provide the configuration pgmoneta ?

[pgmoneta]
host = *
metrics = 5001

base_dir = /home/postgres/cluster/Backup/

compression = zstd

retention = 7

log_type = file
log_level = debug1 
log_path = /home/postgres/cluster/Logs/pgmoneta.log
log_mode = create

unix_socket_dir = /tmp/

[primary]
host = localhost
port = 6000
user = repl
wal_slot = backup
#wal_shipping = /mnt/backup/cluster/

Debug logs

Can you provide any debug logs (log_level = debug5) of the issue ?

Tip

Use ``` before and after the text to keep the output as is.

[jesperpedersen]

coredumpctl debug
addr2line -e ./libpgmoneta.so.0.8.0 -fp 0x1926a
delete_wal_older_than at src/libpgmoneta/delete.c:266 (discriminator 3)

[Jubilee101]

I think it's fixed. It entered error section by mistake. So sorry for the trouble, this part is quite hard to test out.

[Jubilee101]

And thanks a lot for the core dump. That saves me a lot of time.

[johnaohara]

Thanks!