Nextcloud postgres fails to start

[frznfngrs]

root@truenas:~# 2022 May 16 08:00:54 truenas Process 4037959 (postgres) of user 999 dumped core.
The following cli message is printed on container failing to start
Stack trace of thread 26:
#0 0x00007f73f2fb17bb n/a (/lib/x86_64-linux-gnu/libc-2.28.so + 0x377bb)
#1 0x0000560c72c06a1d n/a (/usr/lib/postgresql/13/bin/postgres + 0x15ea1d)
#2 0x0000560c72e1ca63 n/a (/usr/lib/postgresql/13/bin/postgres + 0x374a63)
#3 0x0000560c72c13326 n/a (/usr/lib/postgresql/13/bin/postgres + 0x16b326)
#4 0x0000560c72e186df n/a (/usr/lib/postgresql/13/bin/postgres + 0x3706df)
#5 0x0000560c72e1c00d n/a (/usr/lib/postgresql/13/bin/postgres + 0x37400d)
#6 0x0000560c72b67f3f n/a (/usr/lib/postgresql/13/bin/postgres + 0xbff3f)
#7 0x00007f73f2f9e09b n/a (/lib/x86_64-linux-gnu/libc-2.28.so + 0x2409b)

The pod logs for Postgres are below
PostgreSQL Database directory appears to contain a database; Skipping initialization

2022-05-16 13:04:19.800 UTC [1] LOG: starting PostgreSQL 13.1 (Debian 13.1-1.pgdg100+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 8.3.0-6) 8.3.0, 64-bit
2022-05-16 13:04:19.800 UTC [1] LOG: listening on IPv4 address "0.0.0.0", port 5432
2022-05-16 13:04:19.800 UTC [1] LOG: listening on IPv6 address "::", port 5432
2022-05-16 13:04:20.005 UTC [1] LOG: listening on Unix socket "/var/run/postgresql/.s.PGSQL.5432"
2022-05-16 13:04:20.011 UTC [26] LOG: database system was shut down at 2022-05-09 18:31:12 UTC
2022-05-16 13:04:20.011 UTC [26] LOG: invalid record length at 0/860A2558: wanted 24, got 0
2022-05-16 13:04:20.012 UTC [26] LOG: invalid primary checkpoint record
2022-05-16 13:04:20.012 UTC [26] PANIC: could not locate a valid checkpoint record
2022-05-16 13:04:21.233 UTC [27] FATAL: the database system is starting up
2022-05-16 13:04:22.014 UTC [1] LOG: startup process (PID 26) was terminated by signal 6: Aborted
2022-05-16 13:04:22.014 UTC [1] LOG: aborting startup due to startup process failure
2022-05-16 13:04:22.033 UTC [1] LOG: database system is shut down

This just loops indefinitely.
nextcloud
23.0.3_1.5.0
Up to date

I am trying to figure out how I may make use of the ix-postgres_backups at /mnt/Vol2/ix-applications/releases/nextcloud/volumes/ix_volumes/ix-postgres_backups
If thats even the right solution...

[frznfngrs]

I fixed this by chmod +x /usr/bin/apt on truenas scale machine
then installing postgres-13
then creating a user to match uid 999 which owned the postgres data directory
then running
sudo -u 999 /usr/lib/postgresql/13/bin/pg_resetwal -f --pgdata /mnt/Vol2/ix-applications/releases/nextcloud/volumes/ix_volumes/ix-postgres_data

[sonicaj]

@frznfngrs can you please create an issue at https://jira.ixsystems.com outlining how we can reproduce this issue and we can have this fixed then. Thanks!