PANIC: could not locate a valid checkpoint record

[martinr92]

Hello,

What happened:
The database is running with instances: 2 and a podAntiAffinityType: required as the cluster configuration.
Now one of the nodes crashed (pod with "-1" at the end). So the pod with "-2" took over.
~30 min later, the crashed node came back online. So a new pod with "-1" started. But this one is now restarting endless.

Now I stated analyzing the logs of the container and found:

PANIC: could not locate a valid checkpoint record

and

{"level":"error","ts":"2023-10-17T18:22:14Z","msg":"Failed to execute pg_rewind","logging_pod":"gts-xxxxxx-gotosocial-db-1","options":["-P","--source-server","host=gts-xxxxxx-gotosocial-db-rw user=streaming_replica port=5432 sslkey=/controller/certificates/streaming_replica.key sslcert=/controller/certificates/streaming_replica.crt sslrootcert=/controller/certificates/server-ca.crt application_name=gts-xxxxxx-gotosocial-db-1 sslmode=verify-ca dbname=postgres","--target-pgdata","/var/lib/postgresql/data/pgdata","--restore-target-wal"],"error":"exit status 1","stacktrace":"github.com/cloudnative-pg/cloudnative-pg/pkg/management/log.(*logger).Error\n\tpkg/management/log/log.go:128\ngithub.com/cloudnative-pg/cloudnative-pg/pkg/management/log.Error\n\tpkg/management/log/log.go:166\ngithub.com/cloudnative-pg/cloudnative-pg/pkg/management/postgres.(*Instance).Rewind\n\tpkg/management/postgres/instance.go:783\ngithub.com/cloudnative-pg/cloudnative-pg/internal/management/controller.(*InstanceReconciler).verifyPgDataCoherenceForPrimary\n\tinternal/management/controller/instance_startup.go:235\ngithub.com/cloudnative-pg/cloudnative-pg/internal/management/controller.(*InstanceReconciler).initialize\n\tinternal/management/controller/instance_controller.go:327\ngithub.com/cloudnative-pg/cloudnative-pg/internal/management/controller.(*InstanceReconciler).Reconcile\n\tinternal/management/controller/instance_controller.go:141\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:235"}
{"level":"error","ts":"2023-10-17T18:22:15Z","msg":"Reconciler error","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"gts-xxxxxx-gotosocial-db","namespace":"gts"},"namespace":"gts","name":"gts-xxxxxx-gotosocial-db","reconcileID":"c6099101-f51b-4cd2-9fce-6947d9493f63","error":"while activating instance: error starting PostgreSQL instance: exit status 1","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.14.5/pkg/internal/controller/controller.go:235"}

Can someone explain me what this messages exactly mean?
Do you have any ideas, how to get out of this situation (e.g. throw the "-1" instance away and start with a fresh copy of "-2")?

Thanks for any ideas and help.
postgres2.log

[gbartolini]

From the logs, I see you are running version 1.19.1. It is 7 months old, and many bugs have been fixed in the meantime. Please see: https://cloudnative-pg.io/documentation/1.19/release_notes/v1.19/

I cannot locate any more interesting information from the provided log. It looks like that no WAL archive is configured, am I wrong?

Workaround is correct: destroy the PVC and the pod (use the destroy command of the cnpg plugin if you want).

[martinr92]

Thanks. I already deleted the PVC, so it's now back up and running.

Yes, WAL archiving was not configured, but is now.
I also plan an upgrade to 1.20 during next week.