[Bug]: Primary PostgreSQL Instance Not Gracefully Shutting Down and Missing WAL File #3680
Open
4 tasks done
Assignees
Labels
triage
Pending triage
Comments
|
From what I can see, it looks like an issue with the underlying file system (data corruption of that file). You need to recreate the PVC of that instance. |
|
It happens 2 days in a row, we are using EC2 spot instances, and we managed to solve it by deleting the problematic PVC (gp3) but that manual PVC creation every day is probably not the desired behavior. |
4 tasks
|
This might be related to #3698. Give us some time to investigate. |
|
I face the same issue in multiple clusters in the same k8s cluster. I think it occurs when a network error happens. The only way to resolve it is to delete the PVC of the failing pod and remove the pod. In this case, the operator is going to recreate the failing replica. Logs: |
|
We are facing the same problem on Google GKE clusters. After each node upgrade random instances have problems with recovering. The only workaround we found is removing one of failing instance PVC. |
|
Just encountered into the same issue in GKE |
|
I keep running into this every other day |
|
Also discovered this now.. is CNPG team aware of this? Whom to ping about this issue? only working solution was to delete the failing Pod and PVC so a new node is created to join the cluster |
|
I am facing similar issue. Do we have any fix available for below issue ? I tried to delete the pod but did not help.
|
|
@gbartolini is it possible to modify the health check behavior of the pods managed by CNPG? |
@gbartolini Any updates on this would be greatly appreciated 🙏🏿 |
|
The only way I've been able to recover from this is to delete both the PVC and Pod |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Is there an existing issue already for this bug?
I have read the troubleshooting guide
I am running a supported version of CloudNativePG
Contact Details
oren@aperio.ai
Version
1.22.0
What version of Kubernetes are you using?
1.27
What is your Kubernetes environment?
Cloud: Amazon EKS
How did you install the operator?
Helm
What happened?
We deployed the operator using a Helm chart and the PostgreSQL cluster resource using a separate Helm chart. Three instances of PostgreSQL were deployed on EC2 Spot instances. After a few hours, it appears that the primary instance (postgres-1) did not gracefully shut down, leading to the absence of the relevant Write-Ahead Logging (WAL) file.
Subsequently, attempts to use pg_rewind have failed, with the process encountering an empty file. Additionally, the file "pg_xact/0000" is also empty, contributing to the inability to resolve the issue and leaving the old primary instance in a stuck state.
Cluster resource
apiVersion: postgresql.cnpg.io/v1 kind: Cluster metadata: annotations: meta.helm.sh/release-name: postgres-cn meta.helm.sh/release-namespace: stg creationTimestamp: "2024-01-09T12:06:49Z" generation: 4 labels: app.kubernetes.io/managed-by: Helm name: postgres namespace: stg resourceVersion: "346234286" uid: 69ddea55-1c72-4e9f-abf6-aa2e43a32db4 spec: affinity: podAntiAffinityType: preferred bootstrap: initdb: database: app encoding: UTF8 import: databases: - webapi source: externalCluster: xxxxxx type: microservice localeCType: C localeCollate: C owner: app enableSuperuserAccess: true externalClusters: - connectionParameters: dbname: webapi host: xxxxxxxxxxxx user: postgres name: xxxxxxx password: key: postgres-password name: xxxxxxxxxxxxxx failoverDelay: 0 imageName: xxxxxxx/postgresql:16.1-xxxxxxxxx inheritedMetadata: labels: app.kubernetes.io/name: postgres app.kubernetes.io/version: "16.1" instances: 3 logLevel: info maxSyncReplicas: 2 minSyncReplicas: 1 monitoring: customQueriesConfigMap: - key: queries name: cnpg-default-monitoring disableDefaultQueries: false enablePodMonitor: false postgresGID: 26 postgresUID: 26 postgresql: parameters: archive_mode: "on" archive_timeout: 5min dynamic_shared_memory_type: posix log_destination: csvlog log_directory: /controller/log log_filename: postgres log_rotation_age: "0" log_rotation_size: "0" log_truncate_on_rotation: "false" logging_collector: "on" max_parallel_workers: "32" max_replication_slots: "32" max_worker_processes: "32" shared_memory_type: mmap shared_preload_libraries: "" ssl_max_protocol_version: TLSv1.3 ssl_min_protocol_version: TLSv1.3 wal_keep_size: 512MB wal_receiver_timeout: 5s wal_sender_timeout: 5s syncReplicaElectionConstraint: enabled: false primaryUpdateMethod: restart primaryUpdateStrategy: unsupervised replicationSlots: highAvailability: enabled: true slotPrefix: _cnpg_ updateInterval: 30 resources: {} smartShutdownTimeout: 180 startDelay: 3600 stopDelay: 1800 storage: resizeInUseVolumes: true size: 30Gi storageClass: gp3 superuserSecret: name: postgres-superuser switchoverDelay: 3600 status: certificates: clientCASecret: postgres-ca expirations: postgres-ca: 2024-04-08 12:01:49 +0000 UTC postgres-replication: 2024-04-08 12:01:49 +0000 UTC postgres-server: 2024-04-08 12:01:49 +0000 UTC replicationTLSSecret: postgres-replication serverAltDNSNames: - postgres-rw - postgres-rw.stg - postgres-rw.stg.svc - postgres-r - postgres-r.stg - postgres-r.stg.svc - postgres-ro - postgres-ro.stg - postgres-ro.stg.svc serverCASecret: postgres-ca serverTLSSecret: postgres-server cloudNativePGCommitHash: 86b9dc80 cloudNativePGOperatorHash: 411a67468c5e1357d387221ce5b11d906bd33d62324846a8c776d206b6c94162 conditions: - lastTransitionTime: "2024-01-10T08:20:16Z" message: Cluster Is Not Ready reason: ClusterIsNotReady status: "False" type: Ready - lastTransitionTime: "2024-01-10T12:18:14Z" message: Continuous archiving is working reason: ContinuousArchivingSuccess status: "True" type: ContinuousArchiving configMapResourceVersion: metrics: cnpg-default-monitoring: "345484626" currentPrimary: postgres-2 currentPrimaryTimestamp: "2024-01-10T08:21:59.663283Z" healthyPVC: - postgres-1 - postgres-2 - postgres-3 instanceNames: - postgres-1 - postgres-2 - postgres-3 instances: 3 instancesReportedState: postgres-1: isPrimary: false postgres-2: isPrimary: true timeLineID: 2 postgres-3: isPrimary: false timeLineID: 2 instancesStatus: healthy: - postgres-2 - postgres-3 replicating: - postgres-1 latestGeneratedNode: 3 managedRolesStatus: {} phase: Waiting for the instances to become active phaseReason: Some instances are not yet active. Please wait. poolerIntegrations: pgBouncerIntegration: {} pvcCount: 3 readService: postgres-r readyInstances: 2 secretsResourceVersion: applicationSecretVersion: "345484596" clientCaSecretVersion: "345484593" externalClusterSecretVersion: stg-postgres-postgresql: "231319709" replicationSecretVersion: "345484595" serverCaSecretVersion: "345484593" serverSecretVersion: "345484594" superuserSecretVersion: "341642273" targetPrimary: postgres-2 targetPrimaryTimestamp: "2024-01-10T08:20:16.701057Z" timelineID: 2 topology: instances: postgres-1: {} postgres-2: {} postgres-3: {} nodesUsed: 2 successfullyExtracted: true writeService: postgres-rwRelevant log output
{"level":"error","ts":"2024-01-10T10:58:19Z","msg":"Reconciler error","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"postgres","namespace":"stg"},"namespace":"stg","name":"postgres","reconcileID":"9b64cee9-7e71-4e05-9bb0-e1cbaee76c9d","error":"while activating instance: error starting PostgreSQL instance: exit status 1","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"} {"level":"info","ts":"2024-01-10T10:58:28Z","msg":"Cluster status","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"postgres","namespace":"stg"},"namespace":"stg","name":"postgres","reconcileID":"b6dbc5ac-aab1-4528-bc0c-f67aa8dac9b1","uuid":"2f587c83-afa7-11ee-9aef-929973a308f2","logging_pod":"postgres-1","currentPrimary":"postgres-2","targetPrimary":"postgres-2"} {"level":"info","ts":"2024-01-10T10:58:28Z","msg":"This is an old primary instance, waiting for the switchover to finish","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"postgres","namespace":"stg"},"namespace":"stg","name":"postgres","reconcileID":"b6dbc5ac-aab1-4528-bc0c-f67aa8dac9b1","uuid":"2f587c83-afa7-11ee-9aef-929973a308f2","logging_pod":"postgres-1","currentPrimary":"postgres-2","targetPrimary":"postgres-2"} {"level":"info","ts":"2024-01-10T10:58:28Z","msg":"Switchover completed","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"postgres","namespace":"stg"},"namespace":"stg","name":"postgres","reconcileID":"b6dbc5ac-aab1-4528-bc0c-f67aa8dac9b1","uuid":"2f587c83-afa7-11ee-9aef-929973a308f2","logging_pod":"postgres-1","targetPrimary":"postgres-2","currentPrimary":"postgres-2"} {"level":"info","ts":"2024-01-10T10:58:28Z","msg":"Waiting for the new primary to be available","logging_pod":"postgres-1","primaryConnInfo":"host=postgres-rw user=streaming_replica port=5432 sslkey=/controller/certificates/streaming_replica.key sslcert=/controller/certificates/streaming_replica.crt sslrootcert=/controller/certificates/server-ca.crt application_name=postgres-1 sslmode=verify-ca dbname=postgres connect_timeout=5"} {"level":"info","ts":"2024-01-10T10:58:28Z","msg":"Extracting pg_controldata information","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"postgres","namespace":"stg"},"namespace":"stg","name":"postgres","reconcileID":"b6dbc5ac-aab1-4528-bc0c-f67aa8dac9b1","uuid":"2f587c83-afa7-11ee-9aef-929973a308f2","logging_pod":"postgres-1","reason":"before pg_rewind"} {"level":"info","ts":"2024-01-10T10:58:28Z","logger":"pg_controldata","msg":"pg_control version number: 1300\nCatalog version number: 202307071\nDatabase system identifier: 7322068936411099156\nDatabase cluster state: shut down\npg_control last modified: Wed 10 Jan 2024 08:19:35 AM UTC\nLatest checkpoint location: 2/AA000028\nLatest checkpoint's REDO location: 2/AA000028\nLatest checkpoint's REDO WAL file: 0000000100000002000000AA\nLatest checkpoint's TimeLineID: 1\nLatest checkpoint's PrevTimeLineID: 1\nLatest checkpoint's full_page_writes: on\nLatest checkpoint's NextXID: 0:36923\nLatest checkpoint's NextOID: 25092\nLatest checkpoint's NextMultiXactId: 1\nLatest checkpoint's NextMultiOffset: 0\nLatest checkpoint's oldestXID: 722\nLatest checkpoint's oldestXID's DB: 1\nLatest checkpoint's oldestActiveXID: 0\nLatest checkpoint's oldestMultiXid: 1\nLatest checkpoint's oldestMulti's DB: 1\nLatest checkpoint's oldestCommitTsXid:0\nLatest checkpoint's newestCommitTsXid:0\nTime of latest checkpoint: Wed 10 Jan 2024 08:19:35 AM UTC\nFake LSN counter for unlogged rels: 0/3E8\nMinimum recovery ending location: 0/0\nMin recovery ending loc's timeline: 0\nBackup start location: 0/0\nBackup end location: 0/0\nEnd-of-backup record required: no\nwal_level setting: logical\nwal_log_hints setting: on\nmax_connections setting: 100\nmax_worker_processes setting: 32\nmax_wal_senders setting: 10\nmax_prepared_xacts setting: 0\nmax_locks_per_xact setting: 64\ntrack_commit_timestamp setting: off\nMaximum data alignment: 8\nDatabase block size: 8192\nBlocks per segment of large relation: 131072\nWAL block size: 8192\nBytes per WAL segment: 16777216\nMaximum length of identifiers: 64\nMaximum columns in an index: 32\nMaximum size of a TOAST chunk: 1996\nSize of a large-object chunk: 2048\nDate/time type storage: 64-bit integers\nFloat8 argument passing: by value\nData page checksum version: 0\nMock authentication nonce: 4e9ac5de19e20369e90184039bdcf5c867fa08bd811982f2edb29d1f11d8fd7b\n","pipe":"stdout","logging_pod":"postgres-1"} {"level":"info","ts":"2024-01-10T10:58:28Z","msg":"Starting up pg_rewind","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"postgres","namespace":"stg"},"namespace":"stg","name":"postgres","reconcileID":"b6dbc5ac-aab1-4528-bc0c-f67aa8dac9b1","uuid":"2f587c83-afa7-11ee-9aef-929973a308f2","logging_pod":"postgres-1","pgdata":"/var/lib/postgresql/data/pgdata","options":["-P","--source-server","host=postgres-rw user=streaming_replica port=5432 sslkey=/controller/certificates/streaming_replica.key sslcert=/controller/certificates/streaming_replica.crt sslrootcert=/controller/certificates/server-ca.crt application_name=postgres-1 sslmode=verify-ca dbname=postgres","--target-pgdata","/var/lib/postgresql/data/pgdata","--restore-target-wal"]} {"level":"info","ts":"2024-01-10T10:58:28Z","logger":"pg_rewind","msg":"pg_rewind: connected to server","pipe":"stderr","logging_pod":"postgres-1"} {"level":"info","ts":"2024-01-10T10:58:28Z","logger":"pg_rewind","msg":"pg_rewind: servers diverged at WAL location 2/A9000860 on timeline 1","pipe":"stderr","logging_pod":"postgres-1"} {"level":"info","ts":"2024-01-10T10:58:28Z","logger":"pg_rewind","msg":"pg_rewind: error: could not read file \"/var/lib/postgresql/data/pgdata/pg_wal/0000000100000002000000A8\": read 0 of 8192","pipe":"stderr","logging_pod":"postgres-1"} {"level":"info","ts":"2024-01-10T10:58:28Z","logger":"pg_rewind","msg":"pg_rewind: error: could not find previous WAL record at 2/A8241C48","pipe":"stderr","logging_pod":"postgres-1"} {"level":"error","ts":"2024-01-10T10:58:28Z","msg":"Failed to execute pg_rewind","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"postgres","namespace":"stg"},"namespace":"stg","name":"postgres","reconcileID":"b6dbc5ac-aab1-4528-bc0c-f67aa8dac9b1","uuid":"2f587c83-afa7-11ee-9aef-929973a308f2","logging_pod":"postgres-1","options":["-P","--source-server","host=postgres-rw user=streaming_replica port=5432 sslkey=/controller/certificates/streaming_replica.key sslcert=/controller/certificates/streaming_replica.crt sslrootcert=/controller/certificates/server-ca.crt application_name=postgres-1 sslmode=verify-ca dbname=postgres","--target-pgdata","/var/lib/postgresql/data/pgdata","--restore-target-wal"],"error":"exit status 1","stacktrace":"github.com/cloudnative-pg/cloudnative-pg/pkg/management/log.(*logger).Error\n\tpkg/management/log/log.go:128\ngithub.com/cloudnative-pg/cloudnative-pg/pkg/management/postgres.(*Instance).Rewind\n\tpkg/management/postgres/instance.go:972\ngithub.com/cloudnative-pg/cloudnative-pg/internal/management/controller.(*InstanceReconciler).verifyPgDataCoherenceForPrimary\n\tinternal/management/controller/instance_startup.go:235\ngithub.com/cloudnative-pg/cloudnative-pg/internal/management/controller.(*InstanceReconciler).initialize\n\tinternal/management/controller/instance_controller.go:342\ngithub.com/cloudnative-pg/cloudnative-pg/internal/management/controller.(*InstanceReconciler).Reconcile\n\tinternal/management/controller/instance_controller.go:141\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:119\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:316\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"} {"level":"info","ts":"2024-01-10T10:58:28Z","msg":"pg_rewind failed, starting the server to complete the crash recovery","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"postgres","namespace":"stg"},"namespace":"stg","name":"postgres","reconcileID":"b6dbc5ac-aab1-4528-bc0c-f67aa8dac9b1","uuid":"2f587c83-afa7-11ee-9aef-929973a308f2","logging_pod":"postgres-1","err":"error executing pg_rewind: exit status 1"} {"level":"info","ts":"2024-01-10T10:58:28Z","msg":"Waiting for server to complete crash recovery","logging_pod":"postgres-1"} {"level":"info","ts":"2024-01-10T10:58:28Z","msg":"Starting up instance","logging_pod":"postgres-1","pgdata":"/var/lib/postgresql/data/pgdata","options":["start","-w","-D","/var/lib/postgresql/data/pgdata","-o","-c port=5432 -c unix_socket_directories=/controller/run","-t 40000000"]} {"level":"info","ts":"2024-01-10T10:58:28Z","logger":"pg_ctl","msg":"waiting for server to start....2024-01-10 10:58:28.483 UTC [348] LOG: redirecting log output to logging collector process","pipe":"stdout","logging_pod":"postgres-1"} {"level":"info","ts":"2024-01-10T10:58:28Z","logger":"pg_ctl","msg":"2024-01-10 10:58:28.483 UTC [348] HINT: Future log output will appear in directory \"/controller/log\".","pipe":"stdout","logging_pod":"postgres-1"} {"level":"info","ts":"2024-01-10T10:58:28Z","logger":"postgres","msg":"2024-01-10 10:58:28.484 UTC [348] LOG: ending log output to stderr","source":"/controller/log/postgres","logging_pod":"postgres-1"} {"level":"info","ts":"2024-01-10T10:58:28Z","logger":"postgres","msg":"2024-01-10 10:58:28.484 UTC [348] HINT: Future log output will go to log destination \"csvlog\".","source":"/controller/log/postgres","logging_pod":"postgres-1"} {"level":"info","ts":"2024-01-10T10:58:28Z","logger":"postgres","msg":"record","logging_pod":"postgres-1","record":{"log_time":"2024-01-10 10:58:28.484 UTC","process_id":"348","session_id":"659e7854.15c","session_line_num":"1","session_start_time":"2024-01-10 10:58:28 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"ending log output to stderr","hint":"Future log output will go to log destination \"csvlog\".","backend_type":"postmaster","query_id":"0"}} {"level":"info","ts":"2024-01-10T10:58:28Z","logger":"postgres","msg":"record","logging_pod":"postgres-1","record":{"log_time":"2024-01-10 10:58:28.484 UTC","process_id":"348","session_id":"659e7854.15c","session_line_num":"2","session_start_time":"2024-01-10 10:58:28 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"starting PostgreSQL 16.1 (Debian 16.1-1.pgdg110+1) on x86_64-pc-linux-gnu, compiled by gcc (Debian 10.2.1-6) 10.2.1 20210110, 64-bit","backend_type":"postmaster","query_id":"0"}} {"level":"info","ts":"2024-01-10T10:58:28Z","logger":"postgres","msg":"record","logging_pod":"postgres-1","record":{"log_time":"2024-01-10 10:58:28.484 UTC","process_id":"348","session_id":"659e7854.15c","session_line_num":"3","session_start_time":"2024-01-10 10:58:28 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"listening on IPv4 address \"0.0.0.0\", port 5432","backend_type":"postmaster","query_id":"0"}} {"level":"info","ts":"2024-01-10T10:58:28Z","logger":"postgres","msg":"record","logging_pod":"postgres-1","record":{"log_time":"2024-01-10 10:58:28.484 UTC","process_id":"348","session_id":"659e7854.15c","session_line_num":"4","session_start_time":"2024-01-10 10:58:28 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"listening on IPv6 address \"::\", port 5432","backend_type":"postmaster","query_id":"0"}} {"level":"info","ts":"2024-01-10T10:58:28Z","logger":"postgres","msg":"record","logging_pod":"postgres-1","record":{"log_time":"2024-01-10 10:58:28.488 UTC","process_id":"348","session_id":"659e7854.15c","session_line_num":"5","session_start_time":"2024-01-10 10:58:28 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"listening on Unix socket \"/controller/run/.s.PGSQL.5432\"","backend_type":"postmaster","query_id":"0"}} {"level":"info","ts":"2024-01-10T10:58:28Z","logger":"postgres","msg":"record","logging_pod":"postgres-1","record":{"log_time":"2024-01-10 10:58:28.495 UTC","process_id":"352","session_id":"659e7854.160","session_line_num":"1","session_start_time":"2024-01-10 10:58:28 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"database system was shut down at 2024-01-10 08:19:35 UTC","backend_type":"startup","query_id":"0"}} {"level":"info","ts":"2024-01-10T10:58:28Z","logger":"postgres","msg":"record","logging_pod":"postgres-1","record":{"log_time":"2024-01-10 10:58:28.495 UTC","process_id":"352","session_id":"659e7854.160","session_line_num":"2","session_start_time":"2024-01-10 10:58:28 UTC","transaction_id":"0","error_severity":"FATAL","sql_state_code":"XX000","message":"could not access status of transaction 36923","detail":"Could not read from file \"pg_xact/0000\" at offset 8192: read too few bytes.","backend_type":"startup","query_id":"0"}} {"level":"info","ts":"2024-01-10T10:58:28Z","logger":"postgres","msg":"record","logging_pod":"postgres-1","record":{"log_time":"2024-01-10 10:58:28.496 UTC","process_id":"348","session_id":"659e7854.15c","session_line_num":"6","session_start_time":"2024-01-10 10:58:28 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"startup process (PID 352) exited with exit code 1","backend_type":"postmaster","query_id":"0"}} {"level":"info","ts":"2024-01-10T10:58:28Z","logger":"postgres","msg":"record","logging_pod":"postgres-1","record":{"log_time":"2024-01-10 10:58:28.497 UTC","process_id":"348","session_id":"659e7854.15c","session_line_num":"7","session_start_time":"2024-01-10 10:58:28 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"aborting startup due to startup process failure","backend_type":"postmaster","query_id":"0"}} {"level":"info","ts":"2024-01-10T10:58:28Z","logger":"postgres","msg":"record","logging_pod":"postgres-1","record":{"log_time":"2024-01-10 10:58:28.497 UTC","process_id":"348","session_id":"659e7854.15c","session_line_num":"8","session_start_time":"2024-01-10 10:58:28 UTC","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"database system is shut down","backend_type":"postmaster","query_id":"0"}} {"level":"info","ts":"2024-01-10T10:58:28Z","logger":"pg_ctl","msg":" stopped waiting","pipe":"stdout","logging_pod":"postgres-1"} {"level":"info","ts":"2024-01-10T10:58:28Z","logger":"pg_ctl","msg":"pg_ctl: could not start server","pipe":"stderr","logging_pod":"postgres-1"} {"level":"info","ts":"2024-01-10T10:58:28Z","logger":"pg_ctl","msg":"Examine the log output.","pipe":"stderr","logging_pod":"postgres-1"} {"level":"info","ts":"2024-01-10T10:58:28Z","msg":"Exited log pipe","fileName":"/controller/log/postgres.csv","logging_pod":"postgres-1"} {"level":"error","ts":"2024-01-10T10:58:28Z","msg":"Reconciler error","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"postgres","namespace":"stg"},"namespace":"stg","name":"postgres","reconcileID":"b6dbc5ac-aab1-4528-bc0c-f67aa8dac9b1","error":"while activating instance: error starting PostgreSQL instance: exit status 1","stacktrace":"sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:329\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:266\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\tpkg/mod/sigs.k8s.io/controller-runtime@v0.16.3/pkg/internal/controller/controller.go:227"}Code of Conduct
The text was updated successfully, but these errors were encountered: