[Bug]: When the master node disk is insufficient, the disk space is increased, the master node pod is deleted, and the master node pod is lost.

[wupengxiang23]

Is there an existing issue already for this bug?

• I have searched for an existing issue, and could not find anything. I believe this is a new bug.

I have read the troubleshooting guide

• I have read the troubleshooting guide and I think this is a new bug.

I am running a supported version of CloudNativePG

• I have read the troubleshooting guide and I think this is a new bug.

Contact Details

wupengxiang14@gmail.com

Version

1.25 (latest patch)

What version of Kubernetes are you using?

1.29

What is your Kubernetes environment?

Self-managed: k3s

How did you install the operator?

Helm

What happened?

I set up a cluster and found that the disk was insufficient after running for a while. After cleaning the disk, I deleted the Primary instance pod and found that the Primary instance pod was not recreated.

After kubectl get pod, I found that there is no cloud-ssd-pg-1 pod

cloud-ssd-pg-2                                        0/1     Running                      2318 (122m ago)
cloud-ssd-pg-3                                        0/1     Running                      2323 (73m ago)

The status of the cluster is as follows:

Cluster Summary
Name                cloud-ssd-pg
PostgreSQL Image:    ghcr.io/cloudnative-pg/postgresql:16.8
Primary instance:    cloud-ssd-pg-1
Primary start time:  2025-04-21 16:04:57 +0800 CST (uptime 388h20m42s)
Status:              Not enough disk space Insufficient disk space detected in one or more pods is preventing PostgreSQL from running.Please verify your storage settings. Further information inside .status.instancesReportedState
Instances:           3
Ready instances:     0
Size:                container not found

Continuous Backup status
Not configured

Physical backups
Primary instance not found

Streaming Replication status
Primary instance not found

Instances status
Name            Current LSN  Replication role  Status  QoS            Manager Version  Node
----            -----------  ----------------  ------  ---            ---------------  ----
cloud-ssd-pg-2  -            -                 -       InternalError  Burstable        -  178.10.4.220
cloud-ssd-pg-3  -            -                 -       InternalError  Burstable        -  178.10.4.162


Error(s) extracting status
-----------------------------------
failed to get status by proxying to the pod, you might lack permissions to get pods/proxy: an error on the server ("failed to connect to `user=postgres database=postgres`: /controller/run/.s.PGSQL.5432 (/controller/run): server error: FATAL: the database system is not yet accepting connections (SQLSTATE 57P03)") has prevented the request from succeeding (get pods https:cloud-ssd-pg-2:8000)
failed to get status by proxying to the pod, you might lack permissions to get pods/proxy: an error on the server ("failed to connect to `user=postgres database=postgres`: /controller/run/.s.PGSQL.5432 (/controller/run): server error: FATAL: the database system is starting up (SQLSTATE 57P03)") has prevented the request from succeeding (get pods https:cloud-ssd-pg-3:8000)

Why can't I recreate the master pod after deleting it? What should I do? Please help me. Thanks.

Cluster resource

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: cloud-ssd-pg
spec:
  description: "cloud-ssd-pg cluster"
  imageName:  ghcr.io/cloudnative-pg/postgresql:16.8
  instances: 3
  startDelay: 300
  stopDelay: 300
  failoverDelay: 10    
  switchoverDelay: 30 
  primaryUpdateStrategy: unsupervised

  postgresql:
    parameters:
      max_parallel_workers: "16" 
      max_parallel_maintenance_workers: "4"
      max_parallel_workers_per_gather: "4"
      max_connections: "500"
      shared_buffers: "4GB"  
      work_mem: "512MB"
      maintenance_work_mem: "5GB" 
      max_worker_processes: "19"
      wal_buffers: "128MB"
      checkpoint_completion_target: "0.9"
      synchronous_commit: "on"
      random_page_cost: "1.1"
      effective_cache_size: "8GB"
      pg_stat_statements.max: "10000"
      pg_stat_statements.track: "all"
      auto_explain.log_min_duration: "10s"
      hnsw.iterative_scan: "strict_order"
      hnsw.ef_search: "400"
      autovacuum: "off"
      ssl_min_protocol_version: "TLSv1.2"
      ssl_max_protocol_version: "TLSv1.3"
      max_wal_size: "80GB"
      min_wal_size: "20GB"
      wal_keep_size: "60GB"
      checkpoint_timeout: "30min"
      idle_session_timeout: "10min"
      idle_in_transaction_session_timeout: "5min"
      reserved_connections: "10"
      superuser_reserved_connections: "5"
      tcp_keepalives_idle: "60"
      tcp_keepalives_interval: "60"
      tcp_keepalives_count: "5"
      tcp_user_timeout: "30000"
      client_connection_check_interval: "10000"
      effective_io_concurrency: "2" 
  bootstrap:
    initdb:
      database: app
      owner: app
      secret:
        name: cloud-ssd-pg-user-app
  enableSuperuserAccess: true
  superuserSecret:
    name: cloud-ssd-pg-superuser
  storage:
    storageClass: local-storage
    size: 500Gi


  resources:
    requests:
      memory: "8Gi"
      cpu: "4"
    limits:
      memory: "16Gi"
      cpu: "8"

  affinity:
    enablePodAntiAffinity: true
    topologyKey: kubernetes.io/hostname
    podAntiAffinityType: required
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: cloud-ssd-pg-app
                operator: In
                values:
                  - cluster1
                  - cluster2
                  - cluster3
      preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            preference:
              matchExpressions:
              - key: cloud-ssd-pg-app
                operator: In
                values: ["cluster1"]
          - weight: 80
            preference:
              matchExpressions:
              - key: cloud-ssd-pg-app
                operator: In
                values: ["cluster2"]
          - weight: 10
            preference:
              matchExpressions:
              - key: cloud-ssd-pg-app
                operator: In
                values: ["cluster3"]

  monitoring:
    enablePodMonitor: true

Relevant log output

Code of Conduct

• I agree to follow this project's Code of Conduct

[dosubot]

Hello @wupengxiang23! 👋 I'm Dosu, a bot here to help you tackle bugs, answer questions, and assist you on your journey as a contributor. I'm ready to assist while you wait for a human maintainer. Let's dive in! 🚀

I found a similar discussion thread titled "Pods are not getting created after I delete the pvc and pod" which is still open and might be relevant to your issue ^[1].

Regarding your issue, the master pod in a CloudNativePG cluster cannot be recreated after deletion due to insufficient disk space because the lack of space prevents PostgreSQL from running. To resolve this issue, you should increase the size of the WAL PVC using the kubectl patch command. For example:

kubectl patch pvc <cluster-name>-wal -n <namespace> -p '{"spec":{"resources":{"requests":{"storage":"32Gi"}}}}'

Ensure that your storage class supports volume expansion. You can verify this by checking the allowVolumeExpansion field in your storage class configuration. If your storage class supports online volume resizing, the change is immediately applied to the pods. If not, you must delete the pod to trigger the resize, proceeding one pod at a time, starting from replicas and waiting for each pod to be back up ^[2][3].

Once the disk space issue is resolved, the pod should restart, and the cluster should become healthy. Monitoring disk usage is crucial to prevent such issues, as exhausted disk storage can lead to a database server shutdown ^[2].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[wupengxiang23]

My storage class does not support online volume resizing. I need to delete the Pod to trigger the resizing. Why does the Pod not restart after deleting it? Do I need to delete cloud-ssd-pg-2 and cloud-ssd-pg-3 as well? @dosu

[dosubot]

The Pod does not restart after deleting it because your storage class does not support online volume resizing, requiring you to recreate all the PVCs and Pods to move the cluster to a different storage area. You should disable the resizeInUseVolumes flag in your cluster configuration to prevent the operator from changing the existing PVCs. Yes, you need to delete cloud-ssd-pg-2 and cloud-ssd-pg-3 as well, as the operation requires recreating all instances in different PVCs. Ensure that you leave the resizing of the disk associated with the primary instance as the last step, after promoting a new resized Pod to primary using kubectl cnpg promote. This process is feasible only when the cluster contains more than one node ^[1][2][3].

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[wupengxiang23]

After I set the pvc configuration '{"spec":{"resources":{"requests":{"storage":"1000Gi"}}}}', the disk capacity is sufficient, but the operator log still has insufficient disk exceptions, and the Primary instance pod is not restarted. @dosu
operator log ：

stderr F {"level":"info","ts":"2025-05-07T21:06:45.57147485+08:00","logger":"ensure_sufficient_disk_space","msg":"Insufficient disk space detected in a pod. PostgreSQL cannot proceed until the PVC group is enlarged","controller":"cluster","controllerGroup":"postgresql.cnpg.io","controllerKind":"Cluster","Cluster":{"name":"cloud-ssd-pg","namespace":"default"},"namespace":"default","name":"cloud-ssd-pg","reconcileID":"dd402f7b-5872-4fe6-a1d0-e72d00fdf4fd","instanceNames":["cloud-ssd-pg-2","cloud-ssd-pg-3"]}

[wupengxiang23]

有办法 帮我手动创建出cloud-ssd-pg-1 主实例节点pod吗？目前cloud-ssd-pg-2 和 cloud-ssd-pg-3 启动不成功是因为cloud-ssd-pg-1 主实例节点pod不存在。
cloud-ssd-pg-2 log：

{"level":"info","ts":"2025-05-08T11:44:21.542345727+08:00","logger":"postgres","msg":"record","logging_pod":"cloud-ssd-pg-2","record":{"log_time":"2025-05-08 11:44:21.542 CST","process_id":"583351","session_id":"681c2895.8e6b7","session_line_num":"1","session_start_time":"2025-05-08 11:44:21 CST","transaction_id":"0","error_severity":"FATAL","sql_state_code":"08006","message":"could not connect to the primary server: connection to server at \"cloud-ssd-pg-rw\" , port 5432 failed: Connection refused\n\tIs the server running on that host and accepting TCP/IP connections?","backend_type":"walreceiver","query_id":"0"}}
{"level":"info","ts":"2025-05-08T11:44:21.672475826+08:00","logger":"postgres","msg":"record","logging_pod":"cloud-ssd-pg-2","record":{"log_time":"2025-05-08 11:44:21.672 CST","process_id":"39","session_id":"681b3851.27","session_line_num":"12298","session_start_time":"2025-05-07 18:39:13 CST","virtual_transaction_id":"1/0","transaction_id":"0","error_severity":"LOG","sql_state_code":"00000","message":"waiting for WAL to become available at 1A3/AF002000","backend_type":"startup","query_id":"0"}}
{"level":"info","ts":"2025-05-08T11:44:22.039941523+08:00","logger":"postgres","msg":"record","logging_pod":"cloud-ssd-pg-2","record":{"log_time":"2025-05-08 11:44:22.039 CST","user_name":"postgres","database_name":"postgres","process_id":"583365","connection_from":"[local]","session_id":"681c2896.8e6c5","session_line_num":"1","session_start_time":"2025-05-08 11:44:22 CST","transaction_id":"0","error_severity":"FATAL","sql_state_code":"57P03","message":"the database system is not yet accepting connections","detail":"Consistent recovery state has not been yet reached.","backend_type":"client backend","query_id":"0"}}

@dosu

[dosubot]

To manually create a primary instance pod in a CloudNativePG cluster when it is not automatically recreated, you can follow these steps:

1. Ensure Disk Space is Sufficient: Verify that the PVC size has been increased and that the storage class supports the changes. If the operator still logs insufficient disk space, double-check the PVC configuration and ensure that all pods have the updated storage size.

2. Delete Existing Pods: Since your storage class does not support online resizing, you need to delete the existing pods (cloud-ssd-pg-2 and cloud-ssd-pg-3) to allow the PVCs to resize. Use the following command:

   kubectl delete pod <pod-name> -n <namespace>

3. Recreate the Primary Pod: If the primary pod (cloud-ssd-pg-1) is missing, you can manually create it by scaling the cluster down and then back up. First, scale down the cluster:

   kubectl scale cluster cloud-ssd-pg --replicas=0 -n <namespace>

4. Scale the Cluster Back Up: After scaling down, scale the cluster back up to the desired number of instances:

   kubectl scale cluster cloud-ssd-pg --replicas=3 -n <namespace>

5. Monitor the Cluster: Check the status of the pods and ensure that the primary instance is recreated. Use the following command to monitor the pods:

   kubectl get pods -n <namespace>

6. Check Logs for Errors: If the primary pod is still not created, check the operator logs and the logs of the remaining pods for any errors that might indicate why the primary pod is not being recreated.

These steps should help in manually recreating the primary instance pod. If issues persist, ensure that all configurations are correct and that there are no underlying issues with the Kubernetes environment or the storage setup.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[wupengxiang23]

hi @dosu
I tried to use your method. When recreating the master Pod in step 3, --replicas=0 is incorrect. It needs to be greater than or equal to 1.

The Cluster "cloud-ssd-pg" is invalid: spec.instances: Invalid value: 0: spec.instances in body should be greater than or equal to 1

[dosubot]

To recreate the primary instance pod, you should adjust the number of replicas without setting it to zero. Instead, try scaling down to one replica and then scaling back up:

1. Scale Down to One Replica:

   kubectl scale cluster cloud-ssd-pg --replicas=1 -n <namespace>

2. Scale Back Up to Desired Number:

   kubectl scale cluster cloud-ssd-pg --replicas=3 -n <namespace>

This approach ensures that at least one pod remains active, allowing the cluster to maintain its state while resizing the PVCs. After scaling back up, monitor the pods to confirm that the primary instance is recreated. If issues persist, check the logs for any errors or warnings that might indicate further problems.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Bug Report | Other}

[github-actions]

This issue was closed because it has been inactive for 14 days since being marked as stale.