[Bug]: On PersistentVolumeClaim accessModes duplicate

[albundy83]

Is there an existing issue already for this bug?

• I have searched for an existing issue, and could not find anything. I believe this is a new bug.

I have read the troubleshooting guide

• I have read the troubleshooting guide and I think this is a new bug.

I am running a supported version of CloudNativePG

• I have read the troubleshooting guide and I think this is a new bug.

Contact Details

gregoire.bellon-gervais@docaposte.fr

Version

1.23.1

What version of Kubernetes are you using?

1.27

What is your Kubernetes environment?

Self-managed: RKE

How did you install the operator?

Helm

What happened?

Hello,

(I'm under RKE2 not RKE but it was not present in the list).

when I create a Cluster with CloudNativePG, provisionned PVC contain twice ReadWriteOnce in accessModes.
Like this:

spec:
  accessModes:
    - ReadWriteOnce
    - ReadWriteOnce

Here the full yaml for created PVC for DATA:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: precious-1
  namespace: precious-pprod
  labels:
    app: precious
    app.kubernetes.io/component: database
    app.kubernetes.io/instance: precious
    app.kubernetes.io/managed-by: Kustomize
    app.kubernetes.io/name: precious
    cnpg.io/cluster: precious
    cnpg.io/instanceName: precious-1
    cnpg.io/instanceRole: primary
    cnpg.io/pvcRole: PG_DATA
    role: primary
  annotations:
    cnpg.io/nodeSerial: '1'
    cnpg.io/operatorVersion: 1.22.2
    cnpg.io/pvcStatus: ready
    pv.kubernetes.io/bind-completed: 'yes'
    pv.kubernetes.io/bound-by-controller: 'yes'
    volume.beta.kubernetes.io/storage-provisioner: bsu.csi.outscale.com
    volume.kubernetes.io/storage-provisioner: bsu.csi.outscale.com
status:
  phase: Bound
  accessModes:
    - ReadWriteOnce
    - ReadWriteOnce
  capacity:
    storage: 100Gi
spec:
  accessModes:
    - ReadWriteOnce
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi
  volumeName: pvc-xxxxxxxx-yyyy-yyyy-yyyy-xxxxxxxxxxxx
  storageClassName: io1-retain-immediate
  volumeMode: Filesystem

and the one for the WAL:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: precious-1-wal
  namespace: precious-pprod
  labels:
    app: precious
    app.kubernetes.io/component: database
    app.kubernetes.io/instance: precious
    app.kubernetes.io/managed-by: Kustomize
    app.kubernetes.io/name: precious
    cnpg.io/cluster: precious
    cnpg.io/instanceName: precious-1
    cnpg.io/instanceRole: primary
    cnpg.io/pvcRole: PG_WAL
    role: primary
  annotations:
    cnpg.io/nodeSerial: '1'
    cnpg.io/operatorVersion: 1.22.2
    cnpg.io/pvcStatus: ready
    pv.kubernetes.io/bind-completed: 'yes'
    pv.kubernetes.io/bound-by-controller: 'yes'
    volume.beta.kubernetes.io/storage-provisioner: bsu.csi.outscale.com
    volume.kubernetes.io/storage-provisioner: bsu.csi.outscale.com
status:
  phase: Bound
  accessModes:
    - ReadWriteOnce
    - ReadWriteOnce
  capacity:
    storage: 100Gi
spec:
  accessModes:
    - ReadWriteOnce
    - ReadWriteOnce
  resources:
    requests:
      storage: 100Gi
  volumeName: pvc-xxxxxxxx-yyyy-yyyy-yyyy-xxxxxxxxxxxx
  storageClassName: io1-retain-immediate
  volumeMode: Filesystem

I install CloudNativePG using helm and the following command:

helm upgrade --install cnpg cloudnative-pg \
  --repo https://cloudnative-pg.github.io/charts \
  --namespace cnpg-system \
  --create-namespace \
  --set config.secret=true \
  --set-string config.data.ENABLE_INSTANCE_MANAGER_INPLACE_UPDATES=true \
  --set config.data.INHERITED_LABELS='app.kubernetes.io/*\, app' \
  --set monitoring.grafanaDashboard.create=true \
  --set-string monitoring.grafanaDashboard.sidecarLabelValue=1

Cluster resource

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  annotations:
    argocd.argoproj.io/sync-wave: '1'
  labels:
    app: precious
    app.kubernetes.io/component: database
    app.kubernetes.io/instance: precious
    app.kubernetes.io/managed-by: Kustomize
    app.kubernetes.io/name: precious
    argocd.argoproj.io/instance: precious-pprod
spec:
  affinity:
    podAntiAffinityType: preferred
  backup:
    barmanObjectStore:
      data:
        compression: snappy
        encryption: AES256
        jobs: 2
      destinationPath: s3://cloudnative-pg-precious-pprod
      endpointURL: https://oos.region.outscale.com
      s3Credentials:
        accessKeyId:
          key: ACCESS_KEY_ID
          name: cloudnative-pg-s3
        secretAccessKey:
          key: ACCESS_SECRET_KEY
          name: cloudnative-pg-s3
      serverName: precious
      wal:
        compression: snappy
        encryption: AES256
        maxParallel: 8
    retentionPolicy: 90d
    target: prefer-standby
    volumeSnapshot:
      className: csi-osc-vsc-delete
      online: true
      onlineConfiguration:
        immediateCheckpoint: true
        waitForArchive: true
      snapshotOwnerReference: backup
  bootstrap:
    recovery:
      database: precious
      owner: precious
      source: clusterBackup
  enablePDB: true
  enableSuperuserAccess: false
  externalClusters:
    - barmanObjectStore:
        destinationPath: s3://cloudnative-pg-precious-prod
        endpointURL: https://oos.region.outscale.com
        s3Credentials:
          accessKeyId:
            key: ACCESS_KEY_ID
            name: cloudnative-pg-s3
          secretAccessKey:
            key: ACCESS_SECRET_KEY
            name: cloudnative-pg-s3
        serverName: precious
        wal:
          maxParallel: 8
      name: clusterBackup
  imageName: ghcr.io/cloudnative-pg/postgresql:16.2
  instances: 1
  logLevel: info
  maxSyncReplicas: 0
  minSyncReplicas: 0
  monitoring:
    customQueriesConfigMap:
      - key: queries
        name: cnpg-default-monitoring
    disableDefaultQueries: false
    enablePodMonitor: true
  postgresGID: 26
  postgresUID: 26
  postgresql:
    parameters:
      archive_mode: 'on'
      archive_timeout: 5min
      dynamic_shared_memory_type: posix
      log_destination: csvlog
      log_directory: /controller/log
      log_filename: postgres
      log_rotation_age: '0'
      log_rotation_size: '0'
      log_truncate_on_rotation: 'false'
      logging_collector: 'on'
      max_parallel_workers: '32'
      max_replication_slots: '32'
      max_worker_processes: '32'
      pgaudit.log: all, -misc
      pgaudit.log_catalog: 'off'
      pgaudit.log_parameter: 'on'
      pgaudit.log_relation: 'on'
      shared_memory_type: mmap
      shared_preload_libraries: ''
      ssl_max_protocol_version: TLSv1.3
      ssl_min_protocol_version: TLSv1.3
      wal_keep_size: 512MB
      wal_level: logical
      wal_log_hints: 'on'
      wal_receiver_timeout: 5s
      wal_sender_timeout: 5s
    syncReplicaElectionConstraint:
      enabled: false
  primaryUpdateMethod: restart
  primaryUpdateStrategy: unsupervised
  replicationSlots:
    highAvailability:
      enabled: true
      slotPrefix: _cnpg_
    synchronizeReplicas:
      enabled: true
    updateInterval: 30
  resources:
    limits:
      cpu: '2'
      ephemeral-storage: 1Gi
      memory: 8Gi
    requests:
      cpu: 100m
      ephemeral-storage: '0'
      memory: 128Mi
  smartShutdownTimeout: 180
  startDelay: 3600
  stopDelay: 1800
  storage:
    pvcTemplate:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 100Gi
      storageClassName: io1-retain-immediate
      volumeMode: Filesystem
    resizeInUseVolumes: true
  switchoverDelay: 3600
  walStorage:
    pvcTemplate:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 100Gi
      storageClassName: io1-retain-immediate
      volumeMode: Filesystem
    resizeInUseVolumes: true

Relevant log output

No response

Code of Conduct

• I agree to follow this project's Code of Conduct

[yacinebbt]

Hello,

I have the same isssue i've created: cloudnative-pg/charts#287, here is the details of the ticket:

Hello,

When creating a cluster and specifying the access mode type, the type is appended to a hardcoded value which is the ReadWriteOnce:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: yacine-rwm
  namespace: source
spec:
  instances: 1
  storage:
    pvcTemplate:
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 1Gi
      storageClassName: sc-cus
      volumeMode: Filesystem
  walStorage:
    pvcTemplate:
      accessModes:
        - ReadWriteMany
      resources:
        requests:
          storage: 1Gi
      storageClassName: sc-cus
      volumeMode: Filesystem

The auto created pvcs have the following specs:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: yacine-rwm-1
  namespace: source
spec:
  accessModes:
    - ReadWriteMany
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: sc-cus
  volumeMode: Filesystem
  volumeName: pvc-xx-xxxx-xxxxxx-xxxxxxxxxx-xxxxxxxxx-x

and

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: yacine-rwm-1-wal
  namespace: source
spec:
  accessModes:
    - ReadWriteOnce
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  volumeName: pvc-xx-xxxx-xxxxxx-xxxxxxxxxx-xxxxxxxxx-x
  storageClassName: sc-cus
  volumeMode: Filesystem

The same thing happen if we create a cluster with ReadWriteOnce access mode:

apiVersion: postgresql.cnpg.io/v1
kind: Cluster
metadata:
  name: yacine-rwo
  namespace: source
spec:
  instances: 1
  storage:
    pvcTemplate:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi
      storageClassName: sc-cus
      volumeMode: Filesystem
  walStorage:
    pvcTemplate:
      accessModes:
        - ReadWriteOnce
      resources:
        requests:
          storage: 1Gi
      storageClassName: sc-cus
      volumeMode: Filesystem

The pvcs created have ReadWriteOnce access mode duplicated:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: yacine-rwo-1
  namespace: source
spec:
  accessModes:
    - ReadWriteOnce
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  volumeName: pvc-xx-xxxx-xxxxxx-xxxxxxxxxx-xxxxxxxxx-x
  storageClassName: sc-cus
  volumeMode: Filesystem

and:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: yacine-rwo-1
  namespace: source
spec:
  accessModes:
    - ReadWriteOnce
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  volumeName: pvc-xx-xxxx-xxxxxx-xxxxxxxxxx-xxxxxxxxx-x
  storageClassName: sc-cus
  volumeMode: Filesystem

The same result if we mix the types, meaning that if we use RWO for storage data and RWX for wals or vice-versa, the pvcs will always have the ReadWriteOnce value + the one we configure in our cluster definition.
we can find the definition of the funtion that generate the pvc here:

cloudnative-pg/internal/cmd/plugin/fio/fio.go

Line 92 in 267e7cc

func (cmd *fioCommand) generatePVCObject() (*corev1.PersistentVolumeClaim, error) {

I think what should be done, is to check if the user don't define a value for AccessMode we keep the ReadWriteOnce as a default value, otherwise, use the defined value by the user. (https://github.com/cloudnative-pg/cloudnative-pg/blob/267e7cc1416241453c3e0a1d5ec257e806b184fd/internal/cmd/plugin/fio/fio.go#L103C4-L103C74)

Thank you for fixing this issue :).
Regards,

[albundy83]

Hello,

if you can confirm that it's a trending that you observe also please.
Just to clarify the issue, it is also a problem with prometheus and alert PrometheusDuplicateTimestamps.
Due to the fact that you fill twice with the same value, it creates from kube-state-metrics the same metrics twice.

We have prometheus log filled with:

ts=2024-05-22T11:52:17.520Z caller=scrape.go:1738 level=warn component="scrape manager" scrape_pool=serviceMonitor/monitoring/kube-prometheus-stack-kube-state-metrics/0 target=http://10.42.xx.yy:8080/metrics msg="Error on ingesting samples with different value but same timestamp" num_dropped=12

and with debug enabled:

ts=2024-05-22T11:52:17.486Z caller=scrape.go:1777 level=debug component="scrape manager" scrape_pool=serviceMonitor/monitoring/kube-prometheus-stack-kube-state-metrics/0 target=http://10.42.xx.yy:8080/metrics msg="Duplicate sample for timestamp" series="kube_persistentvolumeclaim_access_mode{namespace=\"precious-dev\",persistentvolumeclaim=\"precious-1-wal\",access_mode=\"ReadWriteOnce\"}"
ts=2024-05-22T11:52:17.487Z caller=scrape.go:1777 level=debug component="scrape manager" scrape_pool=serviceMonitor/monitoring/kube-prometheus-stack-kube-state-metrics/0 target=http://10.42.xx.yy:8080/metrics msg="Duplicate sample for timestamp" series="kube_persistentvolumeclaim_access_mode{namespace=\"precious-prod\",persistentvolumeclaim=\"precious-1-wal\",access_mode=\"ReadWriteOnce\"}"
ts=2024-05-22T11:52:17.487Z caller=scrape.go:1777 level=debug component="scrape manager" scrape_pool=serviceMonitor/monitoring/kube-prometheus-stack-kube-state-metrics/0 target=http://10.42.xx.yy:8080/metrics msg="Duplicate sample for timestamp" series="kube_persistentvolumeclaim_access_mode{namespace=\"precious-pprod\",persistentvolumeclaim=\"precious-1\",access_mode=\"ReadWriteOnce\"}"
ts=2024-05-22T11:52:17.487Z caller=scrape.go:1777 level=debug component="scrape manager" scrape_pool=serviceMonitor/monitoring/kube-prometheus-stack-kube-state-metrics/0 target=http://10.42.xx.yy:8080/metrics msg="Duplicate sample for timestamp" series="kube_persistentvolumeclaim_access_mode{namespace=\"precious-rct\",persistentvolumeclaim=\"precious-1\",access_mode=\"ReadWriteOnce\"}"
ts=2024-05-22T11:52:17.487Z caller=scrape.go:1777 level=debug component="scrape manager" scrape_pool=serviceMonitor/monitoring/kube-prometheus-stack-kube-state-metrics/0 target=http://10.42.xx.yy:8080/metrics msg="Duplicate sample for timestamp" series="kube_persistentvolumeclaim_access_mode{namespace=\"precious-rcti\",persistentvolumeclaim=\"precious-1\",access_mode=\"ReadWriteOnce\"}"
ts=2024-05-22T11:52:17.487Z caller=scrape.go:1777 level=debug component="scrape manager" scrape_pool=serviceMonitor/monitoring/kube-prometheus-stack-kube-state-metrics/0 target=http://10.42.xx.yy:8080/metrics msg="Duplicate sample for timestamp" series="kube_persistentvolumeclaim_access_mode{namespace=\"precious-dev\",persistentvolumeclaim=\"precious-1\",access_mode=\"ReadWriteOnce\"}"
ts=2024-05-22T11:52:17.487Z caller=scrape.go:1777 level=debug component="scrape manager" scrape_pool=serviceMonitor/monitoring/kube-prometheus-stack-kube-state-metrics/0 target=http://10.42.xx.yy:8080/metrics msg="Duplicate sample for timestamp" series="kube_persistentvolumeclaim_access_mode{namespace=\"precious-pprod\",persistentvolumeclaim=\"precious-1-wal\",access_mode=\"ReadWriteOnce\"}"
ts=2024-05-22T11:52:17.487Z caller=scrape.go:1777 level=debug component="scrape manager" scrape_pool=serviceMonitor/monitoring/kube-prometheus-stack-kube-state-metrics/0 target=http://10.42.xx.yy:8080/metrics msg="Duplicate sample for timestamp" series="kube_persistentvolumeclaim_access_mode{namespace=\"precious-prod\",persistentvolumeclaim=\"precious-3-wal\",access_mode=\"ReadWriteOnce\"}"
ts=2024-05-22T11:52:17.487Z caller=scrape.go:1777 level=debug component="scrape manager" scrape_pool=serviceMonitor/monitoring/kube-prometheus-stack-kube-state-metrics/0 target=http://10.42.xx.yy:8080/metrics msg="Duplicate sample for timestamp" series="kube_persistentvolumeclaim_access_mode{namespace=\"precious-rcti\",persistentvolumeclaim=\"precious-1-wal\",access_mode=\"ReadWriteOnce\"}"
ts=2024-05-22T11:52:17.487Z caller=scrape.go:1777 level=debug component="scrape manager" scrape_pool=serviceMonitor/monitoring/kube-prometheus-stack-kube-state-metrics/0 target=http://10.42.xx.yy:8080/metrics msg="Duplicate sample for timestamp" series="kube_persistentvolumeclaim_access_mode{namespace=\"precious-rct\",persistentvolumeclaim=\"precious-1-wal\",access_mode=\"ReadWriteOnce\"}"
ts=2024-05-22T11:52:17.487Z caller=scrape.go:1777 level=debug component="scrape manager" scrape_pool=serviceMonitor/monitoring/kube-prometheus-stack-kube-state-metrics/0 target=http://10.42.xx.yy:8080/metrics msg="Duplicate sample for timestamp" series="kube_persistentvolumeclaim_access_mode{namespace=\"precious-prod\",persistentvolumeclaim=\"precious-1\",access_mode=\"ReadWriteOnce\"}"
ts=2024-05-22T11:52:17.487Z caller=scrape.go:1777 level=debug component="scrape manager" scrape_pool=serviceMonitor/monitoring/kube-prometheus-stack-kube-state-metrics/0 target=http://10.42.xx.yy:8080/metrics msg="Duplicate sample for timestamp" series="kube_persistentvolumeclaim_access_mode{namespace=\"precious-prod\",persistentvolumeclaim=\"precious-3\",access_mode=\"ReadWriteOnce\"}"

Problem is referenced here and here for example.