permissions error

[bilemedimkq]

I started getting these errors on the master node. database is not working.

I think it might have something to do with this but I don't know: zalando/postgres-operator#972

pgo client version 4.7.0
pgo-apiserver version 4.7.0

2021-07-23 07:04:35,320 INFO: Lock owner: chk-db-idjt-7c89cfb6f4-dgqp4; I am chk-db-idjt-7c89cfb6f4-dgqp4
2021-07-23 07:04:35,320 INFO: Lock owner: chk-db-idjt-7c89cfb6f4-dgqp4; I am chk-db-idjt-7c89cfb6f4-dgqp4
2021-07-23 07:04:35,321 INFO: starting as readonly because i had the session lock
2021-07-23 07:04:35.461 UTC [3639730] FATAL:  data directory "/pgdata/chk-db-idjt" has invalid permissions
2021-07-23 07:04:35.461 UTC [3639730] DETAIL:  Permissions should be u=rwx (0700) or u=rwx,g=rx (0750).
2021-07-23 07:04:35,467 INFO: postmaster pid=3639730
/tmp:5432 - no response
2021-07-23 07:04:35,474 WARNING: Postgresql is not running.
2021-07-23 07:04:35,474 INFO: Lock owner: chk-db-idjt-7c89cfb6f4-dgqp4; I am chk-db-idjt-7c89cfb6f4-dgqp4
2021-07-23 07:04:35,532 INFO: Reaped pid=3639732, exit status=0
2021-07-23 07:04:35,533 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202007201
  Database system identifier: 6969939212774363298
  Database cluster state: in production
  pg_control last modified: Thu Jul 22 20:36:42 2021
  Latest checkpoint location: 5F/3C000060
  Latest checkpoint's REDO location: 5F/3C000028
  Latest checkpoint's REDO WAL file: 000000070000005F0000003C
  Latest checkpoint's TimeLineID: 7
  Latest checkpoint's PrevTimeLineID: 7
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:100176
  Latest checkpoint's NextOID: 49152
  Latest checkpoint's NextMultiXactId: 1428
  Latest checkpoint's NextMultiOffset: 3114
  Latest checkpoint's oldestXID: 479
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 100176
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Thu Jul 22 20:36:41 2021
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: logical
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 6
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: f48d37ed6cda9f992727c8eaf81c0478bff3fb0f547160a421bfae54644bcc06

(I didn't do anything 12 hours ago)

[bilemedimkq]

I deleted the chk-db-idjt-7c89cfb6f4-dgqp4 pod and everything seems to be working fine now. but what is the cause of this problem? It is unacceptable to have such an error in the production environment.

[bilemedimkq]

I get a similar error in another database:

2021-07-23 07:38:33,446 INFO: Lock owner: sm-db-xhen-59cd9db585-q5kkp; I am sm-db-hglo-55f6797f74-9bnnm
2021-07-23 07:38:33,452 INFO: Reaped pid=1755106, exit status=0
2021-07-23 07:38:33,456 INFO: Reaped pid=1755107, exit status=0
2021-07-23 07:38:33,456 INFO: Local timeline=None lsn=None
2021-07-23 07:38:33,456 INFO: Lock owner: sm-db-xhen-59cd9db585-q5kkp; I am sm-db-hglo-55f6797f74-9bnnm
2021-07-23 07:38:33,457 INFO: starting as a secondary
2021-07-23 07:38:33,457 ERROR: Exception when saving file: /pgdata/sm-db-hglo/patroni.dynamic.json
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/patroni/config.py", line 164, in save_cache
    (fd, tmpfile) = tempfile.mkstemp(prefix=self.__CACHE_FILENAME, dir=self._data_dir)
  File "/usr/lib64/python3.6/tempfile.py", line 344, in mkstemp
    return _mkstemp_inner(dir, prefix, suffix, flags, output_type)
  File "/usr/lib64/python3.6/tempfile.py", line 262, in _mkstemp_inner
    fd = _os.open(file, flags, 0o600)
OSError: [Errno 30] Read-only file system: '/pgdata/sm-db-hglo/patroni.dynamic.jsonwt1ccbtu'
2021-07-23 07:38:33,457 ERROR: Exception during execution of long running task restarting after failure
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/patroni/async_executor.py", line 97, in run
    wakeup = func(*args) if args else func()
  File "/usr/local/lib/python3.6/site-packages/patroni/postgresql/__init__.py", line 768, in follow
    self.config.write_recovery_conf(recovery_params)
  File "/usr/local/lib/python3.6/site-packages/patroni/postgresql/config.py", line 761, in write_recovery_conf
    open(self._standby_signal, 'w').close()
OSError: [Errno 30] Read-only file system: '/pgdata/sm-db-hglo/standby.signal'
2021-07-23 07:38:33,458 WARNING: Postgresql is not running.
2021-07-23 07:38:33,458 INFO: Lock owner: sm-db-xhen-59cd9db585-q5kkp; I am sm-db-hglo-55f6797f74-9bnnm
2021-07-23 07:38:33,460 INFO: Reaped pid=1755109, exit status=0
2021-07-23 07:38:33,460 INFO: pg_controldata:
  pg_control version number: 1300
  Catalog version number: 202007201
  Database system identifier: 6969521368737177764
  Database cluster state: in production
  pg_control last modified: Thu Jul 22 18:40:41 2021
  Latest checkpoint location: F5/C90022B8
  Latest checkpoint's REDO location: F5/C80156E8
  Latest checkpoint's REDO WAL file: 00000009000000F5000000C8
  Latest checkpoint's TimeLineID: 9
  Latest checkpoint's PrevTimeLineID: 9
  Latest checkpoint's full_page_writes: on
  Latest checkpoint's NextXID: 0:2461375
  Latest checkpoint's NextOID: 34923
  Latest checkpoint's NextMultiXactId: 338
  Latest checkpoint's NextMultiOffset: 750
  Latest checkpoint's oldestXID: 479
  Latest checkpoint's oldestXID's DB: 1
  Latest checkpoint's oldestActiveXID: 2461375
  Latest checkpoint's oldestMultiXid: 1
  Latest checkpoint's oldestMulti's DB: 1
  Latest checkpoint's oldestCommitTsXid: 0
  Latest checkpoint's newestCommitTsXid: 0
  Time of latest checkpoint: Thu Jul 22 18:40:30 2021
  Fake LSN counter for unlogged rels: 0/3E8
  Minimum recovery ending location: 0/0
  Min recovery ending loc's timeline: 0
  Backup start location: 0/0
  Backup end location: 0/0
  End-of-backup record required: no
  wal_level setting: logical
  wal_log_hints setting: on
  max_connections setting: 100
  max_worker_processes setting: 8
  max_wal_senders setting: 6
  max_prepared_xacts setting: 0
  max_locks_per_xact setting: 64
  track_commit_timestamp setting: off
  Maximum data alignment: 8
  Database block size: 8192
  Blocks per segment of large relation: 131072
  WAL block size: 8192
  Bytes per WAL segment: 16777216
  Maximum length of identifiers: 64
  Maximum columns in an index: 32
  Maximum size of a TOAST chunk: 1996
  Size of a large-object chunk: 2048
  Date/time type storage: 64-bit integers
  Float8 argument passing: by value
  Data page checksum version: 1
  Mock authentication nonce: 85cd640a363a71d485f68734540e89d6bfd79b2c19152ed87c7ba95937b6afb8

[jkatz]

OSError: [Errno 30] Read-only file system

There is an issue with your underlying storage system as it's been placed into read-only mode. You need to check to see what is wrong with it. Perhaps you are out of space, or there is some other issue.

This is not an issue with the Operator itself.

[bilemedimkq]

No, there is no problem in the storage part. I'm sure of that, the postgresql operator isn't the only application that works, there are loads of applications that use storage and others have no problem.

Can you look at the first log I posted, not the second log?
@jkatz

[bilemedimkq]

Here is the output of kubectl describe command @jkatz

> kubectl describe pod  sm-db-hglo-55f6797f74-9bnnm -n sm-pgo-db
Name:         sm-db-hglo-55f6797f74-9bnnm
Namespace:    sm-pgo-db
Priority:     0
Node:         n2-alpha-node/<IPADDRESS>
Start Time:   Mon, 07 Jun 2021 22:39:18 +0000
Labels:       crunchy-pgha-scope=sm-db
              deployment-name=sm-db-hglo
              name=sm-db-replica
              pg-cluster=sm-db
              pg-pod-anti-affinity=preferred
              pgo-pg-database=true
              pgo-version=4.7.0
              pod-template-hash=55f6797f74
              service-name=sm-db-replica
              vendor=crunchydata
              workflowid=59952df1-b1a0-4660-a1bb-b9c7ed2688a4
Annotations:  cni.projectcalico.org/podIP: 10.42.2.166/32
              cni.projectcalico.org/podIPs: 10.42.2.166/32
              kubernetes.io/psp: global-unrestricted-psp
              status:
                {"conn_url":"postgres://10.42.2.166:5432/postgres","api_url":"http://10.42.2.166:8009/patroni","state":"crashed","role":"demoted","version...
Status:       Running
IP:           10.42.2.166
IPs:
  IP:           10.42.2.166
Controlled By:  ReplicaSet/sm-db-hglo-55f6797f74
Containers:
  database:
    Container ID:   containerd://672e2f5cd4684119fb623860e8720429c007e7024bbd0e939f072f84903a3f94
    Image:          registry.developers.crunchydata.com/crunchydata/crunchy-postgres-ha:centos8-13.3-4.7.0
    Image ID:       registry.developers.crunchydata.com/crunchydata/crunchy-postgres-ha@sha256:3306cb1c16b5f4afd9a1ada13ad45f61b368d90267d9513302108ec80d0db1bf
    Ports:          5432/TCP, 8009/TCP
    Host Ports:     0/TCP, 0/TCP
    State:          Running
      Started:      Mon, 07 Jun 2021 22:39:37 +0000
    Ready:          False
    Restart Count:  0
    Limits:
      memory:  6Gi
    Requests:
      memory:   4Gi
    Liveness:   exec [/opt/crunchy/bin/postgres-ha/health/pgha-liveness.sh] delay=30s timeout=10s period=15s #success=1 #failure=3
    Readiness:  exec [/opt/crunchy/bin/postgres-ha/health/pgha-readiness.sh] delay=15s timeout=1s period=10s #success=1 #failure=3
    Environment:
      MODE:                               postgres
      PGHA_PG_PORT:                       5432
      PGHA_USER:                          postgres
      PATRONI_POSTGRESQL_DATA_DIR:        /pgdata/sm-db-hglo
      PGBACKREST_STANZA:                  db
      PGBACKREST_REPO1_HOST:              sm-db-backrest-shared-repo
      BACKREST_SKIP_CREATE_STANZA:        true
      PGHA_PGBACKREST:                    true
      PGBACKREST_REPO1_PATH:              /backrestrepo/sm-db-backrest-shared-repo
      PGBACKREST_DB_PATH:                 /pgdata/sm-db-hglo
      ENABLE_SSHD:                        true
      PGBACKREST_LOG_PATH:                /tmp
      PGBACKREST_PG1_SOCKET_PATH:         /tmp
      PGBACKREST_PG1_PORT:                5432
      PGBACKREST_REPO1_TYPE:              posix
      PGHA_PGBACKREST_LOCAL_S3_STORAGE:   false
      PGHA_PGBACKREST_LOCAL_GCS_STORAGE:  false
      PGHA_DATABASE:                      sm-db
      PGHA_REPLICA_REINIT_ON_START_FAIL:  true
      PGHA_SYNC_REPLICATION:              false
      PGHA_TLS_ENABLED:                   false
      PGHA_TLS_ONLY:                      false
      PGHA_PASSWORD_TYPE:                 
      PGHA_STANDBY:                       false
      PATRONI_KUBERNETES_NAMESPACE:       sm-pgo-db (v1:metadata.namespace)
      PATRONI_KUBERNETES_SCOPE_LABEL:     crunchy-pgha-scope
      PATRONI_SCOPE:                       (v1:metadata.labels['crunchy-pgha-scope'])
      PATRONI_KUBERNETES_LABELS:          {vendor: "crunchydata"}
      PATRONI_LOG_LEVEL:                  INFO
      PGHOST:                             /tmp
      LD_PRELOAD:                         /usr/lib64/libnss_wrapper.so
      NSS_WRAPPER_PASSWD:                 /tmp/nss_wrapper/postgres/passwd
      NSS_WRAPPER_GROUP:                  /tmp/nss_wrapper/postgres/group
    Mounts:
      /dev/shm from dshm (rw)
      /etc/pgbackrest/conf.d from pgbackrest-config (rw)
      /etc/podinfo from podinfo (rw)
      /etc/ssh from ssh-config (ro)
      /pgconf from pgconf-volume (rw)
      /pgconf/pgreplicator from primary-volume (rw)
      /pgconf/pgsuper from root-volume (rw)
      /pgconf/pguser from user-volume (rw)
      /pgdata from pgdata (rw)
      /sshd from sshd (ro)
      /tmp from tmp (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from pgo-pg-token-bh7f7 (ro)
Conditions:
  Type              Status
  Initialized       True 
  Ready             False 
  ContainersReady   False 
  PodScheduled      True 
Volumes:
  pgdata:
    Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
    ClaimName:  sm-db-hglo
    ReadOnly:   false
  user-volume:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  sm-db-testuser-secret
    Optional:    false
  primary-volume:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  sm-db-primaryuser-secret
    Optional:    false
  sshd:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  sm-db-backrest-repo-config
    Optional:    false
  ssh-config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  sm-db-backrest-repo-config
    Optional:    false
  root-volume:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  sm-db-postgres-secret
    Optional:    false
  report:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  64Mi
  dshm:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  <unset>
  tmp:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:     Memory
    SizeLimit:  16Mi
  pgbackrest-config:
    Type:                Projected (a volume that contains injected data from multiple sources)
    ConfigMapName:       sm-db-config-backrest
    ConfigMapOptional:   0xc000d3cb28
    SecretName:          sm-db-config-backrest
    SecretOptionalName:  0xc000d3cb29
  pgconf-volume:
    Type:               Projected (a volume that contains injected data from multiple sources)
    ConfigMapName:      sm-db-pgha-config
    ConfigMapOptional:  0xc000d3cb4d
  podinfo:
    Type:  DownwardAPI (a volume populated by information about the pod)
    Items:
      limits.cpu -> cpu_limit
      requests.cpu -> cpu_request
      limits.memory -> mem_limit
      requests.memory -> mem_request
      metadata.labels -> labels
      metadata.annotations -> annotations
  pgo-pg-token-bh7f7:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  pgo-pg-token-bh7f7
    Optional:    false
QoS Class:       Burstable
Node-Selectors:  <none>
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age                     From     Message
  ----     ------     ----                    ----     -------
  Warning  Unhealthy  2m56s (x9182 over 25h)  kubelet  Readiness probe failed: