Cluster Pod CrashLoopBackOff : Permission denied

[sorinpepelea]

On a Microk8s cluster (+3 nodes debian ) , i have installed postgres-operator and also client.

On cluster create, pod get's crashed by insuficient rights : [could not create directory "/pgdata/hippo": Permission denied]
from pod logs. I cannot gain root to chown path for future investigation.

Environment:

• Operating System: Debian Buster
• Running: Local
• Storage : Hostpath
• Platform : Microk8s
• Platform Version: 1.20

[jkatz]

• Which version of the Operator are you using?
• How did you install the Operator?
• Did you modify the default installation settings at all?
• Is disableFSGroup in the installation set to false?

Without any additional information, if you are using all of the defaults, this sounds like an issue at your storage level. Ensure that your storage provider is able to set the permissions correctly at your disk level.

[sorinpepelea]

Operator version is 4.5.1

and i installed all this with command :
kubectl apply -f https://raw.githubusercontent.com/CrunchyData/postgres-operator/v4.5.1/installers/kubectl/postgres-operator.yml
I didn't try to modify anything. And pod hippo-backrest-shared-repo is running ok. only hippo pod is crashed.
i installed cluster with client command 👍
pgo create cluster hippo.
From the log i see : initdb: error: could not create directory "/pgdata/hippo": Permission denied

[sorinpepelea]

i tried --disableFSGroup=false ... didn't work. how can i set this param to false ?

[jkatz]

disable_fsgroup is an installation time option. You can read the full installation configuration reference here: https://access.crunchydata.com/documentation/postgres-operator/latest/installation/configuration/. I would recommend reviewing this and seeing if there are any options that may help you to install in your local environment.

With that said, without logs from your various Pods, it seems like this issue is isolated either to your local Kubernetes environment or how your storage provisioner works.

[sorinpepelea]

in pgo-config configmap , DisableFSGroup is already false.

[sorinpepelea]

`search
Search
Workloads
keyboard_arrow_right
Pods
keyboard_arrow_right
grid-5fdd95ddb7-68pxh
keyboard_arrow_right
Logs
Namespace
pgo

Logs from
database
in
grid-5fdd95ddb7-68pxh
keep_existing_recovery_conf: true
dcs:
postgresql:
parameters:
unix_socket_directories: /tmp,/crunchyadm
wal_level: logical
archive_mode: on
archive_command: 'source /opt/cpm/bin/pgbackrest/pgbackrest-set-env.sh &&
pgbackrest archive-push "%p"'
archive_timeout: 60
log_directory: pg_log
shared_buffers: 128MB
temp_buffers: 8MB
log_min_duration_statement: 60000
log_statement: none
work_mem: 4MB
max_wal_senders: 6
shared_preload_libraries: pgaudit.so,pg_stat_statements.so,pgnodemx.so
use_slots: false
recovery_conf:
restore_command: 'source /opt/cpm/bin/pgbackrest/pgbackrest-set-env.sh &&
pgbackrest archive-get %f "%p"'
use_pg_rewind: true
post_bootstrap: /opt/cpm/bin/bootstrap/post-bootstrap.sh
initdb:

• encoding: UTF8
• data-checksums
  postgresql:
  use_unix_socket: true
  pgpass: /tmp/.pgpass
  create_replica_methods:
• pgbackrest
• basebackup
  pgbackrest:
  command: '/opt/cpm/bin/pgbackrest/pgbackrest-create-replica.sh replica'
  keep_data: true
  no_params: true
  pgbackrest_standby:
  command: '/opt/cpm/bin/pgbackrest/pgbackrest-create-replica.sh standby'
  keep_data: true
  no_params: true
  no_master: 1
  remove_data_directory_on_rewind_failure: true
  callbacks:
  on_role_change: /opt/cpm/bin/callbacks/pgha-on-role-change.sh
  pg_hba:
• local all postgres peer
• local all crunchyadm peer
• host replication primaryuser 0.0.0.0/0 md5
• host all primaryuser 0.0.0.0/0 reject
• host all all 0.0.0.0/0 md5
  Thu Dec 31 16:32:16 UTC 2020 INFO: Applying SSHD..
  Thu Dec 31 16:32:16 UTC 2020 INFO: Checking for SSH Host Keys in /sshd..
  Thu Dec 31 16:32:16 UTC 2020 INFO: Checking for authorized_keys in /sshd
  Thu Dec 31 16:32:16 UTC 2020 INFO: Checking for sshd_config in /sshd
  Thu Dec 31 16:32:16 UTC 2020 INFO: setting up .ssh directory
  Thu Dec 31 16:32:16 UTC 2020 INFO: Starting SSHD..
  Thu Dec 31 16:32:16 UTC 2020 INFO: Starting background process to monitor Patroni initization and restart the database if needed
  Thu Dec 31 16:32:16 UTC 2020 INFO: Initializing cluster bootstrap with command: '/usr/local/bin/patroni /tmp/postgres-ha-bootstrap.yaml'
  Thu Dec 31 16:32:16 UTC 2020 INFO: Running Patroni as PID 1
  2020-12-31 16:32:17,211 INFO: No PostgreSQL configuration items changed, nothing to reload.
  2020-12-31 16:32:17,214 INFO: Lock owner: None; I am grid-5fdd95ddb7-68pxh
  2020-12-31 16:32:17,289 INFO: trying to bootstrap a new cluster
  The files belonging to this database system will be owned by user "postgres".
  This user must also own the server process.
  The database cluster will be initialized with locale "en_US.utf-8".
  The default text search configuration will be set to "english".
  Data page checksums are enabled.
  initdb: error: could not create directory "/pgdata/grid": Permission denied
  pg_ctl: database system initialization failed
  2020-12-31 16:32:17,308 INFO: removing initialize key after failed attempt to bootstrap the cluster
  Process Process-1:
  Traceback (most recent call last):
  File "/usr/lib64/python3.6/multiprocessing/process.py", line 258, in _bootstrap
  self.run()
  File "/usr/lib64/python3.6/multiprocessing/process.py", line 93, in run
  self._target(*self._args, **self._kwargs)
  File "/usr/local/lib/python3.6/site-packages/patroni/init.py", line 138, in patroni_main
  abstract_main(Patroni, schema)
  File "/usr/local/lib/python3.6/site-packages/patroni/daemon.py", line 100, in abstract_main
  controller.run()
  File "/usr/local/lib/python3.6/site-packages/patroni/init.py", line 108, in run
  super(Patroni, self).run()
  File "/usr/local/lib/python3.6/site-packages/patroni/daemon.py", line 59, in run
  self._run_cycle()
  File "/usr/local/lib/python3.6/site-packages/patroni/init.py", line 111, in _run_cycle
  logger.info(self.ha.run_cycle())
  File "/usr/local/lib/python3.6/site-packages/patroni/ha.py", line 1452, in run_cycle
  info = self._run_cycle()
  File "/usr/local/lib/python3.6/site-packages/patroni/ha.py", line 1346, in _run_cycle
  return self.post_bootstrap()
  File "/usr/local/lib/python3.6/site-packages/patroni/ha.py", line 1242, in post_bootstrap
  self.cancel_initialization()
  File "/usr/local/lib/python3.6/site-packages/patroni/ha.py", line 1235, in cancel_initialization
  raise PatroniFatalException('Failed to bootstrap cluster')
  patroni.exceptions.PatroniFatalException: 'Failed to bootstrap cluster'
  creating directory /pgdata/grid ...
  Logs from Dec 31, 2020 to Dec 31, 2020 UTC
  `

[sorinpepelea]

Could you tell me the root password , so i can least try to set privilege manually for postgres user on that path ?

[jkatz]

Connecting to a PostgreSQL cluster is covered in the tutorial, amongst other topics. By default, user credentials are randomly generated. However, given that you cannot bootstrap your cluster, you will not be able to log in as the postgres user unless you connect directly to the Pod.

The Pod itself runs as the postgres user account. We specifically do not run as a root user at runtime, as that is considered a container anti-pattern.

If the issue is how your storage layer provisions persistent volumes, then the problem is outside the scope of the Operator. It sounds like there is some sort of disconnect between how the storage is mounted and what is the acceptable user allowed to write to it.

You may want to double check on the various security policies employed in your cluster. You may need to set a security policy (see #1362 for a long discussion around that).

[sorinpepelea]

I know about anti-patern. But i need to be able to use the root user just to test if there is indeed a security policy issue. However same security policy and storage layer provision did not block the
grid-backrest-shared-repo pod to run.
the backrest pod uses same type of pv ... and had no problem with creating folder.
I cannot use postres user, since this user does not have the rights for mkdir in the pod .. so i am stucked on privilege.
Thx for your support and prompt response. Maybe creating that dir with root will least reveal the problem.

[jkatz]

The pgBackRest Pod does not attempt any writes until after the PostgreSQL Pod finishes provisioning -- it cannot accept backups until PostgreSQL is ready. When the PostgreSQL Pod becomes ready, the Operator will issue both a create-stanza followed by a backup command, which at that point the pgBackRest Pod will attempt to write to its PVC. I would suspect that, based on the above, it would be unable to do so due to your storage system.

PostgreSQL also refuses to run as the root user, that is built into the software itself.

Anyway, as mentioned above, I would recommend determining what is happening with your storage provisioning and why it is preventing data from being written. Perhaps experiment by mounting one of the generated PVCs to a busybox to get a better understanding of what is going on.

[sorinpepelea]

ok. I'll test it this way

[kukosek]

Hello, I've had the same problems. After creating the cluster, I've had to create two local persistent volumes with the size of 1Gi like this:

apiVersion: v1
kind: PersistentVolume
metadata:
  name: hippo-vol-2
  namespace: pgo
  labels:
    type: local
spec:
  capacity:
    storage: 1Gi
  hostPath:
    path: "/home/hippo-vol-2"
  accessModes:
    - ReadWriteOnce
  volumeMode: Filesystem

Then the containers started working, but they got into a crash loop as described here. I solved it by running chmod 777 on the folders (/home/hippo-vol-x)