Skip to content
This repository has been archived by the owner on Feb 22, 2022. It is now read-only.

[stable/pgadmin] files in /var/lib/pgadmin/sessions crash the pod #19690

Closed
mfbrj opened this issue Dec 19, 2019 · 16 comments
Closed

[stable/pgadmin] files in /var/lib/pgadmin/sessions crash the pod #19690

mfbrj opened this issue Dec 19, 2019 · 16 comments

Comments

@mfbrj
Copy link

mfbrj commented Dec 19, 2019

Describe the bug
Pod crashes after running for 45 hours and 30 minutes.

Version of Helm and Kubernetes:
Kubernetes: 1.16.3
Helm: not shure, I installed the chart from Rancher v.2.3.2 frontend
Template version: 1.0.5

Which chart: stable/pgadmin

What happened: Every 3 and 7 seconds, 2 files are created on /var/lib/pgadmin/sessions, totaling exactly 24 new files per minute, like that:

-rw-r--r--    1 root     root           198 2019-12-19 02:25:00 +0000 bdfce962-5b44-4b22-977e-f5c0b36c8ccb
-rw-r--r--    1 root     root           262 2019-12-19 02:25:00 +0000 ab25098b-b486-4b84-8524-2412238463cb
-rw-r--r--    1 root     root           198 2019-12-19 02:25:03 +0000 d4d787ff-d598-468e-8ae8-c3da8465f207
-rw-r--r--    1 root     root           262 2019-12-19 02:25:03 +0000 c1a52773-6270-430b-8366-bd689ecc3268
-rw-r--r--    1 root     root           262 2019-12-19 02:25:10 +0000 86dd579a-4e85-41d4-86f8-848fcf481f7b
-rw-r--r--    1 root     root           198 2019-12-19 02:25:10 +0000 3c54aaa0-0883-49d7-9a26-f7ab9026e806
-rw-r--r--    1 root     root           262 2019-12-19 02:25:13 +0000 d453dbfb-a88b-413e-8bd5-c78507789996
-rw-r--r--    1 root     root           198 2019-12-19 02:25:13 +0000 8d6d82da-cb36-4929-a9a1-f592851f9a73
-rw-r--r--    1 root     root           262 2019-12-19 02:25:20 +0000 cdbc0454-f0fb-4699-b245-97eee6da51be
-rw-r--r--    1 root     root           198 2019-12-19 02:25:20 +0000 4328443c-7fc9-4739-b40d-5eda892d9b8a
-rw-r--r--    1 root     root           198 2019-12-19 02:25:23 +0000 a84bd798-efd5-4849-b1c4-398c7f09bac3
-rw-r--r--    1 root     root           262 2019-12-19 02:25:23 +0000 81301d38-3289-4f04-b6fb-179aefd47837
-rw-r--r--    1 root     root           198 2019-12-19 02:25:30 +0000 4e8cb125-83de-4474-92a7-909690950874
-rw-r--r--    1 root     root           262 2019-12-19 02:25:30 +0000 29bc63df-31b3-43d7-99f4-faa75e7b8dbd
-rw-r--r--    1 root     root           198 2019-12-19 02:25:33 +0000 e23ba6c3-f117-4207-b114-0deb09a58a2a
-rw-r--r--    1 root     root           262 2019-12-19 02:25:33 +0000 0d8f42ad-dab5-4d69-860c-cfc5fef52777
-rw-r--r--    1 root     root           262 2019-12-19 02:25:40 +0000 efcfbe7d-3291-472b-a640-1e90ca2fcc7b
-rw-r--r--    1 root     root           198 2019-12-19 02:25:40 +0000 dd045e74-89e0-4585-8dec-9de7b0b2cf62
-rw-r--r--    1 root     root           262 2019-12-19 02:25:43 +0000 7fa04f67-8494-4082-b5c6-aa521d949500
-rw-r--r--    1 root     root           198 2019-12-19 02:25:43 +0000 40c5af89-cc60-4086-886f-eda465832d4e
-rw-r--r--    1 root     root           262 2019-12-19 02:25:50 +0000 f5ecbd00-31f4-4003-84cc-2e9b40e5f4da
-rw-r--r--    1 root     root           198 2019-12-19 02:25:50 +0000 591ff7c6-7294-43c9-8c8e-a1b4c9b86293
-rw-r--r--    1 root     root           262 2019-12-19 02:25:53 +0000 caf932aa-0aaa-4e53-8891-1b05ef773eec
-rw-r--r--    1 root     root           198 2019-12-19 02:25:53 +0000 ad70c61c-b40e-4364-8b79-add7a4bcded3

So, when there are 65520 files on folder /var/lib/pgadmin/sessions, pod crashes.

What you expected to happen: Pod lives forever.

How to reproduce it (as minimally and precisely as possible): Install the chart with the following values.yml:

---
ingress:
  enabled: true
  annotations:
    kubernetes.io/ingress.class: nginx
    kubernetes.io/tls-acme: "false"
  hosts:
    - host: pgadmin.mydomain.com
      paths:
        - /
  tls:
    - secretName: mysecret
      hosts:
        - pgadmin.mydomain.com
env:
  email: someone@somewhere.com
  password: SuperSecret
persistence:
  enabled: true
  existingClaim: "pgadmin"

Anything else we need to know: Pod works fine when I set the Health Check to None

@KIRY4
Copy link

KIRY4 commented Jan 14, 2020

Have the same issue with stable pgadmin chart on AKS. But in my case it failed immediately after deployment with CrashLoopbackOff here is logs any ideas???

########

kubectl logs pgadmin-6fc96b8c64-d664d -n pgadmin
WARNING: Failed to set ACL on the directory containing the configuration database: [Errno 1] Operation not permitted: '/var/lib/pgadmin'
Traceback (most recent call last):
File "run_pgadmin.py", line 4, in
from pgAdmin4 import app
File "/pgadmin4/pgAdmin4.py", line 109, in
app = create_app()
File "/pgadmin4/pgadmin/init.py", line 244, in create_app
create_app_data_directory(config)
File "/pgadmin4/pgadmin/setup/data_directory.py", line 42, in create_app_data_directory
os.chmod(config.SESSION_DB_PATH, 0o700)
PermissionError: [Errno 1] Operation not permitted: '/var/lib/pgadmin/sessions'
[2020-01-14 14:13:58 +0000] [1] [INFO] Starting gunicorn 19.9.0
[2020-01-14 14:13:58 +0000] [1] [INFO] Listening at: http://[::]:80 (1)
[2020-01-14 14:13:58 +0000] [1] [INFO] Using worker: threads
[2020-01-14 14:13:58 +0000] [19] [INFO] Booting worker with pid: 19
[2020-01-14 14:13:59 +0000] [19] [ERROR] Exception in worker process
Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
worker.init_process()
File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/gthread.py", line 104, in init_process
super(ThreadWorker, self).init_process()
File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base.py", line 129, in init_process
self.load_wsgi()
File "/usr/local/lib/python3.7/site-packages/gunicorn/workers/base.py", line 138, in load_wsgi
self.wsgi = self.app.wsgi()
File "/usr/local/lib/python3.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
self.callable = self.load()
File "/usr/local/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 52, in load
return self.load_wsgiapp()
File "/usr/local/lib/python3.7/site-packages/gunicorn/app/wsgiapp.py", line 41, in load_wsgiapp
return util.import_app(self.app_uri)
File "/usr/local/lib/python3.7/site-packages/gunicorn/util.py", line 350, in import_app
import(module)
File "/pgadmin4/run_pgadmin.py", line 4, in
from pgAdmin4 import app
File "/pgadmin4/pgAdmin4.py", line 109, in
app = create_app()
File "/pgadmin4/pgadmin/init.py", line 244, in create_app
create_app_data_directory(config)
File "/pgadmin4/pgadmin/setup/data_directory.py", line 42, in create_app_data_directory
os.chmod(config.SESSION_DB_PATH, 0o700)
PermissionError: [Errno 1] Operation not permitted: '/var/lib/pgadmin/sessions'
[2020-01-14 14:13:59 +0000] [19] [INFO] Worker exiting (pid: 19)
WARNING: Failed to set ACL on the directory containing the configuration database: [Errno 1] Operation not permitted: '/var/lib/pgadmin'
[2020-01-14 14:13:59 +0000] [1] [INFO] Shutting down: Master
[2020-01-14 14:13:59 +0000] [1] [INFO] Reason: Worker failed to boot.

@benoit-parmentier
Copy link

Hello,

The pgadmin Dockerfile ACL have changed few days ago. https://github.com/postgres/pgadmin4/blob/master/Dockerfile

Now, a user with the UID 5050 is required.

So, you have to update your StorageClass with and uid.

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: azurefilepgadmin
provisioner: kubernetes.io/azure-file
mountOptions:
  - dir_mode=0777
  - file_mode=0777
  - uid=5050
  - nobrl
  - cache=none
parameters:
  skuName: Standard_LRS

@KIRY4
Copy link

KIRY4 commented Jan 29, 2020

Hello,

The pgadmin Dockerfile ACL have changed few days ago. https://github.com/postgres/pgadmin4/blob/master/Dockerfile

Now, a user with the UID 5050 is required.

So, you have to update your StorageClass with and uid.

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: azurefilepgadmin
provisioner: kubernetes.io/azure-file
mountOptions:
  - dir_mode=0777
  - file_mode=0777
  - uid=5050
  - nobrl
  - cache=none
parameters:
  skuName: Standard_LRS

Thank you!! This solution fixed my issue!!

@mfbrj
Copy link
Author

mfbrj commented Jan 29, 2020

  • uid=5050

This config didn't work on vSphere. Error message:
MountVolume.MountDevice failed for volume "pvc-5e8a1416-f678-4ff3-9578-440a676f65ba" : mount failed: exit status 32 Mounting command: mount Mounting arguments: -t ext4 -o uid=5050,defaults /dev/disk/by-id/wwn-0x6000c296d65e764b672b052b3247c4bb /var/lib/kubelet/plugins/kubernetes.io/vsphere-volume/mounts/[VV_VMWARE_EVABACKUP_FC_RD5] kubevols/kubernetes-dynamic-pvc-5e8a1416-f678-4ff3-9578-440a676f65ba.vmdk Output: mount: wrong fs type, bad option, bad superblock on /dev/sdb, missing codepage or helper program, or other error In some cases useful info is found in syslog - try dmesg | tail or so.

What I'm missing?

@rowanruseler
Copy link
Contributor

@fredbeltrao apologies for the late reply, but please try to pull in the latest build I provided. It has a fix for this specific issue. Be aware that I adjusted some namings, persistance towards persistentVolume.

The fix itself was actually adjusting the path of the livenessProbe and readinessProbe:

livenessProbe:
  httpGet:
    path: /misc/ping
readinessProbe:
  httpGet:
    path: /misc/ping

@rowanruseler
Copy link
Contributor

@KIRY4 the issue you are having is also resolved in the latest build, it got merged Saterday. Check out version: 1.1.0 and appVersion: 4.17.0

@mfbrj
Copy link
Author

mfbrj commented Feb 4, 2020

@rowanruseler It worked great. Thank you for your work!

@mfbrj mfbrj closed this as completed Feb 4, 2020
@nela
Copy link

nela commented Mar 27, 2020

I am having same issues in docker compose.
Any idea how I would go about fixing this?

The error I am getting is:

pgadmin_1  | PermissionError: [Errno 13] Permission denied: '/var/lib/pgadmin/sessions'
pgadmin_1  | [2020-03-27 13:02:15 +0000] [87] [INFO] Worker exiting (pid: 87)
pgadmin_1  | WARNING: Failed to set ACL on the directory containing the configuration database: [Errno 1] Operation not permitted: '/var/lib/pgadmin'
pgadmin_1  | [2020-03-27 13:02:15 +0000] [1] [INFO] Shutting down: Master
pgadmin_1  | [2020-03-27 13:02:15 +0000] [1] [INFO] Reason: Worker failed to boot.
db:
    image: postgres
    restart: always
    ports:
      - 5432:5432
    volumes:
      - db-data:/var/lib/postgresql/data
    environment:
      POSTGRES_DB: db
      POSTGRES_USER: user
      POSTGRES_PASSWORD: passw
  pgadmin:
    image: dpage/pgadmin4
    user: 5050:5050
    depends_on:
      - db
    environment:
      PGADMIN_DEFAULT_EMAIL: supersecret@email.com
      PGADMIN_DEFAULT_PASSWORD: passw
    volumes:
      - pgadmin:/var/lib/pgadmin
    ports:
      - 5050:80

volumes:
  db-data:
  pgadmin:

@rowanruseler
Copy link
Contributor

rowanruseler commented Mar 27, 2020

@Nemox this is a helm build for this repo images. I would advice you to look at the documentation of the image itself: dpage/pgadmin4. The way I resolved it within kubernetes is first init a initcontainer and change the permissions. After that it launches the container with the pgadmin default user credentials.

@amatosg
Copy link

amatosg commented Apr 23, 2020

@Nemox if you solved your issue, please post the steps you followed. Thanks! :)

@nela
Copy link

nela commented Apr 24, 2020

@amatosg Hey, I just did a quick fix so my solution is not viable for sensitive deployments. But I made a directory, granted permissions to all on it and mounted pgadmin to this directory.

mkdir pgadmin_volume
sudo chmod -R 777 pgadmin_volume

Then in docker-compose.yml

    
volumes:
- ./pgadmin_volume:/var/lib/pgadmin

@pavelzamyatin
Copy link

Hello,

The pgadmin Dockerfile ACL have changed few days ago. https://github.com/postgres/pgadmin4/blob/master/Dockerfile

Now, a user with the UID 5050 is required.

So, you have to update your StorageClass with and uid.

kind: StorageClass
apiVersion: storage.k8s.io/v1
metadata:
  name: azurefilepgadmin
provisioner: kubernetes.io/azure-file
mountOptions:
  - dir_mode=0777
  - file_mode=0777
  - uid=5050
  - nobrl
  - cache=none
parameters:
  skuName: Standard_LRS

Legend! Thanks!

@amatosg
Copy link

amatosg commented Aug 25, 2020

I know this has been closed a while ago, but I have the same issue when using docker-compose. I created a folder and assigned full access to that folder (777) and added the volume /opt/pgadmin:/var/lib/pgadmin as suggested by @nela to no avail. I still have the same error message. The latest version that works is 4.15

Is there a solution for this when using docker-compose?

@nela
Copy link

nela commented Aug 26, 2020

@amatosg Where did you create the folder? I don´t remember the specifics of it but the pgadmin could not access root files. If it is not already in or user-folder, try creating it there: /home/$USER/mypgadminfolder, or anywhere else in the $USER space.

@amatosg
Copy link

amatosg commented Aug 26, 2020

@nela is the same volume used in 4.15. Even using a clean folder (same /opt folder) I get the same result. I don't think it's the parent folder as that's what I also use for the postgres volume

@TusharRoy23
Copy link

@Nemox this is a helm build for this repo images. I would advice you to look at the documentation of the image itself: dpage/pgadmin4. The way I resolved it within kubernetes is first init a initcontainer and change the permissions. After that it launches the container with the pgadmin default user credentials.

I solved with these below steps -
Step one:

docker run -p 8080:80 \
    -e "PGADMIN_DEFAULT_EMAIL=account@domain.com" \
    -e "PGADMIN_DEFAULT_PASSWORD=yourPassword" \
    -d dpage/pgadmin4

Step Two: Get login with the default credential which you wrote in docker-compose.yml
Step Three: Then stop that container.
Step Four: Then run your docker-compose.

It should work.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests

8 participants