Running zammad as NONROOT still has issue in postgresql-init container

[ajeetratnu]

Is this a request for help?: YES

---

Is this a BUG REPORT or FEATURE REQUEST? (choose one): BUG

Version of Helm and Kubernetes:
helm version

version.BuildInfo{Version:"v3.5.2", GitCommit:"167aac70832d3a384f65f9745335e9fb40169dc2", GitTreeState:"dirty", GoVersion:"go1.15.7"}

kubectl version

Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:50:19Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"windows/amd64"}
Server Version: version.Info{Major:"1", Minor:"19+", GitVersion:"v1.19.3-dhc", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"dirty", BuildDate:"2020-10-15T07:10:10Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"linux/amd64"}

What happened:
Trying to install zammad as NONROOT user in private cloud. It stuck in postgresql-init container at ruby related commands.

What you expected to happen:
Zammad should be installed successfully.

How to reproduce it (as minimally and precisely as possible):
Downloaded zammad-helm project from GIT to local.
created myValues.yaml file

image:
  repository: myRepoPath/zammad-docker-compose
  tag: zammad-3.6.0-65
  pullPolicy: Always
  imagePullSecrets: 
   - name: "my-pull-secret"

elasticsearch:
  enabled: false
  enableInitialisation: false
  
memcached:
  image:
      registry: myRegistry
      repository: myPath/memcached

postgresql:
  image:
      registry: myRegistry
      repository: myPath/postgresql

Install zammad in private cloud using below helm command.

helm install zammad . --values=myValues.yaml -n zammad

Anything else we need to know:
I am trying with latest zammad v3.6.0-65 which has fix for bug #82 in it.
First of all thanks for your fixes in bug #82 , the updated commands in zammad-init container are working fine now.

Now when it reaches postgresql-init, it stuck at "bundle exec rake db:migrate" step.

I have added few echo statements in templates/configmap-init.yaml file as shown below (in and around the if statement) :

..........
  postgresql-init: |-
    #!/bin/bash
    set -e
    sed -e "s#.*adapter:.*#  adapter: postgresql#g" -e "s#.*database:.*#  database: {{ .Values.envConfig.postgresql.db }}#g" -e "s#.*username:.*#  username: {{ .Values.envConfig.postgresql.user }}#g" -e "s#.*password:.*#  password: ${POSTGRESQL_PASS}\\n  host: {{ if .Values.postgresql.enabled }}{{ .Release.Name }}-postgresql{{ else }}{{ .Values.envConfig.postgresql.host }}{{ end }}\\n  port: {{ .Values.envConfig.postgresql.port }}#g" < contrib/packager.io/database.yml.pkgr > config/database.yml
    echo "level 1"
    if ! (bundle exec rails r 'puts User.any?' 2> /dev/null | grep -q true); then
        echo "level 2"
        bundle exec rake db:migrate
        bundle exec rake db:seed
    else
        echo "level 3"
        bundle exec rake db:migrate
    fi
    echo "postgresql init complete :)"

If I check the logs, it print below two statement only, refer to echo statement in above file:

C:\Ajeet\zammad-helm-master-nonroot\zammad>kubectl logs zammad-0 -c postgresql-init -n zammad
level 1
level 2

I even tried to execute it manually as shown below, but it will exit (with code 137) after long time (>10 mins) but nothing shows in logs.

C:\Ajeet\zammad-helm-master-nonroot\zammad>kubectl exec -it zammad-0 -c postgresql-init -n zammad bash
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
zammad@zammad-0:~$ bundle exec rake db:migrate
command terminated with exit code 137

This is pod status

C:\Ajeet\zammad-helm-master-nonroot\zammad>kubectl get pod -n zammad
NAME                                READY   STATUS     RESTARTS   AGE
zammad-0                            0/4     Init:1/2   1          20m
zammad-memcached-5fbc5dc6db-hfxl6   1/1     Running    0          20m
zammad-postgresql-0                 1/1     Running    0          20m

when I am doing describe pod zammad-0 it is showing postgresql-init Terminated with Error code =1.

...........
  postgresql-init:
    Container ID:   containerd://d36149fd04a99e462ec33cef9d373e1f496d39aa666d9e9c7eb48581f9812610
    Image:          myRepoPath/zammad-docker-compose:zammad-3.6.0-65
    Image ID:       myRepoPath/zammad-docker-compose@sha256:743a6a93e0744738f396438869f082068a3e627747ed96a0c6000ce890485933
    Port:           <none>
    Host Port:      <none>
    State:          Running
      Started:      Mon, 08 Mar 2021 23:20:29 +0530
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
      Started:      Mon, 08 Mar 2021 23:09:16 +0530
      Finished:     Mon, 08 Mar 2021 23:20:29 +0530
    Ready:          False
    Restart Count:  1
    Environment:
      POSTGRESQL_PASS:  <set to the key 'postgresql-pass' in secret 'zammad-postgresql-pass'>  Optional: false
    Mounts:
      /docker-entrypoint.sh from zammad-init (ro,path="postgresql-init")
      /opt/zammad from zammad (rw)
      /var/run/secrets/kubernetes.io/serviceaccount from default-token-df875 (ro)

[monotek]

Can you please use code tags?
It's realy hard to read your issues.
See: https://docs.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks

The postgresql container is started via dependency chart. There were no changes related to the user.
This was onlky changed for zammad containers, so i doubt it's related.

This also worked in the ci run: https://github.com/zammad/zammad-helm/actions/runs/624522521
You might want to have a look in the logs of the postgresql container.

[ajeetratnu]

Can you please use code tags?
It's realy hard to read your issues.
See: https://docs.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks

The postgresql container is started via dependency chart. There were no changes related to the user.
This was onlky changed for zammad containers, so i doubt it's related.

This also worked in the ci run: https://github.com/zammad/zammad-helm/actions/runs/624522521
You might want to have a look in the logs of the postgresql container.

Hi, I have updated the issue with code tags now. Sorry for the trouble.
I have tried to check postgresql-init container logs, but its empty nothing coming there. I added few echo statements (described in issue above) to find out where it is getting stuck.
The problem comes when execution reached "bundle exec rake db:migrate".
For some reason, there is no error, output logs, anything here. Looks like stuck in infinite loop forever.

My guess, it could be due to access issue ? because we are trying as non root user and may be it is not able to either connect to volume or execute this db:migrate command. but I am not sure on this.
Also just a thought, if it help to identify issue, in your case, might be DB /volume is already existing. But in my case, it will try to create fresh as nothing exists.

[monotek]

Ok, so it's about the PostgreSQL init container.

Does the postgres database container work?

[ajeetratnu]

Ok, so it's about the PostgreSQL init container.

Does the postgres database container work?

Yes, the postgres Database container is showing up as shown below in pod status.
Let me know if you want me to verify anything specific in this container.

C:\Ajeet\zammad-helm-master-nonroot\zammad>kubectl get pod -n zammad
NAME                                READY   STATUS     RESTARTS   AGE
zammad-0                            0/4     Init:1/2   1          20m
zammad-memcached-5fbc5dc6db-hfxl6   1/1     Running    0          20m
zammad-postgresql-0                 1/1     Running    0          20m

[ajeetratnu]

Ok, so it's about the PostgreSQL init container.
Does the postgres database container work?

Yes, the postgres Database container is showing up as shown below in pod status.
Let me know if you want me to verify anything specific in this container.

C:\Ajeet\zammad-helm-master-nonroot\zammad>kubectl get pod -n zammad
NAME                                READY   STATUS     RESTARTS   AGE
zammad-0                            0/4     Init:1/2   1          20m
zammad-memcached-5fbc5dc6db-hfxl6   1/1     Running    0          20m
zammad-postgresql-0                 1/1     Running    0          20m

Hi @monotek , Just want to check if any update on this issue or if you need any more information from my side.
Appreciate if you can share any update. Thanks.

[ajeetratnu]

Hi, Just want to check if any update on this issue ? Any hint or suggestion will help us.

[monotek]

Please stop spamming.
I'm busy. If there is progress i will post it here.

[monotek]

I can't reproduce.

Last ci job worked: https://github.com/zammad/zammad-helm/runs/2149306752

Also fresh install in a gke cluster worked without problems:

 k get po
NAME                               READY   STATUS    RESTARTS   AGE
zammad-0                           4/4     Running   0          3m
zammad-master-0                    1/1     Running   0          3m
zammad-memcached-9bf8f96c5-t78jg   1/1     Running   0          3m
zammad-postgresql-0                1/1     Running   0          3m

I can only guess that it's some connection problem to postgres.
If you want to debug the init container a bit more, you could add a "-x" to the set command.

[ajeetratnu]

I can't reproduce.

Last ci job worked: https://github.com/zammad/zammad-helm/runs/2149306752

Also fresh install in a gke cluster worked without problems:

 k get po
NAME                               READY   STATUS    RESTARTS   AGE
zammad-0                           4/4     Running   0          3m
zammad-master-0                    1/1     Running   0          3m
zammad-memcached-9bf8f96c5-t78jg   1/1     Running   0          3m
zammad-postgresql-0                1/1     Running   0          3m

I can only guess that it's some connection problem to postgres.
If you want to debug the init container a bit more, you could add a "-x" to the set command.

Hi @monotek , Thanks for all your help.
I found the problem. My cluster has by default no communication between pods policy.
I have updated network policy to allow communication and now I can see all pods up and running.
I still have to do the configuration, but the first road block is out, I can install zammad successfully as non root.
Thanks for all your help.