pgo user - error command terminated with exit code 2 for crunchy 4.7

[ckvtvm-arch]

I was able to work on crunchy on the previous versions, but with 4.7 , I was not able to retrieve the initial login password for admin or even to create /update the user.

Is this is because of the new pgbackrest ssh authentication which is creating the issue for the new cluster pgadmin access

I am running these clusters in openshift

af71194@LC02CHB58MD6R ~ % pgo show user --show-system-accounts keycloak-postgresql -n vantage-sit

CLUSTER USERNAME PASSWORD EXPIRES STATUS ERROR

---

                              error  command terminated with exit code 2 

af71194@LC02CHB58MD6R ~ % pgo update user --username postgres --set-system-account-password -s name=keycloak-postgresql -n vantage-sit --password admin

CLUSTER USERNAME PASSWORD EXPIRES STATUS ERROR

---

keycloak-postgresql postgres error command terminated with exit code 2
af71194@LC02CHB58MD6R ~ %

[ckvtvm-arch]

all the pgo commands to create / update user have the same issues

af71194@LC02CHB58MD6R ~ % pgo show user --selector=name=keycloak-postgresql -n vantage-sit --debug
DEBU[0000] debug flag is set to true
DEBU[0000] in initConfig with url=https://127.0.0.1:8443
DEBU[0000] GetSessionCredentials called
DEBU[0000] PGOUSER environment variable is being used at /Users/af71194/.pgo/pgo/pgouser
DEBU[0000] pgouser file found at /Users/af71194/.pgo/pgo/pgouser contains admin:admin
DEBU[0000] [admin admin]
DEBU[0000] username=[admin] password=[admin]
DEBU[0000] setting up httpclient with TLS
DEBU[0000] GetTLSTransport called
DEBU[0000] ShowUser called...[https://127.0.0.1:8443/usershow]
DEBU[0000] &{200 OK 200 HTTP/1.1 1 1 map[Content-Length:[164] Content-Type:[application/json] Date:[Sat, 12 Jun 2021 00:22:11 GMT] Www-Authenticate:[Basic realm="Restricted"]] 0xc0007b6600 164 [] false false map[] 0xc00035e600 0xc00028c160}
DEBU[0000] http status code is 200

CLUSTER USERNAME PASSWORD EXPIRES STATUS ERROR

---

                              error  command terminated with exit code 2 

af71194@LC02CHB58MD6R ~ % pgo create user --username=admin --password=admin keycloak-postgresql --managed -n vantage-sit

CLUSTER USERNAME PASSWORD EXPIRES STATUS ERROR

---

keycloak-postgresql admin error command terminated with exit code 2

[ckvtvm-arch]

af71194@LC02CHB58MD6R ~ % pgo update user -s name=keycloak-postgresql --username=postgres --set-system-account-password --password=admin -n vantage-sit

CLUSTER USERNAME PASSWORD EXPIRES STATUS ERROR

---

keycloak-postgresql postgres error command terminated with exit code 2
af71194@LC02CHB58MD6R ~ %

[ckvtvm-arch]

wil this be an issue with new pgo4.7 ? .i am also investigating

[jkatz]

Can you provide logs from the apiserver container in the postgres-operator Pod after running these commands?

[ckvtvm-arch]

time="2021-06-11T23:59:30Z" level=error msg="command terminated with exit code 2" func="internal/kubeapi.ExecToPodThroughAPI()" file="internal/kubeapi/exec.go:76" version=4.7.0
time="2021-06-11T23:59:30Z" level=error msg="command terminated with exit code 2" func="internal/apiserver/userservice.ShowUser()" file="internal/apiserver/userservice/userimpl.go:506" version=4.7.0

[ckvtvm-arch]

time="2021-06-11T23:59:30Z" level=error msg="command terminated with exit code 2" func="internal/kubeapi.ExecToPodThroughAPI()" file="internal/kubeapi/exec.go:76" version=4.7.0
time="2021-06-11T23:59:30Z" level=error msg="command terminated with exit code 2" func="internal/apiserver/userservice.ShowUser()" file="internal/apiserver/userservice/userimpl.go:506" version=4.7.0
time="2021-06-12T00:03:27Z" level=error msg="command terminated with exit code 2" func="internal/kubeapi.ExecToPodThroughAPI()" file="internal/kubeapi/exec.go:76" version=4.7.0
time="2021-06-12T00:03:27Z" level=error msg="command terminated with exit code 2" func="internal/apiserver/userservice.ShowUser()" file="internal/apiserver/userservice/userimpl.go:506" version=4.7.0
time="2021-06-12T00:05:09Z" level=error msg="command terminated with exit code 2" func="internal/kubeapi.ExecToPodThroughAPI()" file="internal/kubeapi/exec.go:76" version=4.7.0
time="2021-06-12T00:05:09Z" level=error msg="command terminated with exit code 2" func="internal/apiserver/userservice.updateUser()" file="internal/apiserver/userservice/userimpl.go:1166" version=4.7.0
time="2021-06-12T00:09:14Z" level=error msg="command terminated with exit code 2" func="internal/kubeapi.ExecToPodThroughAPI()" file="internal/kubeapi/exec.go:76" version=4.7.0
time="2021-06-12T00:09:14Z" level=error msg="command terminated with exit code 2" func="internal/apiserver/userservice.updateUser()" file="internal/apiserver/userservice/userimpl.go:1166" version=4.7.0
time="2021-06-12T00:14:57Z" level=error msg="command terminated with exit code 2" func="internal/kubeapi.ExecToPodThroughAPI()" file="internal/kubeapi/exec.go:76" version=4.7.0
time="2021-06-12T00:14:57Z" level=error msg="command terminated with exit code 2" func="internal/apiserver/userservice.ShowUser()" file="internal/apiserver/userservice/userimpl.go:506" version=4.7.0
time="2021-06-12T00:16:50Z" level=error msg="command terminated with exit code 2" func="internal/kubeapi.ExecToPodThroughAPI()" file="internal/kubeapi/exec.go:76" version=4.7.0
time="2021-06-12T00:16:50Z" level=error msg="command terminated with exit code 2" func="internal/apiserver/userservice.updateUser()" file="internal/apiserver/userservice/userimpl.go:1166" version=4.7.0
time="2021-06-12T00:21:46Z" level=error msg="command terminated with exit code 2" func="internal/kubeapi.ExecToPodThroughAPI()" file="internal/kubeapi/exec.go:76" version=4.7.0
time="2021-06-12T00:21:46Z" level=error msg="command terminated with exit code 2" func="internal/apiserver/userservice.ShowUser()" file="internal/apiserver/userservice/userimpl.go:506" version=4.7.0
time="2021-06-12T00:22:11Z" level=error msg="command terminated with exit code 2" func="internal/kubeapi.ExecToPodThroughAPI()" file="internal/kubeapi/exec.go:76" version=4.7.0
time="2021-06-12T00:22:11Z" level=error msg="command terminated with exit code 2" func="internal/apiserver/userservice.ShowUser()" file="internal/apiserver/userservice/userimpl.go:506" version=4.7.0
time="2021-06-12T00:25:17Z" level=error msg="command terminated with exit code 2" func="internal/kubeapi.ExecToPodThroughAPI()" file="internal/kubeapi/exec.go:76" version=4.7.0
time="2021-06-12T00:25:17Z" level=error msg="command terminated with exit code 2" func="internal/util.SetPostgreSQLPassword()" file="internal/util/cluster.go:510" version=4.7.0
time="2021-06-12T00:25:17Z" level=error msg="command terminated with exit code 2" func="internal/apiserver/userservice.CreateUser()" file="internal/apiserver/userservice/userimpl.go:248" version=4.7.0
time="2021-06-12T00:31:32Z" level=error msg="command terminated with exit code 2" func="internal/kubeapi.ExecToPodThroughAPI()" file="internal/kubeapi/exec.go:76" version=4.7.0
time="2021-06-12T00:31:32Z" level=error msg="command terminated with exit code 2" func="internal/apiserver/userservice.ShowUser()" file="internal/apiserver/userservice/userimpl.go:506" version=4.7.0
time="2021-06-12T00:31:38Z" level=error msg="command terminated with exit code 2" func="internal/kubeapi.ExecToPodThroughAPI()" file="internal/kubeapi/exec.go:76" version=4.7.0
time="2021-06-12T00:31:38Z" level=error msg="command terminated with exit code 2" func="internal/apiserver/userservice.ShowUser()" file="internal/apiserver/userservice/userimpl.go:506" version=4.7.0
time="2021-06-12T00:32:49Z" level=error msg="command terminated with exit code 2" func="internal/kubeapi.ExecToPodThroughAPI()" file="internal/kubeapi/exec.go:76" version=4.7.0
time="2021-06-12T00:32:49Z" level=error msg="command terminated with exit code 2" func="internal/apiserver/userservice.ShowUser()" file="internal/apiserver/userservice/userimpl.go:506" version=4.7.0
time="2021-06-12T00:35:23Z" level=error msg="command terminated with exit code 2" func="internal/kubeapi.ExecToPodThroughAPI()" file="internal/kubeapi/exec.go:76" version=4.7.0
time="2021-06-12T00:35:23Z" level=error msg="command terminated with exit code 2" func="internal/apiserver/userservice.ShowUser()" file="internal/apiserver/userservice/userimpl.go:506" version=4.7.0
time="2021-06-12T00:46:45Z" level=error msg="command terminated with exit code 2" func="internal/kubeapi.ExecToPodThroughAPI()" file="internal/kubeapi/exec.go:76" version=4.7.0
time="2021-06-12T00:46:45Z" level=error msg="command terminated with exit code 2" func="internal/apiserver/userservice.updateUser()" file="internal/apiserver/userservice/userimpl.go:1166" version=4.7.0
time="2021-06-12T00:56:31Z" level=error msg="command terminated with exit code 2" func="internal/kubeapi.ExecToPodThroughAPI()" file="internal/kubeapi/exec.go:76" version=4.7.0
time="2021-06-12T00:56:31Z" level=error msg="command terminated with exit code 2" func="internal/apiserver/userservice.updateUser()" file="internal/apiserver/userservice/userimpl.go:1166" version=4.7.0
time="2021-06-12T00:57:25Z" level=error msg="command terminated with exit code 2" func="internal/kubeapi.ExecToPodThroughAPI()" file="internal/kubeapi/exec.go:76" version=4.7.0
time="2021-06-12T00:57:25Z" level=error msg="command terminated with exit code 2" func="internal/apiserver/userservice.updateUser()" file="internal/apiserver/userservice/userimpl.go:1166" version=4.7.0
time="2021-06-12T01:00:40Z" level=error msg="command terminated with exit code 2" func="internal/kubeapi.ExecToPodThroughAPI()" file="internal/kubeapi/exec.go:76" version=4.7.0
time="2021-06-12T01:00:40Z" level=error msg="command terminated with exit code 2" func="internal/apiserver/userservice.updateUser()" file="internal/apiserver/userservice/userimpl.go:1166" version=4.7.0
time="2021-06-12T01:03:41Z" level=error msg="command terminated with exit code 2" func="internal/kubeapi.ExecToPodThroughAPI()" file="internal/kubeapi/exec.go:76" version=4.7.0
time="2021-06-12T01:03:41Z" level=error msg="command terminated with exit code 2" func="internal/apiserver/userservice.updateUser()" file="internal/apiserver/userservice/userimpl.go:1166" version=4.7.0
time="2021-06-12T01:04:04Z" level=error msg="command terminated with exit code 2" func="internal/kubeapi.ExecToPodThroughAPI()" file="internal/kubeapi/exec.go:76" version=4.7.0
time="2021-06-12T01:04:04Z" level=error msg="command terminated with exit code 2" func="internal/apiserver/userservice.ShowUser()" file="internal/apiserver/userservice/userimpl.go:506" version=4.7.0

[jkatz]

Can you please provide the following information that is helpful for troubleshooting:

• Platform: (Kubernetes, OpenShift, Rancher, GKE, EKS, AKS etc.)
• Platform Version: (e.g. 1.20.3, 4.7.0)
• PGO Image Tag: (e.g. centos8-4.7.0)
• Postgres Version (e.g. 13)
• Storage: (e.g. hostpath, nfs, or the name of your storage class)

Can you also provide the image tag of the container you are using, and if you did anything special while upgrading.

[ckvtvm-arch]

Platform: , OpenShift
Platform Version: Client Version: openshift-clients-4.4.0-202006211643.p0-2-gd89e458c3
Server Version: 4.6.17
Kubernetes Version: v1.19.0+e405995

PGO Image Tag:
pgo client version 4.7.0
pgo-apiserver version 4.7.0

Postgres Version (e.g. 13) - psql (PostgreSQL) 13.3
Storage: nfs (planned to use as soon as the cluster is set up).

[ckvtvm-arch]

I havent cutomized the image:

image: >-
registry.developers.crunchydata.com/crunchydata/crunchy-postgres-ha:centos8-13.3-4.7.0

[jkatz]

The "error" that is being returned is due to the command being executed on the image failing. If you customized the image, this could very well be the case that your customization is creating some interference with the command that is being executed (which is just psql).

If the image is customized, this is likely not a bug with the Operator (also given we test the user operations against standard images).

As a side note, with NFS, there is a lot you need to consider when storing your data on it that could end up leading into issues later on. I suggest reading the PostgreSQL documentation on NFS about this.

[ckvtvm-arch]

we haven't customized the image and also we are not using NFS for now. does it means the PGO command fails on the image 4.7 which was working well in previous versions?

[ckvtvm-arch]

no probs