Postgres Backup not working in vCluster

[farhaan-shamsee]

• Platform: (EKS v1.30.4-eks-a737599, vCluster v0.15.2.)
• PGO Image Tag: (e.g. 5.5.0)
• Postgres Version (e.g. 14.9)
• Storage: (e.g. gp3)

Questions

We are using the PostgresCluster for our keycloak instance. We are observing that the pgbackrest is failing to connect to the *-repo-host pods inside the vCluster

After sometime the Disk is out of space because of WAL keeps getting accumulated. Below is the screenshot for the same.

The below is the screenshot of the database disk usage after I re-created it.

I see the below files in the wal directory

Under the archive_status I see the below:

I read from the docs that the *.ready should become *.done if the backup is successful.

The same configuration works in my host cluster (EKS).

Can anyone help me to figure this out.

[andrewlecuyer]

hi @farhaan-shamsee, just wanted to let you know that I'm seeing the same issue when deploying a PostgresCluster within a vCluster. And from what I can tell, this appears to be an issue with vCluster networking, rather than anything specific to PGO.

More specifically, the DNS name PGO is using to connect to the repo host (e.g. hippo-repo-host-0.hippo-pods.postgres-operator.svc.kubernetes in my PostgresCluster named hippo) should definitely be valid. And the fact that this works just fine outside of a vCluster confirms this.

So it looks like some input from the vCluster project is likely required to get a better idea as to why this isn't working.

[andrewlecuyer]

This appears to be an issue with DNS due to vCluster adding hostAliases to the various Pods it creates, which is affecting the creation of DNS for services.

@farhaan-shamsee can you try setting the following environment variable in your PGO deployment, and let me know what you get?

kubectl -n postgres-operator set env deploy/pgo GODEBUG=netdns=cgo

Thanks!

[farhaan-shamsee]

Thanks a lot for helping me out.

@farhaan-shamsee can you try setting the following environment variable in your PGO deployment, and let me know what you get?

What specifically I should share?

These are the PGO logs:

And after I update the ENV, the pgha1-* pod went into error state. And now I see the below logs:

[andrewlecuyer]

@farhaan-shamsee just wanted to circle-back on this thread to see if you're still having trouble with your vCluster deployments.

Looking at the logs you provided, while there is an error condition with Patroni talking to the DCS, it looks like it was able recover an get back to a healthy state.

[farhaan-shamsee]

Hi @andrewlecuyer,
yes, we were able to resolve this. The issue was with the vCluster helm chart(0.15.2 version) which we were using. Specifically with the coreDNS component. It was not able to direct the traffic properly, and was always searching for cluster.local as the service name. But in vClusters the service name are NS specific. Hence it was failing.

we added the below into the coreDNS configuration
kubernetes{{ if and (.Values.networking.advanced.clusterDomain) (ne .Values.networking.advanced.clusterDomain "cluster.local") }} {{ .Values.networking.advanced.clusterDomain }}{{ end }} cluster.local in-addr.arpa ip6.arpa {

After this fix, it was working. I can hence confirm it was not the issue with Postgres but with vCluster.