How to properly upgrade CRD and cluster

[hagaram]

Please, answer some short questions which should help us to understand your problem / question better?

• Which image of the operator are you using?
  before_upgrade: registry.opensource.zalan.do/acid/postgres-operator:v1.6.2
  after_upgrade: registry.opensource.zalan.do/acid/postgres-operator:v1.6.3

• **Where do you run it - cloud or metal?
  on-premise Kubernetes

• Are you running Postgres Operator in production?
  yes

• Type of issue? [Bug report, question, feature request, etc.]
  question

After several tries, I must ask...
How do you guys proceed when upgrading CRD and Zalando operator in general ?
As I found out, helm isn't of much help when upgrading CRDs, so I had to take "manual" approach via kubectl.

I'm trying to utilize the new flag called enableNamespacedSecret.

So what I did was applied new yaml files, namely operatorconfigurations.yaml and postgresteams.yaml, checked how it looks and everything looked OK.

postgresqls.acid.zalan.do CRD had enableNamedSecret available, so I tried to modify my cluster and apply the new flag
...
spec:
enableLogicalBackup: true
enableNamespacedSecret: true
teamId: "external"
postgresql:
version: "13"
numberOfInstances: 2
volume:
...

Which resulted in
error validating foo.yml": error validating data: ValidationError(postgresql.spec): unknown field "enableNamespacedSecret" in do.zalan.acid.v1.postgresql.spec; if you choose to ignore these errors, turn validation off with --validate=false

So I I tought that, maybe the configuration wasn't applied properly, so I killed (and deleted, tried both) operators POD, so it recreates with new configuration.

What happend is that operatorconfigurations.acid.zalan.do CRD stayed updated, with all new options and everything, but postgresqls.acid.zalan.do CRD lost the new enableNamespacedSecret option. It is still present in last applied configuration, but not present in the YAML itself if that makes sense. It seems like it reverted to older config.

I think, there might be to separate issues

1. operator reverts configuration after POD goes down, which might be unrelated to operator itself, but it might be something Kubernetes do for some reason.
2. Already deployed postgres cluster still uses old postgres resource definition.

So my question is HOW DO YOU UPGRADE ALL COMPONENTS PROPERLY?

[sigi4]

I'm trying to use "eanbleNamespacedSecret" myself right now but after me searching around, it looks like this flag is 5 days old #1490

I'm referencing 1.6.3 in my kustomization file and can't apply it at all.

When i describe the CRD, i can't find the flag either.

[hagaram]

This is what i figured out

This file validates the CRD options:

postgres-operator/pkg/apis/acid.zalan.do/v1/crds.go

Line 294 in b300fca

"enableConnectionPooler": {

The latest commit added the enableNamespacedSecret option, see:

postgres-operator/pkg/apis/acid.zalan.do/v1/crds.go

Line 733 in 75a9e2b

"enableNamespacedSecret": {

But I've looked into binary inside registry.opensource.zalan.do/acid/postgres-operator:v1.6.3 image

with strings /postgres-operator | grep -o --color=auto "enable[A-Z][a-z]*" and there is no sign of enableNamespacedSecret

So what I think happens is that when we add the new option, it simply doesn't pass the validation and operator rewrites the CRD config, to satisfy the validation.

Long story short ... I think a new operator image needs to be built.

[sigi4]

Yes. I switched to a tag of the latest git commit.

As long as it is not released it will not work.

[sigi4]

Here is the tag i'm using: registry.opensource.zalan.do/acid/postgres-operator:v1.6.3-4-gebb3204c found it through the swagger ui which provides access to the docker v2 api

[hagaram]

@sigi4 I went with registry.opensource.zalan.do/acid/postgres-operator:v1.6.3-5-g53fb540c which has the NamespacedSecret option present