Adding active operator feature

[tmjd]

Description

This feature allows the operator to be deployed to alternate namespaces and the ability to designate which will be the active one.

For PR author

• Tests for change.
• If changing pkg/apis/, run make gen-files
• If changing versions, run make gen-versions

For PR reviewers

A note for code reviewers - all pull requests must have the following:

• Milestone set according to targeted release.
• Appropriate labels:
  • kind/bug if this is a bugfix.
  • kind/enhancement if this is a a new feature.
  • enterprise if this PR applies to Calico Enterprise only.

[caseydavenport]

Overall looks pretty good - I haven't reviewed tests yet so will do that next.

Did we decide that we don't need to handle cached secrets during rolling update when switching between operators?

[caseydavenport]

"argument" -> "return value"

[caseydavenport]

If multiple operators hit this, I'm guessing they will fight with each other to maintain this active operator status?

[tmjd]

It would only be an initial race. Both (or multiple) if running at the same time one could create the map and then a second could overwrite the map. But once either re-reads the config they would not touch it again since they only create the configMap if it doesn't already exist.
So I wouldn't say they fight because an operator will only attempt to create this ConfigMap once.

[caseydavenport]

I meant if two or more operators fail to get the active configmap, they will both try to set degraded on the same tigera status resource.

Probably not an issue? Will they try to do anything differently from each other?

[tmjd]

Oh I see. Sorry about misunderstanding that.
Yeah I see what you mean. Maybe we should not SetDegraded in this specific case and just log it? Or because of the possible fight should we include the namespace in the message, that way it would be an indicator if it was coming from an unexpected operator?

[tmjd]

Did we decide that we don't need to handle cached secrets during rolling update when switching between operators?

I forgot about this consideration.
It is already a manual separate step to change the active designation, (1) maybe part of the process for changing the active operator includes copying over the secrets to the new active namespace.
Other options I can imagine:
2. Always use tigera-operator namespace for storing the secrets/config that the operator writes to "its namespace".
3. Include a namespace specification in the active-operator ConfigMap for where the secrets/config will be located.

If we did 2 then the operator would need to create tigera-operator namsepace if it didn't exist. And I think if that was the case then we could move the 'active-operator' ConfigMap there.
Always having the operator secrets/config (and the 'active-operator' CM) in the tigera-operator namespace is appealing to me.
Are there downsides to 2? The only thing I can think of is that then the tigera-operator namespace always must exist but I'm not sure how that is different from always having calico-system.

1 is appealing to me since I already have that done and then we could always expand on that to 3 if we didn't want the extra step of copying the secrets/config.

[caseydavenport]

maybe part of the process for changing the active operator includes copying over the secrets to the new active namespace.

I think I'm happy with this for now.

I don't see any immediate downsides to (2) except that it's maybe a bit weird? Not sure if some day we might want the ability to have separate secrets for the different operators.

So, I think my pref. is (1), which means no code changes here which is doubly good.

[tmjd]

Even though you're pref is 1 I just thought I'd mention a downside for 2 that I thought of. 2 would be a breaking change for anyone already running the operator in a different namespace, or at least it would require special steps on upgrade.

1 it is.

[tmjd]

I verified in a cluster that if I deployed a 2nd operator in a new namespace tigera-operator-alt (along with a new SA and binding) that if I copied over the typha and node cert secrets and the typha-ca configmap from the tigera-operator namespace, I was able to switch the active-operator config map to the tigera-operator-alt namespace and there were no typha or node restarts. The alt operator took over and the tigera-operator operator restarted and began waiting to become active.
Switching the active operator back worked as expected too.